From hch at lst.de Wed Jun 1 00:36:09 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 1 Jun 2005 09:36:09 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1117573386.1373.2.camel@duffman> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> Message-ID: <20050601073609.GA14042@lst.de> On Tue, May 31, 2005 at 02:03:06PM -0700, Tom Duffy wrote: > On Sat, 2005-05-28 at 09:13 +0200, Christoph Hellwig wrote: > > On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: > > > kDAPL is intended as a kernel-level API > > > for RDMA enabled fabrics. As it was initially written, > > > it does not meet the Linux coding style and that is why > > > it is being totally reworked as we speak to meet that goal. > > > > The codingstyle alone isn't the problem. The whole design philosophy > > is rather odd. > > As one of the people trying to clean up kDAPL, I would like to know what > you think, from a design philosophy, is wrong with it. We *can* correct > any daim bramaged parts. kDAPL is supposed to serve two needs: (1) provide an unified API for different RDMA transports (2) provide various higher level helpers as such it's largely duplicating what a proper RDMA stack should be. For (1) doing a proper RDMA stack should solve thing, and the discussion how to do it is already ongoing on this list. Once we have proper RMDA stack that part of KDAPL isn't needed at all anymore. The second is more interesting and there's indeed the need for some higher level helpers than the API at the level of the current OpenIB code offers. But having a separate layer, with different data structures, provider registration and a totally different API for that is utter nonsense. Instead the higher level helpers should operate on the same datastructures as the RDMA stack, or build new ones ontop of that. In addition some of the abstractions don't make much sense, the event handling has already been mentioned. From hch at lst.de Wed Jun 1 03:04:10 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 1 Jun 2005 12:04:10 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050601073609.GA14042@lst.de> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> Message-ID: <20050601100410.GA16265@lst.de> On Wed, Jun 01, 2005 at 09:36:09AM +0200, Christoph Hellwig wrote: > For (1) doing a proper RDMA stack should solve thing, and the discussion > how to do it is already ongoing on this list. Once we have proper RMDA > stack that part of KDAPL isn't needed at all anymore. The second is > more interesting and there's indeed the need for some higher level > helpers than the API at the level of the current OpenIB code offers. But > having a separate layer, with different data structures, provider > registration and a totally different API for that is utter nonsense. > Instead the higher level helpers should operate on the same > datastructures as the RDMA stack, or build new ones ontop of that. > In addition some of the abstractions don't make much sense, the event > handling has already been mentioned. That beeing said, one of the first thing you should get rid of if you want to be able to take code from kdapl to the generic rdma code is way it deals with handles. The kdapl code gives up language type safety just to add its own bandaided type system below. Please always pass the real struct pointer around, or at least the generic struct type which might be embedded into a bigger structure, and get rid of all the magic number checking and magic macros that expect special members at the beginning of all objects behind these handles. From Thomas.Talpey at netapp.com Wed Jun 1 06:07:01 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Wed, 01 Jun 2005 09:07:01 -0400 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050601073609.GA14042@lst.de> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> Message-ID: <6.2.1.2.2.20050601071057.05694e70@exnane01.nane.netapp.com> At 03:36 AM 6/1/2005, Christoph Hellwig wrote: >kDAPL is supposed to serve two needs: > (1) provide an unified API for different RDMA transports > (2) provide various higher level helpers >... >For (1) doing a proper RDMA stack should solve thing, and the discussion >how to do it is already ongoing on this list. Once we have proper RMDA >stack that part of KDAPL isn't needed at all anymore. I certainly agree, and have said so before, that if there is a portable layer (verbs or otherwise) which allows me to have a single NFS module working over both IB and iWARP, then kDAPL is not architecturally necessary. The verbs API however, isn't it. The current NFS/RDMA client, over kDAPL, is 3.6KLOC, on the order of the same amount of code as the sockets NFS: about 1800 lines for connection management and transport handling, about 800 for marshalling to and from RPC, and the rest managing the RPC's themselves. This is a HUGE advantage, because it means none of the RDMA complexity is in the RPC or NFS modules. In fact, the NFS module is completely unchanged (same binary). Which is exactly how want it! So, again, I'm happy to use another RDMA API. But the top-level requirements are: - must enable single upper layer implementation (one RPC/RDMA module) - must perform equally well for all transports - must not expose needless complexity - must exist :-) I think kDAPL already is such a thing. When the OpenIB kDAPL starts to work, we'll run the NFS/RDMA client on it the next day. BTW the Linux NFS/RDMA server (also on kDAPL) is accepting connections at CITI, now moving on to actually executing RPCs. Tom. From tduffy at sun.com Wed Jun 1 11:13:46 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 11:13:46 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050601100410.GA16265@lst.de> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> Message-ID: <1117649626.29566.2.camel@duffman> On Wed, 2005-06-01 at 12:04 +0200, Christoph Hellwig wrote: > That beeing said, one of the first thing you should get rid of if you > want to be able to take code from kdapl to the generic rdma code is way > it deals with handles. The kdapl code gives up language type safety > just to add its own bandaided type system below. Please always pass the > real struct pointer around, or at least the generic struct type which > might be embedded into a bigger structure, and get rid of all the magic > number checking and magic macros that expect special members at the > beginning of all objects behind these handles. I absolutely agree with this and was planning on implementing said changes in the near future. James, can I get a "hell yeah" or a "no freaking way" from you before I go off and start writing patches? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Wed Jun 1 11:23:44 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 11:23:44 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more Message-ID: <1117650224.29566.6.camel@duffman> This patch cleans up dat/ a bit, removing unneeded parenthesis, gratuitous comments, and a few whitespace cleanups. Please apply. Signed-off-by: Tom Duffy api.c | 684 ++++++++++++++++++----------------------------------------- core.c | 24 -- dictionary.c | 90 +------ dr.c | 67 ----- dr.h | 6 5 files changed, 242 insertions(+), 629 deletions(-) Index: linux-kernel-new/dat/dictionary.c =================================================================== --- linux-kernel-new/dat/dictionary.c (revision 2518) +++ linux-kernel-new/dat/dictionary.c (working copy) @@ -33,28 +33,11 @@ #include "dictionary.h" -/********************************************************************* - * * - * Function Declarations * - * * - *********************************************************************/ - static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, struct dat_provider_info *new_key); -static boolean_t -dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, - const struct dat_provider_info *key_b); - -/********************************************************************* - * * - * External Functions * - * * - *********************************************************************/ - -/*********************************************************************** - * Function: dat_dictionary_destroy - ***********************************************************************/ +static boolean_t dat_dict_key_is_equal(const struct dat_provider_info *key_a, + const struct dat_provider_info *key_b); void dat_dictionary_destroy(struct dat_dictionary *dictionary) { @@ -71,10 +54,6 @@ void dat_dictionary_destroy(struct dat_d } } -/*********************************************************************** - * Function: dat_dictionary_size - ***********************************************************************/ - u32 dat_dictionary_size(const struct dat_dictionary *dictionary, int *size) { BUG_ON(NULL == dictionary); @@ -85,10 +64,6 @@ u32 dat_dictionary_size(const struct dat return DAT_SUCCESS; } -/*********************************************************************** - * Function: dat_dictionary_entry_create - ***********************************************************************/ - u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry) { struct dat_dictionary_entry *entry; @@ -105,23 +80,15 @@ u32 dat_dictionary_entry_create(struct d *p_entry = entry; - bail: +bail: return status; } -/*********************************************************************** - * Function: dat_dictionary_entry_destroy - ***********************************************************************/ - void dat_dictionary_entry_destroy(const struct dat_dictionary_entry *entry) { kfree(entry); } -/*********************************************************************** - * Function: dat_dictionary_insert - ***********************************************************************/ - u32 dat_dictionary_insert(struct dat_dictionary *dictionary, const struct dat_provider_info *key, struct dat_dictionary_entry *entry, void *data) @@ -148,14 +115,10 @@ u32 dat_dictionary_insert(struct dat_dic dictionary->size++; - bail: +bail: return status; } -/*********************************************************************** - * Function: dat_dictionary_search - ***********************************************************************/ - u32 dat_dictionary_search(struct dat_dictionary *dictionary, const struct dat_provider_info *key, void **p_data) { @@ -168,7 +131,7 @@ u32 dat_dictionary_search(struct dat_dic status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); list_for_each_entry(cur_entry, &dictionary->list, list) { - if (TRUE == dat_dictionary_key_is_equal(&cur_entry->key, key)) { + if (TRUE == dat_dict_key_is_equal(&cur_entry->key, key)) { if (NULL != p_data) *p_data = cur_entry->data; @@ -177,14 +140,10 @@ u32 dat_dictionary_search(struct dat_dic } } - bail: +bail: return status; } -/*********************************************************************** - * Function: dat_dictionary_enumerate - ***********************************************************************/ - u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, void *array[], int array_size) { @@ -206,14 +165,10 @@ u32 dat_dictionary_enumerate(struct dat_ list_for_each_entry(cur_entry, &dictionary->list, list) array[i++] = cur_entry->data; - bail: +bail: return status; } -/*********************************************************************** - * Function: dat_dictionary_remove - ***********************************************************************/ - u32 dat_dictionary_remove(struct dat_dictionary *dictionary, const struct dat_provider_info *key, struct dat_dictionary_entry **p_entry, void **p_data) @@ -231,7 +186,7 @@ u32 dat_dictionary_remove(struct dat_dic list_for_each_safe(cur_list, next_list, &dictionary->list) { cur_entry = list_entry(cur_list, struct dat_dictionary_entry, list); - if (TRUE == dat_dictionary_key_is_equal(&cur_entry->key, key)) { + if (TRUE == dat_dict_key_is_equal(&cur_entry->key, key)) { if (NULL != p_data) *p_data = cur_entry->data; @@ -246,22 +201,12 @@ u32 dat_dictionary_remove(struct dat_dic } } - bail: +bail: return status; } -/********************************************************************* - * * - * Internal Function Definitions * - * * - *********************************************************************/ - -/*********************************************************************** - * Function: dat_dictionary_key_create - ***********************************************************************/ - -u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, - struct dat_provider_info *new_key) +static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, + struct dat_provider_info *new_key) { BUG_ON(NULL == old_key); BUG_ON(NULL == new_key); @@ -274,12 +219,8 @@ u32 dat_dictionary_key_dup(const struct return DAT_SUCCESS; } -/*********************************************************************** - * Function: dat_dictionary_key_is_equal - ***********************************************************************/ - -boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, - const struct dat_provider_info *key_b) +static boolean_t dat_dict_key_is_equal(const struct dat_provider_info *key_a, + const struct dat_provider_info *key_b) { BUG_ON(NULL == key_a); BUG_ON(NULL == key_b); @@ -288,9 +229,8 @@ boolean_t dat_dictionary_key_is_equal(co (!strncmp(key_a->ia_name, key_b->ia_name, strlen(key_a->ia_name))) && (key_a->dat_version_major == key_b->dat_version_major) && (key_a->dat_version_minor == key_b->dat_version_minor) - && (key_a->is_thread_safe == key_b->is_thread_safe)) { + && (key_a->is_thread_safe == key_b->is_thread_safe)) return TRUE; - } else { + else return FALSE; - } } Index: linux-kernel-new/dat/dr.c =================================================================== --- linux-kernel-new/dat/dr.c (revision 2518) +++ linux-kernel-new/dat/dr.c (working copy) @@ -35,47 +35,21 @@ #include "dictionary.h" #include "dr.h" -/********************************************************************* - * * - * Strucutres * - * * - *********************************************************************/ - struct dat_dr_entry { int ref_count; struct dat_provider_info info; DAT_IA_OPEN_FUNC ia_open_func; }; -/********************************************************************* - * * - * Global Variables * - * * - *********************************************************************/ - static spinlock_t g_dr_lock = SPIN_LOCK_UNLOCKED; static struct dat_dictionary g_dr_dictionary = DAT_DICTIONARY_INIT(g_dr_dictionary); -/********************************************************************* - * * - * External Functions * - * * - *********************************************************************/ - -/************************************************************************ - * Function: dat_dr_fini - ************************************************************************/ - void dat_dr_fini(void) { dat_dictionary_destroy(&g_dr_dictionary); } -/************************************************************************ - * Function: dat_dr_insert - ************************************************************************/ - u32 dat_dr_insert(const struct dat_provider_info *info, const DAT_IA_OPEN_FUNC ia_open_func) { @@ -107,7 +81,7 @@ u32 dat_dr_insert(const struct dat_provi spin_unlock_irqrestore(&g_dr_lock, flags); - bail: +bail: if (DAT_SUCCESS != status) { if (NULL != data) kfree(data); @@ -119,10 +93,6 @@ u32 dat_dr_insert(const struct dat_provi return status; } -/************************************************************************ - * Function: dat_dr_remove - ************************************************************************/ - u32 dat_dr_remove(const struct dat_provider_info *info) { struct dat_dr_entry *data; @@ -150,7 +120,7 @@ u32 dat_dr_remove(const struct dat_provi kfree(data); - bail: +bail: spin_unlock_irqrestore(&g_dr_lock, flags); if (NULL != dict_entry) @@ -159,10 +129,6 @@ u32 dat_dr_remove(const struct dat_provi return status; } -/************************************************************************ - * Function: dat_dr_provider_open - ************************************************************************/ - u32 dat_dr_provider_open(const struct dat_provider_info *info, DAT_IA_OPEN_FUNC *p_ia_open_func) { @@ -184,10 +150,6 @@ u32 dat_dr_provider_open(const struct da return status; } -/************************************************************************ - * Function: dat_dr_provider_close - ************************************************************************/ - u32 dat_dr_provider_close(const struct dat_provider_info *info) { struct dat_dr_entry *data; @@ -206,19 +168,11 @@ u32 dat_dr_provider_close(const struct d return status; } -/************************************************************************ - * Function: dat_dr_size - ************************************************************************/ - u32 dat_dr_size(int *size) { return dat_dictionary_size(&g_dr_dictionary, size); } -/*********************************************************************** - * Function: dat_dr_list - ***********************************************************************/ - u32 dat_dr_list(int max_to_return, int *entries_returned, struct dat_provider_info *(dat_provider_list[])) { @@ -227,9 +181,11 @@ u32 dat_dr_list(int max_to_return, int * unsigned long flags; u32 status = DAT_SUCCESS; - /* The dictionary size may increase between the call to */ - /* dat_dictionary_size() and dat_dictionary_enumerate(). */ - /* Therefore we loop until a successful enumeration is made. */ + /* + * The dictionary size may increase between the call to + * dat_dictionary_size() and dat_dictionary_enumerate(). + * Therefore we loop until a successful enumeration is made. + */ *entries_returned = 0; for (;;) { status = dat_dictionary_size(&g_dr_dictionary, &array_size); @@ -256,9 +212,9 @@ u32 dat_dr_list(int max_to_return, int * spin_unlock_irqrestore(&g_dr_lock, flags); - if (DAT_SUCCESS == status) { + if (DAT_SUCCESS == status) break; - } else { + else { kfree(array); array = NULL; continue; @@ -277,10 +233,9 @@ u32 dat_dr_list(int max_to_return, int * *entries_returned = i; - bail: - if (NULL != array) { +bail: + if (NULL != array) kfree(array); - } return status; } Index: linux-kernel-new/dat/core.c =================================================================== --- linux-kernel-new/dat/core.c (revision 2518) +++ linux-kernel-new/dat/core.c (working copy) @@ -38,26 +38,10 @@ MODULE_LICENSE("Dual BSD/GPL"); MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); MODULE_AUTHOR("James Lentini"); -/********************************************************************* - * * - * Global Variables * - * * - *********************************************************************/ - static DAT_DBG_MASK g_dbg_mask = DAT_DBG_TYPE_ERROR; module_param_named(dbg_mask, g_dbg_mask, int, 0644); MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug message types."); -/********************************************************************* - * * - * Function Definitions * - * * - *********************************************************************/ - -/*********************************************************************** - * Function: dat_dbg_print - ***********************************************************************/ - void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...) { static char buf[1024]; @@ -72,10 +56,6 @@ void dat_dbg_print(DAT_DBG_TYPE type, co } } -/*********************************************************************** - * Function: dat_init - ***********************************************************************/ - static int __init dat_init(void) { dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry started\n"); @@ -84,10 +64,6 @@ static int __init dat_init(void) module_init(dat_init); -/*********************************************************************** - * Function: dat_fini - ***********************************************************************/ - static void __exit dat_fini(void) { dat_dr_fini(); Index: linux-kernel-new/dat/api.c =================================================================== --- linux-kernel-new/dat/api.c (revision 2518) +++ linux-kernel-new/dat/api.c (working copy) @@ -34,547 +34,321 @@ #include "core.h" #include "dr.h" -/********************************************************************* - * * - * Internal Function Definitions * - * * - *********************************************************************/ - -/*********************************************************************** - * Function: dat_strerror_major - ***********************************************************************/ - static u32 dat_strerror_major(u32 value, const char **message) { switch (DAT_GET_TYPE(value)) { case DAT_SUCCESS: - { - *message = "DAT_SUCCESS"; - return DAT_SUCCESS; - } + *message = "DAT_SUCCESS"; + return DAT_SUCCESS; case DAT_ABORT: - { - *message = "DAT_ABORT"; - return DAT_SUCCESS; - } + *message = "DAT_ABORT"; + return DAT_SUCCESS; case DAT_CONN_QUAL_IN_USE: - { - *message = "DAT_CONN_QUAL_IN_USE"; - return DAT_SUCCESS; - } + *message = "DAT_CONN_QUAL_IN_USE"; + return DAT_SUCCESS; case DAT_INSUFFICIENT_RESOURCES: - { - *message = "DAT_INSUFFICIENT_RESOURCES"; - return DAT_SUCCESS; - } + *message = "DAT_INSUFFICIENT_RESOURCES"; + return DAT_SUCCESS; case DAT_INTERNAL_ERROR: - { - *message = "DAT_INTERNAL_ERROR"; - return DAT_SUCCESS; - } + *message = "DAT_INTERNAL_ERROR"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE: - { - *message = "DAT_INVALID_HANDLE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE"; + return DAT_SUCCESS; case DAT_INVALID_PARAMETER: - { - *message = "DAT_INVALID_PARAMETER"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_PARAMETER"; + return DAT_SUCCESS; case DAT_INVALID_STATE: - { - *message = "DAT_INVALID_STATE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE"; + return DAT_SUCCESS; case DAT_LENGTH_ERROR: - { - *message = "DAT_LENGTH_ERROR"; - return DAT_SUCCESS; - } + *message = "DAT_LENGTH_ERROR"; + return DAT_SUCCESS; case DAT_MODEL_NOT_SUPPORTED: - { - *message = "DAT_MODEL_NOT_SUPPORTED"; - return DAT_SUCCESS; - } + *message = "DAT_MODEL_NOT_SUPPORTED"; + return DAT_SUCCESS; case DAT_PROVIDER_NOT_FOUND: - { - *message = "DAT_PROVIDER_NOT_FOUND"; - return DAT_SUCCESS; - } + *message = "DAT_PROVIDER_NOT_FOUND"; + return DAT_SUCCESS; case DAT_PRIVILEGES_VIOLATION: - { - *message = "DAT_PRIVILEGES_VIOLATION"; - return DAT_SUCCESS; - } + *message = "DAT_PRIVILEGES_VIOLATION"; + return DAT_SUCCESS; case DAT_PROTECTION_VIOLATION: - { - *message = "DAT_PROTECTION_VIOLATION"; - return DAT_SUCCESS; - } + *message = "DAT_PROTECTION_VIOLATION"; + return DAT_SUCCESS; case DAT_QUEUE_EMPTY: - { - *message = "DAT_QUEUE_EMPTY"; - return DAT_SUCCESS; - } + *message = "DAT_QUEUE_EMPTY"; + return DAT_SUCCESS; case DAT_QUEUE_FULL: - { - *message = "DAT_QUEUE_FULL"; - return DAT_SUCCESS; - } + *message = "DAT_QUEUE_FULL"; + return DAT_SUCCESS; case DAT_TIMEOUT_EXPIRED: - { - *message = "DAT_TIMEOUT_EXPIRED"; - return DAT_SUCCESS; - } + *message = "DAT_TIMEOUT_EXPIRED"; + return DAT_SUCCESS; case DAT_PROVIDER_ALREADY_REGISTERED: - { - *message = "DAT_PROVIDER_ALREADY_REGISTERED"; - return DAT_SUCCESS; - } + *message = "DAT_PROVIDER_ALREADY_REGISTERED"; + return DAT_SUCCESS; case DAT_PROVIDER_IN_USE: - { - *message = "DAT_PROVIDER_IN_USE"; - return DAT_SUCCESS; - } + *message = "DAT_PROVIDER_IN_USE"; + return DAT_SUCCESS; case DAT_INVALID_ADDRESS: - { - *message = "DAT_INVALID_ADDRESS"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ADDRESS"; + return DAT_SUCCESS; case DAT_INTERRUPTED_CALL: - { - *message = "DAT_INTERRUPTED_CALL"; - return DAT_SUCCESS; - } + *message = "DAT_INTERRUPTED_CALL"; + return DAT_SUCCESS; case DAT_NOT_IMPLEMENTED: - { - *message = "DAT_NOT_IMPLEMENTED"; - return DAT_SUCCESS; - } + *message = "DAT_NOT_IMPLEMENTED"; + return DAT_SUCCESS; default: - { - *message = "unknown error"; - return DAT_INVALID_PARAMETER; - } + *message = "unknown error"; + return DAT_INVALID_PARAMETER; } } -/*********************************************************************** - * Function: dat_strerror_minor - ***********************************************************************/ static u32 dat_strerror_minor(u32 value, const char **message) { switch (DAT_GET_SUBTYPE(value)) { - case DAT_NO_SUBTYPE: - { - *message = ""; - return DAT_SUCCESS; - } + *message = ""; + return DAT_SUCCESS; case DAT_SUB_INTERRUPTED: - { - *message = "DAT_SUB_INTERRUPTED"; - return DAT_SUCCESS; - } + *message = "DAT_SUB_INTERRUPTED"; + return DAT_SUCCESS; case DAT_RESOURCE_MEMORY: - { - *message = "DAT_RESOURCE_MEMORY"; - return DAT_SUCCESS; - } + *message = "DAT_RESOURCE_MEMORY"; + return DAT_SUCCESS; case DAT_RESOURCE_DEVICE: - { - *message = "DAT_RESOURCE_DEVICE"; - return DAT_SUCCESS; - } + *message = "DAT_RESOURCE_DEVICE"; + return DAT_SUCCESS; case DAT_RESOURCE_TEP: - { - *message = "DAT_RESOURCE_TEP"; - return DAT_SUCCESS; - } + *message = "DAT_RESOURCE_TEP"; + return DAT_SUCCESS; case DAT_RESOURCE_TEVD: - { - *message = "DAT_RESOURCE_TEVD"; - return DAT_SUCCESS; - } + *message = "DAT_RESOURCE_TEVD"; + return DAT_SUCCESS; case DAT_RESOURCE_PROTECTION_DOMAIN: - { - *message = "DAT_RESOURCE_PROTECTION_DOMAIN"; - return DAT_SUCCESS; - } + *message = "DAT_RESOURCE_PROTECTION_DOMAIN"; + return DAT_SUCCESS; case DAT_RESOURCE_MEMORY_REGION: - { - *message = "DAT_RESOURCE_MEMORY_REGION"; - return DAT_SUCCESS; - } + *message = "DAT_RESOURCE_MEMORY_REGION"; + return DAT_SUCCESS; case DAT_RESOURCE_ERROR_HANDLER: - { - *message = "DAT_RESOURCE_ERROR_HANDLER"; - return DAT_SUCCESS; - } + *message = "DAT_RESOURCE_ERROR_HANDLER"; + return DAT_SUCCESS; case DAT_RESOURCE_CREDITS: - { - *message = "DAT_RESOURCE_CREDITS"; - return DAT_SUCCESS; - } + *message = "DAT_RESOURCE_CREDITS"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_IA: - { - *message = "DAT_INVALID_HANDLE_IA"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_IA"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_EP: - { - *message = "DAT_INVALID_HANDLE_EP"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_EP"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_LMR: - { - *message = "DAT_INVALID_HANDLE_LMR"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_LMR"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_RMR: - { - *message = "DAT_INVALID_HANDLE_RMR"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_RMR"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_PZ: - { - *message = "DAT_INVALID_HANDLE_PZ"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_PZ"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_PSP: - { - *message = "DAT_INVALID_HANDLE_PSP"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_PSP"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_RSP: - { - *message = "DAT_INVALID_HANDLE_RSP"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_RSP"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_CR: - { - *message = "DAT_INVALID_HANDLE_CR"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_CR"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_CNO: - { - *message = "DAT_INVALID_HANDLE_CNO"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_CNO"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_EVD_CR: - { - *message = "DAT_INVALID_HANDLE_EVD_CR"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_EVD_CR"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_EVD_REQUEST: - { - *message = "DAT_INVALID_HANDLE_EVD_REQUEST"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_EVD_REQUEST"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_EVD_RECV: - { - *message = "DAT_INVALID_HANDLE_EVD_RECV"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_EVD_RECV"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_EVD_CONN: - { - *message = "DAT_INVALID_HANDLE_EVD_CONN"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_EVD_CONN"; + return DAT_SUCCESS; case DAT_INVALID_HANDLE_EVD_ASYNC: - { - *message = "DAT_INVALID_HANDLE_EVD_ASYNC"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_HANDLE_EVD_ASYNC"; + return DAT_SUCCESS; case DAT_INVALID_ARG1: - { - *message = "DAT_INVALID_ARG1"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG1"; + return DAT_SUCCESS; case DAT_INVALID_ARG2: - { - *message = "DAT_INVALID_ARG2"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG2"; + return DAT_SUCCESS; case DAT_INVALID_ARG3: - { - *message = "DAT_INVALID_ARG3"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG3"; + return DAT_SUCCESS; case DAT_INVALID_ARG4: - { - *message = "DAT_INVALID_ARG4"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG4"; + return DAT_SUCCESS; case DAT_INVALID_ARG5: - { - *message = "DAT_INVALID_ARG5"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG5"; + return DAT_SUCCESS; case DAT_INVALID_ARG6: - { - *message = "DAT_INVALID_ARG6"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG6"; + return DAT_SUCCESS; case DAT_INVALID_ARG7: - { - *message = "DAT_INVALID_ARG7"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG7"; + return DAT_SUCCESS; case DAT_INVALID_ARG8: - { - *message = "DAT_INVALID_ARG8"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG8"; + return DAT_SUCCESS; case DAT_INVALID_ARG9: - { - *message = "DAT_INVALID_ARG9"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG9"; + return DAT_SUCCESS; case DAT_INVALID_ARG10: - { - *message = "DAT_INVALID_ARG10"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ARG10"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_UNCONNECTED: - { - *message = "DAT_INVALID_STATE_EP_UNCONNECTED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_UNCONNECTED"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_ACTCONNPENDING: - { - *message = "DAT_INVALID_STATE_EP_ACTCONNPENDING"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_ACTCONNPENDING"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_PASSCONNPENDING: - { - *message = "DAT_INVALID_STATE_EP_PASSCONNPENDING"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_PASSCONNPENDING"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_TENTCONNPENDING: - { - *message = "DAT_INVALID_STATE_EP_TENTCONNPENDING"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_TENTCONNPENDING"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_CONNECTED: - { - *message = "DAT_INVALID_STATE_EP_CONNECTED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_CONNECTED"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_DISCONNECTED: - { - *message = "DAT_INVALID_STATE_EP_DISCONNECTED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_DISCONNECTED"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_RESERVED: - { - *message = "DAT_INVALID_STATE_EP_RESERVED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_RESERVED"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_COMPLPENDING: - { - *message = "DAT_INVALID_STATE_EP_COMPLPENDING"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_COMPLPENDING"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_DISCPENDING: - { - *message = "DAT_INVALID_STATE_EP_DISCPENDING"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_DISCPENDING"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_PROVIDERCONTROL: - { - *message = "DAT_INVALID_STATE_EP_PROVIDERCONTROL"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_PROVIDERCONTROL"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EP_NOTREADY: - { - *message = "DAT_INVALID_STATE_EP_NOTREADY"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EP_NOTREADY"; + return DAT_SUCCESS; case DAT_INVALID_STATE_CNO_IN_USE: - { - *message = "DAT_INVALID_STATE_CNO_IN_USE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_CNO_IN_USE"; + return DAT_SUCCESS; case DAT_INVALID_STATE_CNO_DEAD: - { - *message = "DAT_INVALID_STATE_CNO_DEAD"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_CNO_DEAD"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_OPEN: - { - *message = "DAT_INVALID_STATE_EVD_OPEN"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_OPEN"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_ENABLED: - { - *message = "DAT_INVALID_STATE_EVD_ENABLED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_ENABLED"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_DISABLED: - { - *message = "DAT_INVALID_STATE_EVD_DISABLED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_DISABLED"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_WAITABLE: - { - *message = "DAT_INVALID_STATE_EVD_WAITABLE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_WAITABLE"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_UNWAITABLE: - { - *message = "DAT_INVALID_STATE_EVD_UNWAITABLE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_UNWAITABLE"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_IN_USE: - { - *message = "DAT_INVALID_STATE_EVD_IN_USE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_IN_USE"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_CONFIG_NOTIFY: - { - *message = "DAT_INVALID_STATE_EVD_CONFIG_NOTIFY"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_CONFIG_NOTIFY"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_CONFIG_SOLICITED: - { - *message = "DAT_INVALID_STATE_EVD_CONFIG_SOLICITED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_CONFIG_SOLICITED"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_CONFIG_THRESHOLD: - { - *message = "DAT_INVALID_STATE_EVD_CONFIG_THRESHOLD"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_CONFIG_THRESHOLD"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_WAITER: - { - *message = "DAT_INVALID_STATE_EVD_WAITER"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_WAITER"; + return DAT_SUCCESS; case DAT_INVALID_STATE_EVD_ASYNC: - { - *message = "DAT_INVALID_STATE_EVD_ASYNC"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_EVD_ASYNC"; + return DAT_SUCCESS; case DAT_INVALID_STATE_IA_IN_USE: - { - *message = "DAT_INVALID_STATE_IA_IN_USE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_IA_IN_USE"; + return DAT_SUCCESS; case DAT_INVALID_STATE_LMR_IN_USE: - { - *message = "DAT_INVALID_STATE_LMR_IN_USE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_LMR_IN_USE"; + return DAT_SUCCESS; case DAT_INVALID_STATE_LMR_FREE: - { - *message = "DAT_INVALID_STATE_LMR_FREE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_LMR_FREE"; + return DAT_SUCCESS; case DAT_INVALID_STATE_PZ_IN_USE: - { - *message = "DAT_INVALID_STATE_PZ_IN_USE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_PZ_IN_USE"; + return DAT_SUCCESS; case DAT_INVALID_STATE_PZ_FREE: - { - *message = "DAT_INVALID_STATE_PZ_FREE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_STATE_PZ_FREE"; + return DAT_SUCCESS; case DAT_PRIVILEGES_READ: - { - *message = "DAT_PRIVILEGES_READ"; - return DAT_SUCCESS; - } + *message = "DAT_PRIVILEGES_READ"; + return DAT_SUCCESS; case DAT_PRIVILEGES_WRITE: - { - *message = "DAT_PRIVILEGES_WRITE"; - return DAT_SUCCESS; - } + *message = "DAT_PRIVILEGES_WRITE"; + return DAT_SUCCESS; case DAT_PRIVILEGES_RDMA_READ: - { - *message = "DAT_PRIVILEGES_RDMA_READ"; - return DAT_SUCCESS; - } + *message = "DAT_PRIVILEGES_RDMA_READ"; + return DAT_SUCCESS; case DAT_PRIVILEGES_RDMA_WRITE: - { - *message = "DAT_PRIVILEGES_RDMA_WRITE"; - return DAT_SUCCESS; - } + *message = "DAT_PRIVILEGES_RDMA_WRITE"; + return DAT_SUCCESS; case DAT_PROTECTION_READ: - { - *message = "DAT_PROTECTION_READ"; - return DAT_SUCCESS; - } + *message = "DAT_PROTECTION_READ"; + return DAT_SUCCESS; case DAT_PROTECTION_WRITE: - { - *message = "DAT_PROTECTION_WRITE"; - return DAT_SUCCESS; - } + *message = "DAT_PROTECTION_WRITE"; + return DAT_SUCCESS; case DAT_PROTECTION_RDMA_READ: - { - *message = "DAT_PROTECTION_RDMA_READ"; - return DAT_SUCCESS; - } + *message = "DAT_PROTECTION_RDMA_READ"; + return DAT_SUCCESS; case DAT_PROTECTION_RDMA_WRITE: - { - *message = "DAT_PROTECTION_RDMA_WRITE"; - return DAT_SUCCESS; - } + *message = "DAT_PROTECTION_RDMA_WRITE"; + return DAT_SUCCESS; case DAT_INVALID_ADDRESS_UNSUPPORTED: - { - *message = "DAT_INVALID_ADDRESS_UNSUPPORTED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ADDRESS_UNSUPPORTED"; + return DAT_SUCCESS; case DAT_INVALID_ADDRESS_UNREACHABLE: - { - *message = "DAT_INVALID_ADDRESS_UNREACHABLE"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ADDRESS_UNREACHABLE"; + return DAT_SUCCESS; case DAT_INVALID_ADDRESS_MALFORMED: - { - *message = "DAT_INVALID_ADDRESS_MALFORMED"; - return DAT_SUCCESS; - } + *message = "DAT_INVALID_ADDRESS_MALFORMED"; + return DAT_SUCCESS; case DAT_NAME_NOT_REGISTERED: - { - *message = "DAT_NAME_NOT_REGISTERED"; - return DAT_SUCCESS; - } + *message = "DAT_NAME_NOT_REGISTERED"; + return DAT_SUCCESS; case DAT_MAJOR_NOT_FOUND: - { - *message = "DAT_MAJOR_NOT_FOUND"; - return DAT_SUCCESS; - } + *message = "DAT_MAJOR_NOT_FOUND"; + return DAT_SUCCESS; case DAT_MINOR_NOT_FOUND: - { - *message = "DAT_MINOR_NOT_FOUND"; - return DAT_SUCCESS; - } + *message = "DAT_MINOR_NOT_FOUND"; + return DAT_SUCCESS; case DAT_THREAD_SAFETY_NOT_FOUND: - { - *message = "DAT_THREAD_SAFETY_NOT_FOUND"; - return DAT_SUCCESS; - } + *message = "DAT_THREAD_SAFETY_NOT_FOUND"; + return DAT_SUCCESS; default: - { - *message = "unknown minor error"; - return DAT_INVALID_PARAMETER; - } + *message = "unknown minor error"; + return DAT_INVALID_PARAMETER; } } -/********************************************************************* - * * - * External Function Definitions * - * * - *********************************************************************/ - -/*********************************************************************** - * Function: dat_ia_openv - ***********************************************************************/ - u32 dat_ia_openv(const char *name, int async_event_qlen, DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, boolean_t thread_safety) @@ -589,15 +363,13 @@ u32 dat_ia_openv(const char *name, int a __func__, name, async_event_qlen, dat_major, dat_minor, thread_safety); - if (DAT_IS_BAD_POINTER(name)) { + if (DAT_IS_BAD_POINTER(name)) return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); - } len = strlen(name); - if (DAT_NAME_MAX_LENGTH < len) { + if (DAT_NAME_MAX_LENGTH < len) return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); - } strncpy(info.ia_name, name, len); info.ia_name[len] = '\0'; @@ -615,16 +387,12 @@ u32 dat_ia_openv(const char *name, int a return status; } - return ia_open_func( - name, async_event_qlen, async_event_handle, ia_handle); + return ia_open_func(name, async_event_qlen, async_event_handle, + ia_handle); } EXPORT_SYMBOL(dat_ia_openv); -/*********************************************************************** - * Function: dat_ia_close - ***********************************************************************/ - u32 dat_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags flags) { struct dat_provider *provider; @@ -638,15 +406,15 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle provider = DAT_HANDLE_TO_PROVIDER(ia_handle); ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query( - ia_handle, NULL, NULL, &provider_attr))) { + if (DAT_SUCCESS != (status = dat_ia_query(ia_handle, NULL, NULL, + &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); - } else if (DAT_SUCCESS != - (status = provider->ia_close_func(ia_handle, flags))) { + else if (DAT_SUCCESS != (status = provider->ia_close_func(ia_handle, + flags))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "close function for IA %s failed\n", ia_name); - } else { + else { struct dat_provider_info info; size_t len; @@ -662,12 +430,11 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle info.is_thread_safe = provider_attr.is_thread_safe; status = dat_dr_provider_close(&info); - if (DAT_SUCCESS != status) { + if (DAT_SUCCESS != status) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dynamic registry unable to close " "provider for IA %s\n", ia_name); - } } return status; @@ -675,10 +442,6 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle EXPORT_SYMBOL(dat_ia_close); -/*********************************************************************** - * Function: dat_strerror - ***********************************************************************/ - u32 dat_strerror(u32 value, const char **major_message, const char **minor_message) { @@ -695,23 +458,17 @@ u32 dat_strerror(u32 value, const char * * implementation does not follow the suggested implementation. */ - if (DAT_SUCCESS != dat_strerror_major(value, major_message)) { + if (DAT_SUCCESS != dat_strerror_major(value, major_message)) return DAT_INVALID_PARAMETER; - } else if (minor_message != NULL) { - if (DAT_SUCCESS != dat_strerror_minor(value, minor_message)) { + else if (minor_message != NULL) + if (DAT_SUCCESS != dat_strerror_minor(value, minor_message)) return DAT_INVALID_PARAMETER; - } - } return DAT_SUCCESS; } EXPORT_SYMBOL(dat_strerror); -/*********************************************************************** - * Function: dat_registry_add_provider - ***********************************************************************/ - u32 dat_registry_add_provider(const struct dat_provider *provider, const struct dat_provider_info *provider_info) { @@ -733,10 +490,6 @@ u32 dat_registry_add_provider(const stru EXPORT_SYMBOL(dat_registry_remove_provider); -/*********************************************************************** - * Function: dat_registry_remove_provider - ***********************************************************************/ - u32 dat_registry_remove_provider(const struct dat_provider *provider, const struct dat_provider_info *provider_info) { @@ -751,10 +504,6 @@ u32 dat_registry_remove_provider(const s EXPORT_SYMBOL(dat_registry_add_provider); -/*********************************************************************** - * Function: dat_registry_list_providers - ***********************************************************************/ - u32 dat_registry_list_providers(int max_to_return, int *entries_returned, struct dat_provider_info *(dat_provider_list[])) { @@ -779,10 +528,9 @@ u32 dat_registry_list_providers(int max_ */ (void)dat_dr_size(entries_returned); return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); - } else { + } else status = dat_dr_list(max_to_return, entries_returned, dat_provider_list); - } return status; } Index: linux-kernel-new/dat/dr.h =================================================================== --- linux-kernel-new/dat/dr.h (revision 2518) +++ linux-kernel-new/dat/dr.h (working copy) @@ -36,12 +36,6 @@ #include -/********************************************************************* - * * - * Function Declarations * - * * - *********************************************************************/ - extern void dat_dr_fini(void); extern u32 dat_dr_insert(const struct dat_provider_info *info, From tduffy at sun.com Wed Jun 1 11:34:36 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 11:34:36 -0700 Subject: [openib-general] Send Side RMPP and OpenSM GetTableResp In-Reply-To: <1117480184.4476.110.camel@hal.voltaire.com> References: <1117480184.4476.110.camel@hal.voltaire.com> Message-ID: <1117650876.29566.10.camel@duffman> On Mon, 2005-05-30 at 15:09 -0400, Hal Rosenstock wrote: > Hi Sean, > > When running a Solaris 10 client against OpenSM with RMPP supported > turned on, I see the following sequence: BTW, Hal, I just saw these fixes checked into Solaris Nevada (11) today: 6246111 IBCM confuses a retried REQ for a stale REQ 6247310 IBCM: Third Party SM can return different error code compared to IBSRM 6265425 crload test using abort and/or recycle exposes bugs in ibcm Do you think any of these bugs could be contributing to what you are seeing? It may be a good idea to get you on the Solaris Express train or OpenSolaris.org. I will be hooking up some Nevada systems in my testbed RSN. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From hch at lst.de Wed Jun 1 11:35:44 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 1 Jun 2005 20:35:44 +0200 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <1117650224.29566.6.camel@duffman> References: <1117650224.29566.6.camel@duffman> Message-ID: <20050601183544.GA24939@lst.de> On Wed, Jun 01, 2005 at 11:23:44AM -0700, Tom Duffy wrote: > This patch cleans up dat/ a bit, removing unneeded parenthesis, > gratuitous comments, and a few whitespace cleanups. Please apply. btw, I think most of the code under dat/ should just go away ASAP. Just have a simple linked list of providers with trivial registration/unregisteration/find_get/put functions like all the other subsystems and kill all the dictionaty mess and provider vs provider_info braindamage. From halr at voltaire.com Wed Jun 1 11:49:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 01 Jun 2005 14:49:02 -0400 Subject: [openib-general] Send Side RMPP and OpenSM GetTableResp In-Reply-To: <1117650876.29566.10.camel@duffman> References: <1117480184.4476.110.camel@hal.voltaire.com> <1117650876.29566.10.camel@duffman> Message-ID: <1117651742.4490.29.camel@hal.voltaire.com> Hi Tom, On Wed, 2005-06-01 at 14:34, Tom Duffy wrote: > BTW, Hal, I just saw these fixes checked into Solaris Nevada (11) today: > > 6246111 IBCM confuses a retried REQ for a stale REQ This could come into play on SDP or kdapl interoperation with S10. > 6247310 IBCM: Third Party SM can return different error code compared to IBSRM The juxtaposition of CM and SM is confusing to me so I'm not sure exactly what this is. Perhaps this is something to do with SA client interaction with CM. This is not related to RMPP. > 6265425 crload test using abort and/or recycle exposes bugs in ibcm Is crload an Oracle test ? > Do you think any of these bugs could be contributing to what you are > seeing? Doubtful. > It may be a good idea to get you on the Solaris Express train > or OpenSolaris.org. OK. > I will be hooking up some Nevada systems in my testbed RSN. Glad to hear this. I will be interested in what the S10 SA client thinks is going on in terms of RMPP. -- Hal From tduffy at sun.com Wed Jun 1 12:06:42 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 12:06:42 -0700 Subject: [openib-general] Send Side RMPP and OpenSM GetTableResp In-Reply-To: <1117651742.4490.29.camel@hal.voltaire.com> References: <1117480184.4476.110.camel@hal.voltaire.com> <1117650876.29566.10.camel@duffman> <1117651742.4490.29.camel@hal.voltaire.com> Message-ID: <1117652802.29566.16.camel@duffman> On Wed, 2005-06-01 at 14:49 -0400, Hal Rosenstock wrote: > Hi Tom, > > On Wed, 2005-06-01 at 14:34, Tom Duffy wrote: > > BTW, Hal, I just saw these fixes checked into Solaris Nevada (11) today: > > > > 6246111 IBCM confuses a retried REQ for a stale REQ > > This could come into play on SDP or kdapl interoperation with S10. Further info on this: During interoperability testing with Voltaire SM, IBCM timeout tests exposed a bug in IBCM. Voltaire SM has a high PacketLifeTime (1 sec), this causes IBCM to retry much slowly. Due to the delayed retries the receiving IBCM thinks the retried REQ as a stale REQ and fails the connection. > > 6247310 IBCM: Third Party SM can return different error code compared to IBSRM > > The juxtaposition of CM and SM is confusing to me so I'm not sure > exactly what this is. Perhaps this is something to do with SA client > interaction with CM. This is not related to RMPP. Again, from the bug report: IB Specification does not dictate which is the correct "Error Code" returned by Subnet Manager if a PathLook-up from 'A' to 'B' is Not available. It was left to implementors interpretation from the list of error codes available. IBSRM - return IBMF_SUCCESS with Zero Records, while 3rd Party SM (TopSpin, Voltaire) returns - IBMF_NO_RECORDS/IBMF_REQ_INVALID > > 6265425 crload test using abort and/or recycle exposes bugs in ibcm > > Is crload an Oracle test ? Yes. It was doing something along the lines of: post recv buffer to ep ep_connect ... /* before connect completes */ ep_disconnect > > Do you think any of these bugs could be contributing to what you are > > seeing? > > Doubtful. Good. So, I should see the same behavior on Solaris 10 or 11. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From itamar at mellanox.co.il Wed Jun 1 12:46:58 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 1 Jun 2005 22:46:58 +0300 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more Message-ID: <91DB792C7985D411BEC300B40080D29CC35C15@mtvex01.mtv.mtl.com> I think that if we want to support pci hot plug we need to change the kdapl IF to be like something like the gen2 IF. a link lins of the providers with register/unregister client functions which will register a CB function. we only need to have 4 functions register_client , unregister_client register_provider , unregister_provider Itamar > -----Original Message----- > From: Christoph Hellwig [mailto:hch at lst.de] > Sent: Wednesday, June 01, 2005 9:36 PM > To: Tom Duffy > Cc: openib-general at openib.org > Subject: Re: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more > > > On Wed, Jun 01, 2005 at 11:23:44AM -0700, Tom Duffy wrote: > > This patch cleans up dat/ a bit, removing unneeded parenthesis, > > gratuitous comments, and a few whitespace cleanups. Please apply. > > btw, I think most of the code under dat/ should just go away ASAP. > Just have a simple linked list of providers with trivial > registration/unregisteration/find_get/put functions like all the other > subsystems and kill all the dictionaty mess and provider vs > provider_info braindamage. > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From tduffy at sun.com Wed Jun 1 12:40:31 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 12:40:31 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117576362.1373.16.camel@duffman> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1117576362.1373.16.camel@duffman> Message-ID: <1117654831.29566.21.camel@duffman> On Tue, 2005-05-31 at 14:52 -0700, Tom Duffy wrote: > On Tue, 2005-05-31 at 14:17 -0400, James Lentini wrote: > > Here's the specification's exact description: > > > > timeout: Duration of time, in microseconds, that a consumer waits for > > connection establishment. The value of DAT_TIMEOUT_INFINITE > > represents no timeout, indefinite wait. Values must be > > positive. > > Let me make sure I got this right: timeout is in µs (10^-6 seconds), not > ms (10^-3 seconds). If so, I am off by 3 orders of magnitude in my > calculation. Right? This is from DT_fft_connect() in test/dapltest/test/dapl_fft_util.c: /* attempt to connect, timeout = 10 secs */ rc = dat_ep_connect (conn->ep_handle, conn->remote_netaddr, SERVER_PORT_NUMBER, 10*1000, 0, (void *)0, DAT_QOS_BEST_EFFORT, DAT_CONNECT_DEFAULT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); leading me to believe we are talking about milliseconds. Or is this a bug? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Wed Jun 1 12:48:44 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 12:48:44 -0700 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: Message-ID: <1117655324.19908.3.camel@duffman> This patch takes into account some of the feedback by Sean, it sets the retry to 0, so it will do the timeout right. It also stops the conversion of ms to ib at 63 since 64 overflows. I have tested this version using kdapltest in quit, performance, and transaction modes. James, do you think it good enough to apply? Signed-off-by: Tom Duffy Index: linux-kernel-timer/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (revision 2518) +++ linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (working copy) @@ -538,7 +538,7 @@ DT_performance_wait ( pre_ts = DT_Mdep_GetTimeStamp (); ret = DT_Tdep_evd_wait ( evd_handle, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: linux-kernel-timer/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (revision 2518) +++ linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (working copy) @@ -105,7 +105,7 @@ retry: ret = dat_ep_connect (test_ptr->ep_context.ep_handle, test_ptr->remote_ia_addr, test_ptr->ep_context.port, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, 0, (void *) 0, /* no private data */ test_ptr->cmd->qos, Index: linux-kernel-timer/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_server.c (revision 2518) +++ linux-kernel-timer/test/dapltest/test/dapl_server.c (working copy) @@ -580,7 +580,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Mdep_wait_object_wait ( &pt_ptr->synch_wait_object, - DAT_TIMEOUT_INFINITE); + MAX_SCHEDULE_TIMEOUT); /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); Index: linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (revision 2518) +++ linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -712,7 +712,7 @@ retry: ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, test_ptr->remote_ia_addr, test_ptr->ep_context[i].ia_port, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, strlen (private_data_str), private_data_str, pt_ptr->Params.ReliabilityLevel, DAT_CONNECT_DEFAULT_FLAG); Index: linux-kernel-timer/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_test_util.c (revision 2518) +++ linux-kernel-timer/test/dapltest/test/dapl_test_util.c (working copy) @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -369,7 +369,7 @@ DT_disco_event_wait ( DT_Tdep_Print_Head u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", Index: linux-kernel-timer/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_client.c (revision 2518) +++ linux-kernel-timer/test/dapltest/test/dapl_client.c (working copy) @@ -256,7 +256,7 @@ retry: ret = dat_ep_connect (ep_handle, server_netaddr, SERVER_PORT_NUMBER, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, 0, (void *) 0, /* no private data */ params_ptr->ReliabilityLevel, DAT_CONNECT_DEFAULT_FLAG); Index: linux-kernel-timer/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_tdep.h (revision 2518) +++ linux-kernel-timer/test/dapltest/include/dapl_tdep.h (working copy) @@ -56,7 +56,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_ha u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, - DAT_TIMEOUT timeout, + unsigned long timeout, struct dat_event *event); u32 DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, Index: linux-kernel-timer/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_proto.h (revision 2518) +++ linux-kernel-timer/test/dapltest/include/dapl_proto.h (working copy) @@ -185,7 +185,7 @@ void DT_Mdep_Thread_EXIT ( void * thr int DT_Mdep_wait_object_init ( DT_WAIT_OBJECT *wait_obj); int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val); + unsigned long timeout_val); int DT_Mdep_wait_object_wakeup ( DT_WAIT_OBJECT *wait_obj); int DT_Mdep_wait_object_destroy ( DT_WAIT_OBJECT *wait_obj); Index: linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (revision 2518) +++ linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -79,7 +79,7 @@ typedef struct Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; - DAT_TIMEOUT time_out; + unsigned long time_out; int evd_length; Thread *thread; Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c =================================================================== --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2518) +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) @@ -411,7 +411,7 @@ DT_Mdep_wait_object_init ( int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val) + unsigned long timeout_val) { int dat_status; int pthread_status; Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c =================================================================== --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2518) +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) @@ -252,13 +252,13 @@ DT_Mdep_wait_object_init ( int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val) + unsigned long timeout_val) { int expire; int dat_status; dat_status = DAT_SUCCESS; - if ( DAT_TIMEOUT_INFINITE == timeout_val) + if ( MAX_SCHEDULE_TIMEOUT == timeout_val) { interruptible_sleep_on (wait_obj); } Index: linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2518) +++ linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -213,7 +213,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, - DAT_TIMEOUT timeout, + unsigned long timeout, struct dat_event *dat_event) { u32 dat_status; @@ -239,7 +239,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han return dat_status; } /* wait for an event */ - if (timeout == DAT_TIMEOUT_INFINITE) + if (timeout == MAX_SCHEDULE_TIMEOUT) { rc = wait_event_interruptible (evd_ptr->wait_object, evd_ptr->event_next != NULL); Index: linux-kernel-timer/dat-provider/dapl_ep_connect.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_connect.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_ep_connect.c (working copy) @@ -35,7 +35,6 @@ #include "dapl_ep_util.h" #include "dapl_openib_util.h" #include "dapl_evd_util.h" -#include "dapl_timer_util.h" /* * Request a connection be established between the local Endpoint @@ -44,7 +43,7 @@ */ u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, + DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags connect_flags) { @@ -277,23 +276,13 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); if (ep_ptr->param.ep_state == DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - && timeout != DAT_TIMEOUT_INFINITE) { - ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, - GFP_ATOMIC); - if (!ep_ptr->cxn_timer) { - dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - - dapl_timer_set(ep_ptr->cxn_timer, - dapl_ep_timeout, ep_ptr, timeout); - } + && timeout != MAX_SCHEDULE_TIMEOUT) + ep_ptr->timeout = timeout; spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); } - bail: +bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, "dapl_ep_connect () returns 0x%x\n", dat_status); Index: linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_evd_util.h" #include "dapl_ep_util.h" -#include "dapl_timer_util.h" /* * dapl_evd_connection_callback @@ -87,15 +86,6 @@ void dapl_evd_connection_callback(struct */ spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - /* - * If a connection timer has been set up on this EP, cancel it now - */ - if (ep_ptr->cxn_timer != NULL) { - dapl_timer_cancel(ep_ptr->cxn_timer); - kfree(ep_ptr->cxn_timer); - ep_ptr->cxn_timer = NULL; - } - switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: { Index: linux-kernel-timer/dat-provider/dapl_ep_free.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_free.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_ep_free.c (working copy) @@ -34,7 +34,6 @@ #include "dapl_ep_util.h" #include "dapl_openib_util.h" #include "dapl_ring_buffer_util.h" -#include "dapl_timer_util.h" /* * dapl_ep_free @@ -107,12 +106,6 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->cxn_timer != NULL) { - dapl_timer_cancel(ep_ptr->cxn_timer); - kfree(ep_ptr->cxn_timer); - ep_ptr->cxn_timer = NULL; - } - /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); Index: linux-kernel-timer/dat-provider/dapl_provider.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_provider.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_provider.c (working copy) @@ -39,7 +39,6 @@ #include "dapl_mr_util.h" #include "dapl_util.h" #include "dapl_openib_util.h" -#include "dapl_timer_util.h" MODULE_LICENSE("Dual BSD/GPL"); MODULE_DESCRIPTION("DAT Provider for InfiniBand"); @@ -393,7 +392,6 @@ module_exit(dapl_fini); static int __init dapl_init(void) { dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "provider started\n"); - dapl_timer_init(); return ib_register_client(&dapl_client); } Index: linux-kernel-timer/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_openib_cm.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_openib_cm.c (working copy) @@ -291,6 +291,26 @@ static int dapl_cm_passive_cb_handler(st return ret; } +/* + * approximately transforms miliseconds to 4.096us*2^x + * 63(+8) is max return + */ +static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) { + unsigned long converged = 2; + u8 i; + + if (2 > ms) + return 8; + + for (i = 1; i < 63; i++) { + if (converged >= ms) + break; + converged = 2*converged; + } + + return i+8; +} + static void dapl_path_comp_handler(u64 req_id, void *context, int rec_num) { struct dapl_cm_id *conn = context; @@ -324,6 +344,12 @@ static void dapl_path_comp_handler(u64 r if (conn->dapl_path.mtu > IB_MTU_1024) conn->dapl_path.mtu = IB_MTU_1024; + if (conn->ep && conn->ep->timeout != 0) { + conn->param.remote_cm_response_timeout = + dapl_convert_ms_to_kookyib(conn->ep->timeout/1000); + conn->param.retry_count = 0; + } + conn->param.primary_path = &conn->dapl_path; conn->param.alternate_path = NULL; Index: linux-kernel-timer/dat-provider/dapl_timer_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_timer_util.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_timer_util.c (working copy) @@ -1,313 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Routines to add and cancel timer records. A timer record - * is put on the global timer queue. If the timer thread is - * not running, start it. The timer thread will sleep - * until a timer event or until a process wakes it up - * to notice a new timer is available; we use a DAPL_WAIT_OBJ - * for synchronization. - * - * If a timer is cancelled, it is simlpy removed from the - * queue. The timer may wakeup and notice there is no timer - * record to awaken at this time, so it will reset for the - * next entry. When there are no timer records to manage, - * the timer thread just sleeps until awakened. - * - * This file also contains the timer handler thread, - * embodied in dapl_timer_thread(). - * - * $Id$ - */ - -#include "dapl.h" -#include "dapl_timer_util.h" - -struct dapl_timer_head { - struct dapl_llist_entry *timer_list_head; - spinlock_t lock; - unsigned long flags; - DAPL_OS_WAIT_OBJECT wait_object; - int thread_id; -}; - -static struct dapl_timer_head g_dapl_timer_head; - -static int dapl_timer_thread(void *arg); - -void dapl_timer_init(void) -{ - /* - * Set up the timer thread elements. The timer thread isn't - * started until it is actually needed - */ - g_dapl_timer_head.timer_list_head = NULL; - spin_lock_init(&g_dapl_timer_head.lock); - dapl_os_wait_object_init(&g_dapl_timer_head.wait_object); - g_dapl_timer_head.thread_id = -1; -} - -/* - * dapl_timer_set - * - * Set a timer. The timer will invoke the specified function - * after a number of useconds expires. - * - * Input: - * timer User provided timer structure - * func Function to invoke when timer expires - * data Argument passed to func() - * expires microseconds until timer fires - * - * Returns: - * no return value - * - */ -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), - void *data, DAPL_OS_TIMEVAL expires) -{ - struct dapl_timer_entry *list_ptr; - DAPL_OS_TIMEVAL cur_time; - boolean_t wakeup_tmo_thread; - - /* - * Start the timer thread the first time we need a timer - */ - if ( -1 == g_dapl_timer_head.thread_id ) { - g_dapl_timer_head.thread_id = - kernel_thread(dapl_timer_thread, &g_dapl_timer_head, 0); - } - - dapl_llist_init_entry(&timer->list_entry); - wakeup_tmo_thread = FALSE; - dapl_os_get_time(&cur_time); - timer->expires = cur_time + expires; /* calculate future time */ - timer->function = func; - timer->data = data; - - /* - * Put the element on the queue: sorted by wakeup time, eariliest - * first. - */ - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); - /* - * Deal with 3 cases due to our list structure: - * 1) list is empty: become the list head - * 2) New timer is sooner than list head: become the list head - * 3) otherwise, sort the timer into the list, no need to wake - * the timer thread up - */ - if (dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head)) { - /* Case 1: add entry to head of list */ - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, - timer); - wakeup_tmo_thread = TRUE; - } else { - list_ptr = (struct dapl_timer_entry *) - dapl_llist_peek_head(&g_dapl_timer_head.timer_list_head); - - if (timer->expires < list_ptr->expires) { - /* Case 2: add entry to head of list */ - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, timer); - wakeup_tmo_thread = TRUE; - } else { - /* Case 3: figure out where entry goes in sorted list */ - list_ptr = - dapl_llist_next_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry); - - while (list_ptr != NULL) { - if (timer->expires < list_ptr->expires) { - dapl_llist_add_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry, - (struct dapl_llist_entry *) - &timer->list_entry, - timer); - break; - - } - list_ptr = - dapl_llist_next_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry); - } - if (list_ptr == NULL) { - /* entry goes to the end of the list */ - dapl_llist_add_tail(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, timer); - } - } - - } - spin_unlock_irqrestore(&g_dapl_timer_head.lock, - g_dapl_timer_head.flags); - - if (wakeup_tmo_thread == TRUE) { - dapl_os_wait_object_wakeup(&g_dapl_timer_head.wait_object); - } - - return DAT_SUCCESS; -} - -/* - * dapl_timer_cancel - * - * Cancel a timer. Simply deletes the timer with no function invocations - * - * Input: - * timer User provided timer structure - * - * Returns: - * no return value - */ -void dapl_timer_cancel(struct dapl_timer_entry *timer) -{ - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); - /* - * make sure the entry has not been removed by another thread - */ - if (!dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head) && - timer->list_entry.list_head == &g_dapl_timer_head.timer_list_head) { - dapl_llist_remove_entry(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry); - } - /* - * If this was the first entry on the queue we could awaken the - * thread and have it reset the list; but it will just wake up - * and find that the timer entry has been removed, then go back - * to sleep, so don't bother. - */ - spin_unlock_irqrestore(&g_dapl_timer_head.lock, - g_dapl_timer_head.flags); -} - -/* - * dapl_timer_thread - * - * Core worker thread dealing with all timers. Basic algorithm: - * - Sleep until work shows up - * - Take first element of sorted timer list and wake - * invoke the callback if expired - * - Sleep for the timeout period if not expired - * - * Input: - * timer_head Timer head structure to manage timer lists - * - * Returns: - * int - */ -int dapl_timer_thread(void *arg) -{ - struct dapl_timer_entry *list_ptr; - DAPL_OS_TIMEVAL cur_time; - struct dapl_timer_head *timer_head; - u32 dat_status; - - timer_head = arg; - - for (;;) { - if (dapl_llist_is_empty(&timer_head->timer_list_head)) { - dat_status = - dapl_os_wait_object_wait(&timer_head->wait_object, - DAT_TIMEOUT_INFINITE); - } - - /* - * Lock policy: - * While this thread is accessing the timer list, it holds the - * lock. Otherwise, it doesn't. - */ - spin_lock_irqsave(&timer_head->lock, timer_head->flags); - while (!dapl_llist_is_empty(&timer_head->timer_list_head)) { - list_ptr = (struct dapl_timer_entry *) - dapl_llist_peek_head(&g_dapl_timer_head. - timer_list_head); - dapl_os_get_time(&cur_time); - - if (list_ptr->expires <= cur_time) { - /* - * Remove the entry from the list. Sort out how much - * time we need to sleep for the next one - */ - list_ptr = - dapl_llist_remove_head(&timer_head-> - timer_list_head); - spin_unlock_irqrestore(&timer_head->lock, - timer_head->flags); - - /* - * Invoke the user callback - */ - list_ptr->function(list_ptr->data); - /* timer structure was allocated by caller, we don't - * free it here. - */ - - /* reacquire the lock */ - spin_lock_irqsave(&timer_head->lock, - timer_head->flags); - } else { - spin_unlock_irqrestore(&timer_head->lock, - timer_head->flags); - dat_status = - dapl_os_wait_object_wait(&timer_head-> - wait_object, - (DAT_TIMEOUT) - (list_ptr-> - expires - - cur_time)); - spin_lock_irqsave(&timer_head->lock, - timer_head->flags); - } - } - /* - * release the lock before going back to the top to sleep - */ - spin_unlock_irqrestore(&timer_head->lock, timer_head->flags); - - if (dat_status == DAT_INTERNAL_ERROR) { - /* - * XXX What do we do here? - */ - } - } /* for (;;) */ - - return 0; -} Index: linux-kernel-timer/dat-provider/dapl_timer_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_timer_util.h (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_timer_util.h (working copy) @@ -1,42 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_TIMER_UTIL_H -#define DAPL_TIMER_UTIL_H - -void dapl_timer_init(void); - -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), - void *data, DAPL_OS_TIMEVAL expires); - -void dapl_timer_cancel(struct dapl_timer_entry *timer); - -#endif /* DAPL_TIMER_UTIL_H */ Index: linux-kernel-timer/dat-provider/dapl_ep_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_util.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_ep_util.c (working copy) @@ -168,9 +168,6 @@ void dapl_ep_dealloc(struct dapl_ep *ep_ if (ep_ptr->send_iov) kfree(ep_ptr->send_iov); - if (ep_ptr->cxn_timer) - kfree(ep_ptr->cxn_timer); - kfree(ep_ptr); } @@ -242,32 +239,6 @@ bail: } /* - * dapl_ep_timeout - * - * If this routine is invoked before a connection occurs, generate an - * event - */ -void dapl_ep_timeout(void *arg) -{ - struct dapl_ep *ep_ptr; - - dapl_dbg_log(DAPL_DBG_TYPE_CM, "--> dapl_ep_timeout! ep %lx\n", arg); - - ep_ptr = (struct dapl_ep *)arg; - - /* reset the EP state */ - ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - - /* Clean up the EP and put the underlying QP into the ERROR state. */ - dapl_ib_disconnect_clean(ep_ptr, TRUE); - - (void)dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. - connect_evd_handle, - DAT_CONNECTION_EVENT_TIMED_OUT, - (DAT_HANDLE) ep_ptr, 0, NULL); -} - -/* * dapl_ep_state_subtype * * Return the INVALID_STATE connection subtype associated with an Index: linux-kernel-timer/dat-provider/dapl_ep_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_util.h (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_ep_util.h (working copy) @@ -52,8 +52,6 @@ extern u32 dapl_ep_post_send_req(DAT_EP_ DAPL_DTO_TYPE dto_type, enum ib_wr_opcode op_type); -void dapl_ep_timeout(void *arg); - enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr); #endif /* DAPL_EP_UTIL_H */ Index: linux-kernel-timer/dat-provider/dapl_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_util.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_util.c (working copy) @@ -33,104 +33,6 @@ #include "dapl_provider.h" #include "dapl_util.h" -/* - * dapl_os_get_time - * - * Return 64 bit value of current time in microseconds. - * - * Input: - * loc User location to place current time - * - * Returns: - * DAT_SUCCESS - */ - -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *loc) -{ - struct timeval tv; - - do_gettimeofday(&tv); - *loc = ((u64) (tv.tv_sec) * 1000000L) + (u64) tv.tv_usec; - - return DAT_SUCCESS; -} - -/* - * Wait object routines - */ - -/* - * dapl_os_wait_object_init - * - * Initialize a wait object - * - * Input: - * wait_obj - * - * Returns: - * DAT_SUCCESS - * DAT_INTERNAL_ERROR - */ -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj) -{ - init_waitqueue_head(&wait_obj->wait_queue); - - return DAT_SUCCESS; -} - -/* Wait on the supplied wait object, up to the specified time_out. - * A timeout of DAT_TIMEOUT_INFINITE will wait indefinitely. - * Timeout should be specified in micro seconds. - * - * Functional returns: - * DAT_SUCCESS -- another thread invoked dapl_os_wait object_wakeup - * DAT_INVALID_STATE -- someone else is already waiting in this wait - * object. - * only one waiter is allowed at a time. - * DAT_ABORT -- another thread invoked dapl_os_wait_object_destroy - * DAT_TIMEOUT -- the specified time limit was reached. - */ - -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, - DAT_TIMEOUT timeout_val) -{ - u32 dat_status = DAT_SUCCESS; - - if (DAT_TIMEOUT_INFINITE == timeout_val) { - interruptible_sleep_on(&wait_obj->wait_queue); - } else { - int expire; - - expire = timeout_val * HZ / 1000000; - - while (expire) { - current->state = TASK_INTERRUPTIBLE; - expire = schedule_timeout(expire); - } - dat_status = DAT_ERROR(DAT_TIMEOUT_EXPIRED, 0); - } - return dat_status; -} - -/* - * dapl_os_wait_object_wakeup - * - * Wakeup a thread waiting on a wait object - * - * Input: - * wait_obj - * - * Returns: - * DAT_SUCCESS - * DAT_INTERNAL_ERROR - */ -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj) -{ - wake_up_interruptible(&wait_obj->wait_queue); - - return DAT_SUCCESS; -} - #ifdef DAPL_DBG void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) Index: linux-kernel-timer/dat-provider/dapl_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_util.h (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_util.h (working copy) @@ -107,24 +107,6 @@ static inline int dapl_os_atomic_assign( */ /* - * The wait object invariant: Presuming a call to dapl_os_wait_object_wait - * occurs at some point, there will be at least one wakeup after each call - * to dapl_os_wait_object_signal. I.e. Signals are not ignored, though - * they may be coallesced. - */ -typedef struct dapl_os_wait_object { - wait_queue_head_t wait_queue; -} DAPL_OS_WAIT_OBJECT; - -/* function prototypes */ -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj); - -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, - DAT_TIMEOUT timeout_val); - -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj); - -/* * Memory Functions */ @@ -154,15 +136,6 @@ static inline char *dapl_os_strdup(const } /* - * Timer Functions - */ - -typedef u64 DAPL_OS_TIMEVAL; - -/* timer function prototype */ -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *); - -/* * *printf format helper. We use the C string constant concatenation * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. Index: linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (revision 2518) +++ linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_ep_util.h" #include "dapl_openib_util.h" -#include "dapl_timer_util.h" /* * dapl_ep_dup_connect @@ -63,7 +62,7 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, - DAT_TIMEOUT timeout, int private_data_size, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { struct dapl_ep *ep_dup_ptr; @@ -114,6 +113,6 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep private_data_size, private_data, qos, DAT_CONNECT_DEFAULT_FLAG); - bail: +bail: return dat_status; } Index: linux-kernel-timer/dat-provider/dapl.h =================================================================== --- linux-kernel-timer/dat-provider/dapl.h (revision 2518) +++ linux-kernel-timer/dat-provider/dapl.h (working copy) @@ -236,14 +236,6 @@ struct dapl_evd { struct dapl_cno *cno_ptr; }; -/* DAPL timer entry, used to queue timeouts */ -struct dapl_timer_entry { - struct dapl_llist_entry list_entry; /* link entry on ia struct */ - DAPL_OS_TIMEVAL expires; - void (*function) (void *); - void *data; -}; - /* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { struct dapl_header header; @@ -264,8 +256,7 @@ struct dapl_ep { /* For passive connections we maintain a back pointer to the CR */ void *cr_ptr; - /* pointer to connection timer, if set */ - struct dapl_timer_entry *cxn_timer; + unsigned long timeout; /* private data container */ char private_data[DAPL_MAX_PRIVATE_DATA_SIZE]; @@ -500,14 +491,15 @@ extern u32 dapl_ep_modify(DAT_EP_HANDLE extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, - DAT_TIMEOUT timeout, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, + DAT_EP_HANDLE ep_dup_handle, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); Index: linux-kernel-timer/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel-timer/patches/alt_dat_provider_makefile (revision 2518) +++ linux-kernel-timer/patches/alt_dat_provider_makefile (working copy) @@ -71,7 +71,6 @@ PROVIDER_MODULES := \ dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ - dapl_timer_util \ dapl_util PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) @@ -97,4 +96,4 @@ default: endif clean: - rm -f *.o *.ko + rm -rf *.o *.ko .*.cmd *.mod.* .tmp_versions .*.d .*.o.tmp cscope.out Index: linux-kernel-timer/dat/dat.h =================================================================== --- linux-kernel-timer/dat/dat.h (revision 2518) +++ linux-kernel-timer/dat/dat.h (working copy) @@ -383,11 +383,6 @@ enum dat_completion_flags { DAT_COMPLETION_BARRIER_FENCE_FLAG = 0x08, }; -typedef u32 DAT_TIMEOUT; /* microseconds */ - -/* timeout = infinity */ -#define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) - /* dat handles */ typedef void *DAT_HANDLE; typedef DAT_HANDLE DAT_CR_HANDLE; @@ -1139,12 +1134,12 @@ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP const struct dat_ep_param *); typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, - DAT_CONN_QUAL, DAT_TIMEOUT, int, + DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, - DAT_TIMEOUT, int, const void *, + unsigned long timeout, int, const void *, enum dat_qos); typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); @@ -1493,7 +1488,7 @@ static inline u32 dat_evd_resize(DAT_EVD } static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, - DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, + DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags flags) @@ -1533,7 +1528,8 @@ static inline u32 dat_ep_disconnect(DAT_ } static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, - DAT_TIMEOUT timeout, int private_data_size, + unsigned long timeout, + int private_data_size, const void *private_data, enum dat_qos qos) { return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, From mshefty at ichips.intel.com Wed Jun 1 12:51:04 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 01 Jun 2005 12:51:04 -0700 Subject: [openib-general] minor issue in SA query code Message-ID: <429E11A8.104@ichips.intel.com> It looks like there might be minor race issue in the SA query code. ib_sa_path_rec_get() and ib_sa_mcmember_rec_query() both return query->sa_query.id. However, if a send completes quickly, I think that it's possible that query could have been freed. I don't think that this is affecting anyone. - Sean From roland at topspin.com Wed Jun 1 13:04:13 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 01 Jun 2005 13:04:13 -0700 Subject: [openib-general] Re: minor issue in SA query code In-Reply-To: <429E11A8.104@ichips.intel.com> (Sean Hefty's message of "Wed, 01 Jun 2005 12:51:04 -0700") References: <429E11A8.104@ichips.intel.com> Message-ID: <521x7l7sjm.fsf@topspin.com> Sean> It looks like there might be minor race issue in the SA Sean> query code. ib_sa_path_rec_get() and Sean> ib_sa_mcmember_rec_query() both return query-> sa_query.id. However, if a send completes quickly, I query-> think that it's possible that query could have been freed. You're right. I'll fix that up. - R. From Thomas.Talpey at netapp.com Wed Jun 1 13:12:15 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Wed, 01 Jun 2005 16:12:15 -0400 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117655324.19908.3.camel@duffman> References: <1117655324.19908.3.camel@duffman> Message-ID: <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> At 03:48 PM 6/1/2005, Tom Duffy wrote: >This patch takes into account some of the feedback by Sean, it sets the >retry to 0, so it will do the timeout right. It also stops the >conversion of ms to ib at 63 since 64 overflows. I have tested this >version using kdapltest in quit, performance, and transaction modes. > >James, do you think it good enough to apply? Do any of these tests intentionally trigger timer events? I don't think they do... BTW :-) surely you can think of an even more colorful name than this! >static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) Shouldn't this routine live in an IB-specific area, not core DAPL? Tom (Talpey) From tduffy at sun.com Wed Jun 1 13:19:14 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 13:19:14 -0700 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> Message-ID: <1117657154.25491.5.camel@duffman> On Wed, 2005-06-01 at 16:12 -0400, Talpey, Thomas wrote: > At 03:48 PM 6/1/2005, Tom Duffy wrote: > >I have tested this > >version using kdapltest in quit, performance, and transaction modes. > > Do any of these tests intentionally trigger timer events? I don't think > they do... Tom, do you know of a test that will? > BTW :-) surely you can think of an even more colorful name than this! Trying to be professional here ;-) > >static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) > > Shouldn't this routine live in an IB-specific area, not core DAPL? Well, maybe, if other ULPs want to use ms instead of IB. Do you know if this is the case? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Wed Jun 1 13:15:14 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 01 Jun 2005 16:15:14 -0400 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> Message-ID: <1117656914.4490.73.camel@hal.voltaire.com> On Wed, 2005-06-01 at 16:12, Talpey, Thomas wrote: > BTW :-) surely you can think of an even more colorful name than this! > > >static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) > > Shouldn't this routine live in an IB-specific area, not core DAPL? That appears to be the case as that routine is in dapl_openib_cm.c -- Hal From roland at topspin.com Wed Jun 1 13:48:33 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 01 Jun 2005 13:48:33 -0700 Subject: [openib-general] [PATCH] mthca: fix registration for giant MRs Message-ID: <52r7fl6bxa.fsf@topspin.com> Here's a patch that allows mthca to break up registration of giant userspace MRs into multiple firmware commands. The net effect of this is that there should no longer be any limit (beyond physical memory) on the size of MRs that userspace can register. Arlin -- could you give this a shot and see if it works for you? MST -- can you eyeball this patch and see if it looks OK? If no one sees problems with this, I'll commit it soon. - R. --- mthca/mthca_dev.h (revision 2470) +++ mthca/mthca_dev.h (working copy) @@ -387,6 +387,12 @@ void mthca_uar_free(struct mthca_dev *de int mthca_pd_alloc(struct mthca_dev *dev, int privileged, struct mthca_pd *pd); void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd); +struct mthca_mtt *mthca_alloc_mtt(struct mthca_dev *dev, int size, int fmr); +void mthca_free_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt); +int mthca_write_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt, + int start_index, u64 *buffer_list, int list_len); +int mthca_mr_alloc(struct mthca_dev *dev, u32 pd, int buffer_size_shift, + u64 iova, u64 total_size, u32 access, struct mthca_mr *mr); int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, u32 access, struct mthca_mr *mr); int mthca_mr_alloc_phys(struct mthca_dev *dev, u32 pd, --- mthca/mthca_provider.c (revision 2490) +++ mthca/mthca_provider.c (working copy) @@ -787,13 +787,13 @@ static struct ib_mr *mthca_reg_phys_mr(s static struct ib_mr *mthca_reg_user_mr(struct ib_pd *pd, struct ib_umem *region, int acc, const void __user *udata, int udatalen) { + struct mthca_dev *dev = to_mdev(pd->device); struct ib_umem_chunk *chunk; - int npages = 0; - u64 *page_list; struct mthca_mr *mr; - int shift; + u64 *pages; + int shift, n, len; int i, j, k; - int err; + int err = 0; shift = ffs(region->page_size) - 1; @@ -801,37 +801,68 @@ static struct ib_mr *mthca_reg_user_mr(s if (!mr) return ERR_PTR(-ENOMEM); + n = 0; list_for_each_entry(chunk, ®ion->chunk_list, list) - npages += chunk->nents; + n += chunk->nents; - page_list = kmalloc(npages * sizeof *page_list, GFP_KERNEL); - if (!page_list) { - kfree(mr); - return ERR_PTR(-ENOMEM); + mr->mtt = mthca_alloc_mtt(dev, n, 0); + if (IS_ERR(mr->mtt)) { + err = PTR_ERR(mr->mtt); + goto err; + } + + pages = (u64 *) __get_free_page(GFP_KERNEL); + if (!pages) { + err = -ENOMEM; + goto err_mtt; } - i = 0; + i = n = 0; list_for_each_entry(chunk, ®ion->chunk_list, list) - for (j = 0; j < chunk->nmap; ++j) - for (k = 0; k < sg_dma_len(&chunk->page_list[j]) >> shift; ++k) - page_list[i++] = sg_dma_address(&chunk->page_list[j]) + + for (j = 0; j < chunk->nmap; ++j) { + len = sg_dma_len(&chunk->page_list[j]) >> shift; + for (k = 0; k < len; ++k) { + pages[i++] = sg_dma_address(&chunk->page_list[j]) + region->page_size * k; + /* + * Be friendly to WRITE_MTT command + * and leave two empty slots for the + * index and reserved fields of the + * mailbox. + */ + if (i == PAGE_SIZE / sizeof (u64) - 2) { + err = mthca_write_mtt(dev, mr->mtt, + n, pages, i); + if (err) + goto mtt_done; + n += i; + i = 0; + } + } + } - err = mthca_mr_alloc_phys(to_mdev(pd->device), - to_mpd(pd)->pd_num, - page_list, shift, npages, - region->virt_base, region->length, - convert_access(acc), mr); + if (i) + err = mthca_write_mtt(dev, mr->mtt, n, pages, i); +mtt_done: + free_page((unsigned long) pages); + if (err) + goto err_mtt; - if (err) { - kfree(page_list); - kfree(mr); - return ERR_PTR(err); - } + err = mthca_mr_alloc(dev, to_mpd(pd)->pd_num, shift, region->virt_base, + region->length, convert_access(acc), mr); + + if (err) + goto err_mtt; - kfree(page_list); return &mr->ibmr; + +err_mtt: + mthca_free_mtt(dev, mr->mtt); + +err: + kfree(mr); + return ERR_PTR(err); } static int mthca_dereg_mr(struct ib_mr *mr) --- mthca/mthca_provider.h (revision 2470) +++ mthca/mthca_provider.h (working copy) @@ -62,18 +62,18 @@ struct mthca_ucontext { struct mthca_user_db_table *db_tab; }; +struct mthca_mtt; + struct mthca_mr { - struct ib_mr ibmr; - int order; - u32 first_seg; + struct ib_mr ibmr; + struct mthca_mtt *mtt; }; struct mthca_fmr { - struct ib_fmr ibmr; + struct ib_fmr ibmr; struct ib_fmr_attr attr; - int order; - u32 first_seg; - int maps; + struct mthca_mtt *mtt; + int maps; union { struct { struct mthca_mpt_entry __iomem *mpt; --- mthca/mthca_mr.c (revision 2468) +++ mthca/mthca_mr.c (working copy) @@ -40,6 +40,12 @@ #include "mthca_cmd.h" #include "mthca_memfree.h" +struct mthca_mtt { + int order; + u32 first_seg; + int fmr; +}; + /* * Must be packed because mtt_seg is 64 bits but only aligned to 32 bits. */ @@ -173,8 +179,8 @@ static void __devexit mthca_buddy_cleanu kfree(buddy->bits); } -static u32 mthca_alloc_mtt(struct mthca_dev *dev, int order, - struct mthca_buddy *buddy) +static u32 __mthca_alloc_mtt(struct mthca_dev *dev, int order, + struct mthca_buddy *buddy) { u32 seg = mthca_buddy_alloc(buddy, order); @@ -191,14 +197,99 @@ static u32 mthca_alloc_mtt(struct mthca_ return seg; } -static void mthca_free_mtt(struct mthca_dev *dev, u32 seg, int order, - struct mthca_buddy* buddy) +struct mthca_mtt *mthca_alloc_mtt(struct mthca_dev *dev, int size, int fmr) { - mthca_buddy_free(buddy, seg, order); + struct mthca_mtt *mtt; + int i; + + if (size <= 0) + return ERR_PTR(-EINVAL); + + mtt = kmalloc(sizeof *mtt, GFP_KERNEL); + if (!mtt) + return ERR_PTR(-ENOMEM); + + mtt->fmr = fmr; + mtt->order = 0; + for (i = MTHCA_MTT_SEG_SIZE / 8; i < size; i <<= 1) + ++mtt->order; + + mtt->first_seg = __mthca_alloc_mtt(dev, mtt->order, + fmr ? dev->mr_table.fmr_mtt_buddy : + &dev->mr_table.mtt_buddy); + if (mtt->first_seg == -1) { + kfree(mtt); + return ERR_PTR(-ENOMEM); + } + + return mtt; +} + +void mthca_free_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt) +{ + if (!mtt) + return; + + mthca_buddy_free(mtt->fmr ? dev->mr_table.fmr_mtt_buddy : + &dev->mr_table.mtt_buddy, mtt->first_seg, + mtt->order); if (mthca_is_memfree(dev)) - mthca_table_put_range(dev, dev->mr_table.mtt_table, seg, - seg + (1 << order) - 1); + mthca_table_put_range(dev, dev->mr_table.mtt_table, + mtt->first_seg, + mtt->first_seg + (1 << mtt->order) - 1); + + kfree(mtt); +} + +int mthca_write_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt, + int start_index, u64 *buffer_list, int list_len) +{ + u64 *mtt_entry; + int err = 0; + u8 status; + int i; + + mtt_entry = (u64 *) __get_free_page(GFP_KERNEL); + if (!mtt_entry) + return -ENOMEM; + + while (list_len > 0) { + mtt_entry[0] = cpu_to_be64(dev->mr_table.mtt_base + + mtt->first_seg * MTHCA_MTT_SEG_SIZE + + start_index * 8); + mtt_entry[1] = 0; + for (i = 0; i < list_len && i < PAGE_SIZE / 8 - 2; ++i) + mtt_entry[i + 2] = cpu_to_be64(buffer_list[i] | + MTHCA_MTT_FLAG_PRESENT); + + /* + * If we have an odd number of entries to write, add + * one more dummy entry for firmware efficiency. + */ + if (i & 1) + mtt_entry[i + 2] = 0; + + err = mthca_WRITE_MTT(dev, mtt_entry, (i + 1) & ~1, &status); + if (err) { + mthca_warn(dev, "WRITE_MTT failed (%d)\n", err); + goto out; + } + if (status) { + mthca_warn(dev, "WRITE_MTT returned status 0x%02x\n", + status); + err = -EINVAL; + goto out; + } + + list_len -= i; + start_index += i; + buffer_list += i; + } + +out: + free_page((unsigned long) mtt_entry); + return err; } static inline u32 tavor_hw_index_to_key(u32 ind) @@ -237,18 +328,20 @@ static inline u32 key_to_hw_index(struct return tavor_key_to_hw_index(key); } -int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, - u32 access, struct mthca_mr *mr) +int mthca_mr_alloc(struct mthca_dev *dev, u32 pd, int buffer_size_shift, + u64 iova, u64 total_size, u32 access, struct mthca_mr *mr) { void *mailbox = NULL; struct mthca_mpt_entry *mpt_entry; u32 key; + int i; int err; u8 status; might_sleep(); - mr->order = -1; + WARN_ON(buffer_size_shift >= 32); + key = mthca_alloc(&dev->mr_table.mpt_alloc); if (key == -1) return -ENOMEM; @@ -270,18 +363,36 @@ int mthca_mr_alloc_notrans(struct mthca_ mpt_entry->flags = cpu_to_be32(MTHCA_MPT_FLAG_SW_OWNS | MTHCA_MPT_FLAG_MIO | - MTHCA_MPT_FLAG_PHYSICAL | MTHCA_MPT_FLAG_REGION | access); - mpt_entry->page_size = 0; + if (!mr->mtt) + mpt_entry->flags |= cpu_to_be32(MTHCA_MPT_FLAG_PHYSICAL); + + mpt_entry->page_size = cpu_to_be32(buffer_size_shift - 12); mpt_entry->key = cpu_to_be32(key); mpt_entry->pd = cpu_to_be32(pd); - mpt_entry->start = 0; - mpt_entry->length = ~0ULL; + mpt_entry->start = cpu_to_be64(iova); + mpt_entry->length = cpu_to_be64(total_size); memset(&mpt_entry->lkey, 0, sizeof *mpt_entry - offsetof(struct mthca_mpt_entry, lkey)); + if (mr->mtt) + mpt_entry->mtt_seg = + cpu_to_be64(dev->mr_table.mtt_base + + mr->mtt->first_seg * MTHCA_MTT_SEG_SIZE); + + if (0) { + mthca_dbg(dev, "Dumping MPT entry %08x:\n", mr->ibmr.lkey); + for (i = 0; i < sizeof (struct mthca_mpt_entry) / 4; ++i) { + if (i % 4 == 0) + printk("[%02x] ", i * 4); + printk(" %08x", be32_to_cpu(((u32 *) mpt_entry)[i])); + if ((i + 1) % 4 == 0) + printk("\n"); + } + } + err = mthca_SW2HW_MPT(dev, mpt_entry, key & (dev->limits.num_mpts - 1), &status); @@ -308,150 +419,41 @@ err_out_mpt_free: return err; } +int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, + u32 access, struct mthca_mr *mr) +{ + mr->mtt = NULL; + return mthca_mr_alloc(dev, pd, 12, 0, ~0ULL, access, mr); +} + int mthca_mr_alloc_phys(struct mthca_dev *dev, u32 pd, u64 *buffer_list, int buffer_size_shift, int list_len, u64 iova, u64 total_size, u32 access, struct mthca_mr *mr) { - void *mailbox; - u64 *mtt_entry; - struct mthca_mpt_entry *mpt_entry; - u32 key; - int err = -ENOMEM; - u8 status; - int i; - - might_sleep(); - WARN_ON(buffer_size_shift >= 32); - - key = mthca_alloc(&dev->mr_table.mpt_alloc); - if (key == -1) - return -ENOMEM; - mr->ibmr.rkey = mr->ibmr.lkey = hw_index_to_key(dev, key); - - if (mthca_is_memfree(dev)) { - err = mthca_table_get(dev, dev->mr_table.mpt_table, key); - if (err) - goto err_out_mpt_free; - } - - for (i = MTHCA_MTT_SEG_SIZE / 8, mr->order = 0; - i < list_len; - i <<= 1, ++mr->order) - ; /* nothing */ - - mr->first_seg = mthca_alloc_mtt(dev, mr->order, - &dev->mr_table.mtt_buddy); - if (mr->first_seg == -1) - goto err_out_table; - - /* - * If list_len is odd, we add one more dummy entry for - * firmware efficiency. - */ - mailbox = kmalloc(max(sizeof *mpt_entry, - (size_t) 8 * (list_len + (list_len & 1) + 2)) + - MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) - goto err_out_free_mtt; - - mtt_entry = MAILBOX_ALIGN(mailbox); + int err; - mtt_entry[0] = cpu_to_be64(dev->mr_table.mtt_base + - mr->first_seg * MTHCA_MTT_SEG_SIZE); - mtt_entry[1] = 0; - for (i = 0; i < list_len; ++i) - mtt_entry[i + 2] = cpu_to_be64(buffer_list[i] | - MTHCA_MTT_FLAG_PRESENT); - if (list_len & 1) { - mtt_entry[i + 2] = 0; - ++list_len; - } + mr->mtt = mthca_alloc_mtt(dev, list_len, 0); + if (IS_ERR(mr->mtt)) + return PTR_ERR(mr->mtt); - if (0) { - mthca_dbg(dev, "Dumping MPT entry\n"); - for (i = 0; i < list_len + 2; ++i) - printk(KERN_ERR "[%2d] %016llx\n", - i, (unsigned long long) be64_to_cpu(mtt_entry[i])); - } - - err = mthca_WRITE_MTT(dev, mtt_entry, list_len, &status); + err = mthca_write_mtt(dev, mr->mtt, 0, buffer_list, list_len); if (err) { - mthca_warn(dev, "WRITE_MTT failed (%d)\n", err); - goto err_out_mailbox_free; - } - if (status) { - mthca_warn(dev, "WRITE_MTT returned status 0x%02x\n", - status); - err = -EINVAL; - goto err_out_mailbox_free; - } - - mpt_entry = MAILBOX_ALIGN(mailbox); - - mpt_entry->flags = cpu_to_be32(MTHCA_MPT_FLAG_SW_OWNS | - MTHCA_MPT_FLAG_MIO | - MTHCA_MPT_FLAG_REGION | - access); - - mpt_entry->page_size = cpu_to_be32(buffer_size_shift - 12); - mpt_entry->key = cpu_to_be32(key); - mpt_entry->pd = cpu_to_be32(pd); - mpt_entry->start = cpu_to_be64(iova); - mpt_entry->length = cpu_to_be64(total_size); - memset(&mpt_entry->lkey, 0, - sizeof *mpt_entry - offsetof(struct mthca_mpt_entry, lkey)); - mpt_entry->mtt_seg = cpu_to_be64(dev->mr_table.mtt_base + - mr->first_seg * MTHCA_MTT_SEG_SIZE); - - if (0) { - mthca_dbg(dev, "Dumping MPT entry %08x:\n", mr->ibmr.lkey); - for (i = 0; i < sizeof (struct mthca_mpt_entry) / 4; ++i) { - if (i % 4 == 0) - printk("[%02x] ", i * 4); - printk(" %08x", be32_to_cpu(((u32 *) mpt_entry)[i])); - if ((i + 1) % 4 == 0) - printk("\n"); - } + mthca_free_mtt(dev, mr->mtt); + return err; } - err = mthca_SW2HW_MPT(dev, mpt_entry, - key & (dev->limits.num_mpts - 1), - &status); + err = mthca_mr_alloc(dev, pd, buffer_size_shift, iova, + total_size, access, mr); if (err) - mthca_warn(dev, "SW2HW_MPT failed (%d)\n", err); - else if (status) { - mthca_warn(dev, "SW2HW_MPT returned status 0x%02x\n", - status); - err = -EINVAL; - } - - kfree(mailbox); - return err; - -err_out_mailbox_free: - kfree(mailbox); - -err_out_free_mtt: - mthca_free_mtt(dev, mr->first_seg, mr->order, &dev->mr_table.mtt_buddy); - -err_out_table: - if (mthca_is_memfree(dev)) - mthca_table_put(dev, dev->mr_table.mpt_table, key); + mthca_free_mtt(dev, mr->mtt); -err_out_mpt_free: - mthca_free(&dev->mr_table.mpt_alloc, key); return err; } /* Free mr or fmr */ -static void mthca_free_region(struct mthca_dev *dev, u32 lkey, int order, - u32 first_seg, struct mthca_buddy *buddy) +static void mthca_free_region(struct mthca_dev *dev, u32 lkey) { - if (order >= 0) - mthca_free_mtt(dev, first_seg, order, buddy); - if (mthca_is_memfree(dev)) mthca_table_put(dev, dev->mr_table.mpt_table, arbel_key_to_hw_index(lkey)); @@ -476,8 +478,8 @@ void mthca_free_mr(struct mthca_dev *dev mthca_warn(dev, "HW2SW_MPT returned status 0x%02x\n", status); - mthca_free_region(dev, mr->ibmr.lkey, mr->order, mr->first_seg, - &dev->mr_table.mtt_buddy); + mthca_free_region(dev, mr->ibmr.lkey); + mthca_free_mtt(dev, mr->mtt); } int mthca_fmr_alloc(struct mthca_dev *dev, u32 pd, @@ -522,21 +524,15 @@ int mthca_fmr_alloc(struct mthca_dev *de mr->mem.tavor.mpt = dev->mr_table.tavor_fmr.mpt_base + sizeof *(mr->mem.tavor.mpt) * idx; - for (i = MTHCA_MTT_SEG_SIZE / 8, mr->order = 0; - i < list_len; - i <<= 1, ++mr->order) - ; /* nothing */ - - mr->first_seg = mthca_alloc_mtt(dev, mr->order, - dev->mr_table.fmr_mtt_buddy); - if (mr->first_seg == -1) + mr->mtt = mthca_alloc_mtt(dev, list_len, 1); + if (IS_ERR(mr->mtt)) goto err_out_table; - mtt_seg = mr->first_seg * MTHCA_MTT_SEG_SIZE; + mtt_seg = mr->mtt->first_seg * MTHCA_MTT_SEG_SIZE; if (mthca_is_memfree(dev)) { mr->mem.arbel.mtts = mthca_table_find(dev->mr_table.mtt_table, - mr->first_seg); + mr->mtt->first_seg); BUG_ON(!mr->mem.arbel.mtts); } else mr->mem.tavor.mtts = dev->mr_table.tavor_fmr.mtt_base + mtt_seg; @@ -592,8 +588,7 @@ err_out_mailbox_free: kfree(mailbox); err_out_free_mtt: - mthca_free_mtt(dev, mr->first_seg, mr->order, - dev->mr_table.fmr_mtt_buddy); + mthca_free_mtt(dev, mr->mtt); err_out_table: if (mthca_is_memfree(dev)) @@ -609,8 +604,9 @@ int mthca_free_fmr(struct mthca_dev *dev if (fmr->maps) return -EBUSY; - mthca_free_region(dev, fmr->ibmr.lkey, fmr->order, fmr->first_seg, - dev->mr_table.fmr_mtt_buddy); + mthca_free_region(dev, fmr->ibmr.lkey); + mthca_free_mtt(dev, fmr->mtt); + return 0; } @@ -826,7 +822,7 @@ int __devinit mthca_init_mr_table(struct if (dev->limits.reserved_mtts) { i = fls(dev->limits.reserved_mtts - 1); - if (mthca_alloc_mtt(dev, i, dev->mr_table.fmr_mtt_buddy) == -1) { + if (__mthca_alloc_mtt(dev, i, dev->mr_table.fmr_mtt_buddy) == -1) { mthca_warn(dev, "MTT table of order %d is too small.\n", dev->mr_table.fmr_mtt_buddy->max_order); err = -ENOMEM; From tduffy at sun.com Wed Jun 1 13:51:01 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 13:51:01 -0700 Subject: [openib-general] opensm: new segv on shutdown Message-ID: <1117659061.25491.12.camel@duffman> I am putting together a network with a dumb IB switch, a couple of Linux OpenIB boxes, a Solaris 10 box, a Solaris Nevada box, etc. I fired up opensm on one of the Linux nodes, tried to plumb Solaris, no luck. I then hit control-c on opensm and it crashed. Here is the messages and then crash. OpenSM: Got signal 2 - exiting... ------- OpenSM Exiting (in 3 seconds) ------- There are still 44 mads out. Forcing the exit of the OpenSM application... Segmentation fault (core dumped) #0 stack_dump () at src/stack.c:72 72 if (!__builtin_frame_address(2)) #0 stack_dump () at src/stack.c:72 #1 0x00002aaaaacbd1a6 in handler (x=11) at src/stack.c:151 #2 #3 __match_notice_to_inf_rec (p_list_item=0x585e50, context=0x10) at osm_inform.c:413 #4 0x00002aaaaadc49ff in cl_qlist_apply_func (p_list=0x550680, pfn_func=0x4085e0 <__match_notice_to_inf_rec>, context=0x43004f80) at cl_list.c:354 #5 0x0000000000408c57 in osm_report_notice (p_log=0x552498, p_subn=0x5502e0, p_ntc=0x430050d0) at osm_inform.c:680 #6 0x000000000042b472 in __osm_trap_rcv_process_request (p_rcv=0x5511e8, p_madw=0x0) at osm_trap_rcv.c:646 #7 0x000000000042b7fb in osm_trap_rcv_process (p_rcv=0x5511e8, p_madw=0x566e60) at osm_trap_rcv.c:724 #8 0x00000000004033bb in __cl_disp_worker (context=0x585e50) at cl_dispatcher.c:105 #9 0x00002aaaaadc928f in __cl_thread_pool_routine (context=0x552558) at cl_threadpool.c:78 #10 0x00002aaaaadc911e in __cl_thread_wrapper (arg=0x51) at cl_thread.c:61 #11 0x00000036d28060aa in start_thread () from /lib64/tls/libpthread.so.0 #12 0x00000036d19c53d3 in clone () from /lib64/tls/libc.so.6 #13 0x0000000000000000 in ?? () Also, here are the last 500 lines of osm.log. Jun 01 13:37:00 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:00 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:01 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:01 [44808960] -> osm_vendor_get: [ Jun 01 13:37:01 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x566178, size = 256. Jun 01 13:37:01 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03210, size = 256. Jun 01 13:37:01 [44808960] -> osm_vendor_get: ] Jun 01 13:37:01 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x566160, p_mad = 0x2aaaaaf03240, size = 256. Jun 01 13:37:01 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 99 QP1 MADs received. Jun 01 13:37:01 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000007 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:02 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:02 [44808960] -> osm_vendor_get: [ Jun 01 13:37:02 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5660a8, size = 256. Jun 01 13:37:02 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03350, size = 256. Jun 01 13:37:02 [44808960] -> osm_vendor_get: ] Jun 01 13:37:02 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x566090, p_mad = 0x2aaaaaf03380, size = 256. Jun 01 13:37:02 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 100 QP1 MADs received. Jun 01 13:37:02 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000007 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:03 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:03 [44808960] -> osm_vendor_get: [ Jun 01 13:37:03 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565fd8, size = 256. Jun 01 13:37:03 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03490, size = 256. Jun 01 13:37:03 [44808960] -> osm_vendor_get: ] Jun 01 13:37:03 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565fc0, p_mad = 0x2aaaaaf034c0, size = 256. Jun 01 13:37:03 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 101 QP1 MADs received. Jun 01 13:37:03 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000007 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:04 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:04 [44808960] -> osm_vendor_get: [ Jun 01 13:37:04 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565f08, size = 256. Jun 01 13:37:04 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf035d0, size = 256. Jun 01 13:37:04 [44808960] -> osm_vendor_get: ] Jun 01 13:37:04 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565ef0, p_mad = 0x2aaaaaf03600, size = 256. Jun 01 13:37:04 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 102 QP1 MADs received. Jun 01 13:37:04 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000008 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:04 [44007960] -> osm_state_mgr_process: [ Jun 01 13:37:04 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:37:04 [44007960] -> osm_state_mgr_process: ] Jun 01 13:37:05 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:05 [44808960] -> osm_vendor_get: [ Jun 01 13:37:05 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565e38, size = 256. Jun 01 13:37:05 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03710, size = 256. Jun 01 13:37:05 [44808960] -> osm_vendor_get: ] Jun 01 13:37:05 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565e20, p_mad = 0x2aaaaaf03740, size = 256. Jun 01 13:37:05 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 103 QP1 MADs received. Jun 01 13:37:05 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000008 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:06 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:06 [44808960] -> osm_vendor_get: [ Jun 01 13:37:06 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565d68, size = 256. Jun 01 13:37:06 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03850, size = 256. Jun 01 13:37:06 [44808960] -> osm_vendor_get: ] Jun 01 13:37:06 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565d50, p_mad = 0x2aaaaaf03880, size = 256. Jun 01 13:37:06 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 104 QP1 MADs received. Jun 01 13:37:06 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000008 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:07 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:07 [44808960] -> osm_vendor_get: [ Jun 01 13:37:07 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565c98, size = 256. Jun 01 13:37:07 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03990, size = 256. Jun 01 13:37:07 [44808960] -> osm_vendor_get: ] Jun 01 13:37:07 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565c80, p_mad = 0x2aaaaaf039c0, size = 256. Jun 01 13:37:07 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 105 QP1 MADs received. Jun 01 13:37:07 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000009 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:09 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:09 [44808960] -> osm_vendor_get: [ Jun 01 13:37:09 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565bc8, size = 256. Jun 01 13:37:09 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03ad0, size = 256. Jun 01 13:37:09 [44808960] -> osm_vendor_get: ] Jun 01 13:37:09 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565bb0, p_mad = 0x2aaaaaf03b00, size = 256. Jun 01 13:37:09 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 106 QP1 MADs received. Jun 01 13:37:09 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000009 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:10 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:10 [44808960] -> osm_vendor_get: [ Jun 01 13:37:10 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565af8, size = 256. Jun 01 13:37:10 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03c10, size = 256. Jun 01 13:37:10 [44808960] -> osm_vendor_get: ] Jun 01 13:37:10 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565ae0, p_mad = 0x2aaaaaf03c40, size = 256. Jun 01 13:37:10 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 107 QP1 MADs received. Jun 01 13:37:10 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976571200000009 attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:11 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:11 [44808960] -> osm_vendor_get: [ Jun 01 13:37:11 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565a28, size = 256. Jun 01 13:37:11 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03d50, size = 256. Jun 01 13:37:11 [44808960] -> osm_vendor_get: ] Jun 01 13:37:11 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565a10, p_mad = 0x2aaaaaf03d80, size = 256. Jun 01 13:37:11 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 108 QP1 MADs received. Jun 01 13:37:11 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x97657120000000a attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:12 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:12 [44808960] -> osm_vendor_get: [ Jun 01 13:37:12 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565958, size = 256. Jun 01 13:37:12 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03e90, size = 256. Jun 01 13:37:12 [44808960] -> osm_vendor_get: ] Jun 01 13:37:12 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565940, p_mad = 0x2aaaaaf03ec0, size = 256. Jun 01 13:37:12 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 109 QP1 MADs received. Jun 01 13:37:12 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x97657120000000a attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:13 [44808960] -> osm_mad_pool_get: [ Jun 01 13:37:13 [44808960] -> osm_vendor_get: [ Jun 01 13:37:13 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565888, size = 256. Jun 01 13:37:13 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03fd0, size = 256. Jun 01 13:37:13 [44808960] -> osm_vendor_get: ] Jun 01 13:37:13 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565870, p_mad = 0x2aaaaaf04000, size = 256. Jun 01 13:37:13 [44808960] -> osm_mad_pool_get: ] Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 110 QP1 MADs received. Jun 01 13:37:13 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x97657120000000a attr_id.................0x38 (MCMemberRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000008081 Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_process: ] Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] Jun 01 13:37:14 [44007960] -> osm_state_mgr_process: [ Jun 01 13:37:14 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:37:14 [44007960] -> osm_state_mgr_process: ] Jun 01 13:37:24 [44007960] -> osm_state_mgr_process: [ Jun 01 13:37:24 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:37:24 [44007960] -> osm_state_mgr_process: ] Jun 01 13:37:34 [44007960] -> osm_state_mgr_process: [ Jun 01 13:37:34 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:37:34 [44007960] -> osm_state_mgr_process: ] Jun 01 13:37:44 [44007960] -> osm_state_mgr_process: [ Jun 01 13:37:44 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:37:44 [44007960] -> osm_state_mgr_process: ] Jun 01 13:37:54 [44007960] -> osm_state_mgr_process: [ Jun 01 13:37:54 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:37:54 [44007960] -> osm_state_mgr_process: ] Jun 01 13:38:04 [44007960] -> osm_state_mgr_process: [ Jun 01 13:38:04 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:38:04 [44007960] -> osm_state_mgr_process: ] Jun 01 13:38:14 [44007960] -> osm_state_mgr_process: [ Jun 01 13:38:14 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:38:14 [44007960] -> osm_state_mgr_process: ] Jun 01 13:38:24 [44007960] -> osm_state_mgr_process: [ Jun 01 13:38:24 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:38:24 [44007960] -> osm_state_mgr_process: ] Jun 01 13:38:34 [44007960] -> osm_state_mgr_process: [ Jun 01 13:38:34 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:38:34 [44007960] -> osm_state_mgr_process: ] Jun 01 13:38:44 [44007960] -> osm_state_mgr_process: [ Jun 01 13:38:44 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:38:44 [44007960] -> osm_state_mgr_process: ] Jun 01 13:38:54 [44007960] -> osm_state_mgr_process: [ Jun 01 13:38:54 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:38:54 [44007960] -> osm_state_mgr_process: ] Jun 01 13:39:04 [44007960] -> osm_state_mgr_process: [ Jun 01 13:39:04 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:39:04 [44007960] -> osm_state_mgr_process: ] Jun 01 13:39:14 [44007960] -> osm_state_mgr_process: [ Jun 01 13:39:14 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:39:14 [44007960] -> osm_state_mgr_process: ] Jun 01 13:39:24 [44007960] -> osm_state_mgr_process: [ Jun 01 13:39:24 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:39:24 [44007960] -> osm_state_mgr_process: ] Jun 01 13:39:34 [44007960] -> osm_state_mgr_process: [ Jun 01 13:39:34 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:39:34 [44007960] -> osm_state_mgr_process: ] Jun 01 13:39:44 [44007960] -> osm_state_mgr_process: [ Jun 01 13:39:44 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. Jun 01 13:39:44 [44007960] -> osm_state_mgr_process: ] Jun 01 13:39:54 [44007960] -> osm_state_mgr_process: [ Jun 01 13:39:54 [44007960] -> osm_state_mgr_process: ] Jun 01 13:39:56 [AAEE0B60] -> osm_vl15_shutdown: [ Jun 01 13:39:56 [AAEE0B60] -> osm_vl15_shutdown: ] Jun 01 13:39:56 [AAEE0B60] -> osm_vendor_set_sm: [ Jun 01 13:39:56 [AAEE0B60] -> osm_vendor_set_sm: ] Jun 01 13:39:56 [AAEE0B60] -> clear_madw: [ Jun 01 13:39:56 [AAEE0B60] -> clear_madw: ] Jun 01 13:39:56 [43005960] -> osm_physp_share_pkey: ] Jun 01 13:39:56 [43005960] -> osm_port_share_pkey: ] Jun 01 13:39:56 [43005960] -> __match_notice_to_inf_rec: Mismatch by Pkey Jun 01 13:39:56 [43005960] -> __match_notice_to_inf_rec: ] Jun 01 13:39:56 [AAEE0B60] -> osm_sm_destroy: [ Jun 01 13:39:56 [44007960] -> __osm_sm_sweeper: Off schedule sweep signalled. Jun 01 13:39:56 [44007960] -> __osm_sm_sweeper: ] -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From Thomas.Talpey at netapp.com Wed Jun 1 14:10:04 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Wed, 01 Jun 2005 17:10:04 -0400 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117657154.25491.5.camel@duffman> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> <1117657154.25491.5.camel@duffman> Message-ID: <6.2.1.2.2.20050601170353.057b0d60@exnane01.nane.netapp.com> At 04:19 PM 6/1/2005, Tom Duffy wrote: >> Do any of these tests intentionally trigger timer events? I don't think >> they do... > >Tom, do you know of a test that will? Can't think of one, actually. Seems like it would take some kind of error injection. Maybe drop the peer into a debugger while the tests are running. The main goal being to see that a timer fired, and at about the right time. >> >static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) >> >> Shouldn't this routine live in an IB-specific area, not core DAPL? > >Well, maybe, if other ULPs want to use ms instead of IB. Do you know if >this is the case? Hal's right - it does live in an IB-specific file, though that seems to live in a common directory. I don't think the conversion is sufficiently general to warrant any special status. Isn't it the case that it is Mellanox-hardware specific, not even IB-specific? IOW, I guess it's fine the way it is. Tom. From mshefty at ichips.intel.com Wed Jun 1 14:26:57 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 01 Jun 2005 14:26:57 -0700 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <6.2.1.2.2.20050601170353.057b0d60@exnane01.nane.netapp.com> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> <1117657154.25491.5.camel@duffman> <6.2.1.2.2.20050601170353.057b0d60@exnane01.nane.netapp.com> Message-ID: <429E2821.6020602@ichips.intel.com> Talpey, Thomas wrote: > Hal's right - it does live in an IB-specific file, though that seems to > live in a common directory. I don't think the conversion is sufficiently > general to warrant any special status. Isn't it the case that it is > Mellanox-hardware specific, not even IB-specific? The time format is IB architecture specific. The CM is exposing a field value defined by the spec. If a different format would be a better alternative, we can look at changing it. - Sean From Thomas.Talpey at netapp.com Wed Jun 1 14:39:13 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Wed, 01 Jun 2005 17:39:13 -0400 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <429E2821.6020602@ichips.intel.com> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> <1117657154.25491.5.camel@duffman> <6.2.1.2.2.20050601170353.057b0d60@exnane01.nane.netapp.com> <429E2821.6020602@ichips.intel.com> Message-ID: <6.2.1.2.2.20050601173720.04061c50@exnane01.nane.netapp.com> At 05:26 PM 6/1/2005, Sean Hefty wrote: >The time format is IB architecture specific. The CM is exposing a field >value defined by the spec. If a different format would be a better >alternative, we can look at changing it. Okay. (But, the spec requires it to be a u8? Sheesh) Anyway, I vote for milliseconds. :-) Tom. From bjordan.ics at gmail.com Wed Jun 1 14:43:42 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Wed, 1 Jun 2005 17:43:42 -0400 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: References: <1117539047.4476.199.camel@hal.voltaire.com> Message-ID: <78d18e20506011443175617b3@mail.gmail.com> >Has anyone seen ib_send_cm_req() return -22? I'm not sure what you are testing with, Jeff, but I ran into the same problem the first time I tried to use ucm_simple. The source and destination lid and guid are embedded in the source, and need to be modified to fit the test systems. Otherwise you will get an errno EINVAL. -- Bill Jordan SilverStorm Technologies From bjordan at sbc2.infiniconsys.com Wed Jun 1 14:52:47 2005 From: bjordan at sbc2.infiniconsys.com (Bill Jordan) Date: Wed, 1 Jun 2005 17:52:47 -0400 Subject: [openib-general] [PATCH] ib_ucm: remove devfs usage Message-ID: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> Libor, Here is a patch to remove the devfs usage from ib_ucm. The udev support already exists, but requires a rule entry of KERNEL="ucm", NAME="infiniband/%k". Is there an appropriate place that the udev rules should be documented? Signed-off-by: Bill Jordan Index: ucm.c =================================================================== --- ucm.c (revision 2519) +++ ucm.c (working copy) @@ -40,7 +40,6 @@ #include #include #include -#include #include @@ -1376,10 +1375,6 @@ static int __init ib_ucm_init(void) NULL, "ucm"); - devfs_mk_cdev(IB_UCM_DEV, - S_IFCHR|S_IRUGO|S_IWUGO, - "infiniband/ucm"); - idr_init(&ctx_id_table); init_MUTEX(&ctx_id_mutex); @@ -1394,7 +1389,6 @@ err_chr: static void __exit ib_ucm_cleanup(void) { - devfs_remove("infiniband/ucm"); class_simple_device_remove(IB_UCM_DEV); class_simple_destroy(ib_ucm_class); cdev_del(&ib_ucm_cdev); From tduffy at sun.com Wed Jun 1 14:55:45 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 14:55:45 -0700 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <429E2821.6020602@ichips.intel.com> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> <1117657154.25491.5.camel@duffman> <6.2.1.2.2.20050601170353.057b0d60@exnane01.nane.netapp.com> <429E2821.6020602@ichips.intel.com> Message-ID: <1117662945.25491.23.camel@duffman> On Wed, 2005-06-01 at 14:26 -0700, Sean Hefty wrote: > Talpey, Thomas wrote: > > Hal's right - it does live in an IB-specific file, though that seems to > > live in a common directory. I don't think the conversion is sufficiently > > general to warrant any special status. Isn't it the case that it is > > Mellanox-hardware specific, not even IB-specific? > > The time format is IB architecture specific. The CM is exposing a field > value defined by the spec. If a different format would be a better > alternative, we can look at changing it. I was looking at some of the ULPs. SDP sets the timeout to 20, without explanation. DAPL has a #define (set to 20 saying it is 4 sec in a comment). simple example? Set to 20, no reason. To me, it ends up just being this opaque value. Maybe some pre#defines in CM would make it clearer. Unfortunately, advertising milliseconds or jiffies will just confuse consumers since the timeout cannot be that granular. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mshefty at ichips.intel.com Wed Jun 1 15:07:20 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 01 Jun 2005 15:07:20 -0700 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117662945.25491.23.camel@duffman> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> <1117657154.25491.5.camel@duffman> <6.2.1.2.2.20050601170353.057b0d60@exnane01.nane.netapp.com> <429E2821.6020602@ichips.intel.com> <1117662945.25491.23.camel@duffman> Message-ID: <429E3198.9090503@ichips.intel.com> Tom Duffy wrote: >>>Hal's right - it does live in an IB-specific file, though that seems to >>>live in a common directory. I don't think the conversion is sufficiently >>>general to warrant any special status. Isn't it the case that it is >>>Mellanox-hardware specific, not even IB-specific? >> >>The time format is IB architecture specific. The CM is exposing a field >>value defined by the spec. If a different format would be a better >>alternative, we can look at changing it. > > I was looking at some of the ULPs. SDP sets the timeout to 20, without > explanation. DAPL has a #define (set to 20 saying it is 4 sec in a > comment). simple example? Set to 20, no reason. To me, it ends up > just being this opaque value. Maybe some pre#defines in CM would make > it clearer. Unfortunately, advertising milliseconds or jiffies will > just confuse consumers since the timeout cannot be that granular. I don't know if the spec provides any help for a ULP to determine the local or remote CM timeouts. And the CM itself requires work from the ULP to process requests, so it cannot set the value itself. The value is encoded into a 5-bit field, and for what it's worth, packet lifetime values are provided by the SA in this same format... - Sean From tduffy at sun.com Wed Jun 1 15:08:43 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 01 Jun 2005 15:08:43 -0700 Subject: [openib-general] [PATCH] ib_ucm: remove devfs usage In-Reply-To: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> References: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> Message-ID: <1117663723.25491.29.camel@duffman> On Wed, 2005-06-01 at 17:52 -0400, Bill Jordan wrote: > The udev support already exists, but requires a > rule entry of KERNEL="ucm", NAME="infiniband/%k". > Is there an appropriate place that the udev rules > should be documented? right now, the udev rules are documented in: gen2/trunk/src/linux-kernel/docs/user_mad.txt which Roland submits to lkml as: linux/Documentation/infiniband/user_mad.txt But, this won't get the rules created in the distros copies of /etc/udev/rules.d/*. For that, you will need to submit a patch to linux-hotplug-devel at lists.sourceforge.net and file bugs with each distro. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From iod00d at hp.com Wed Jun 1 15:13:26 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 1 Jun 2005 15:13:26 -0700 Subject: [openib-general] [PATCH] ib_ucm: remove devfs usage In-Reply-To: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> References: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> Message-ID: <20050601221326.GL25321@esmail.cup.hp.com> On Wed, Jun 01, 2005 at 05:52:47PM -0400, Bill Jordan wrote: > The udev support already exists, but requires a > rule entry of KERNEL="ucm", NAME="infiniband/%k". > Is there an appropriate place that the udev rules > should be documented? I checked the openib_faq.txt and it doesn't mention "udev". userspace/README is the best I could find: http://openib.org/pipermail/openib-commits/2004-December/001053.html But the README is not complete. This has been posted to the openib-general mailing list before: http://openib.org/pipermail/openib-general/2005-March/005069.html hth, grant From iod00d at hp.com Wed Jun 1 15:47:29 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 1 Jun 2005 15:47:29 -0700 Subject: [openib-general] [PATCH] mthca: fix registration for giant MRs In-Reply-To: <52r7fl6bxa.fsf@topspin.com> References: <52r7fl6bxa.fsf@topspin.com> Message-ID: <20050601224729.GM25321@esmail.cup.hp.com> On Wed, Jun 01, 2005 at 01:48:33PM -0700, Roland Dreier wrote: > Here's a patch that allows mthca to break up registration of giant > userspace MRs into multiple firmware commands. The net effect of this > is that there should no longer be any limit (beyond physical memory) > on the size of MRs that userspace can register. There is a limit on some platforms: IO Virtual Address (IOVA) space I'm not worried about it on ia64 platforms since they mostly direct map for 64-bit devices. Either they have no IOMMU or 64-bit devices are allowed to bypass the IOMMU. parisc only has 1GB per IOMMU (and several can be in a system). Roland also once mentioned a PPC box that also forces everything through the IOMMU. The problem is sga_dma_map* functions do not return an error. One needs to call dma_mapping_error() to verify the IO address returned was valid. This is a relatively new interface so I wouldn't expect that many driver writers are not aware of it. Most don't need to be aware of it since they don't attempt to map GBs of physical memory at a time. grant From roland at topspin.com Wed Jun 1 15:58:47 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 01 Jun 2005 15:58:47 -0700 Subject: [openib-general] Re: minor issue in SA query code In-Reply-To: <429E11A8.104@ichips.intel.com> (Sean Hefty's message of "Wed, 01 Jun 2005 12:51:04 -0700") References: <429E11A8.104@ichips.intel.com> Message-ID: <52y89t4rbs.fsf@topspin.com> Sean> It looks like there might be minor race issue in the SA Sean> query code. ib_sa_path_rec_get() and Sean> ib_sa_mcmember_rec_query() both return Sean> query->sa_query.id. However, if a send completes Sean> quickly, I think that it's possible that query could have Sean> been freed. I think this patch should fix it -- does this seem right to you? - R. --- sa_query.c (revision 2518) +++ sa_query.c (working copy) @@ -551,6 +551,7 @@ int ib_sa_path_rec_get(struct ib_device struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client); struct ib_sa_port *port = &sa_dev->port[port_num - sa_dev->start_port]; struct ib_mad_agent *agent = port->agent; + int id; int ret; query = kmalloc(sizeof *query, gfp_mask); @@ -578,6 +579,8 @@ int ib_sa_path_rec_get(struct ib_device rec, query->sa_query.mad->data); *sa_query = &query->sa_query; + id = query->sa_query.id; + ret = send_mad(&query->sa_query, timeout_ms); if (ret) { *sa_query = NULL; @@ -585,7 +588,7 @@ int ib_sa_path_rec_get(struct ib_device kfree(query); } - return ret ? ret : query->sa_query.id; + return ret ? ret : id; } EXPORT_SYMBOL(ib_sa_path_rec_get); @@ -627,6 +630,7 @@ int ib_sa_mcmember_rec_query(struct ib_d struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client); struct ib_sa_port *port = &sa_dev->port[port_num - sa_dev->start_port]; struct ib_mad_agent *agent = port->agent; + int id; int ret; query = kmalloc(sizeof *query, gfp_mask); @@ -654,6 +658,8 @@ int ib_sa_mcmember_rec_query(struct ib_d rec, query->sa_query.mad->data); *sa_query = &query->sa_query; + id = query->sa_query.id; + ret = send_mad(&query->sa_query, timeout_ms); if (ret) { *sa_query = NULL; @@ -661,7 +667,7 @@ int ib_sa_mcmember_rec_query(struct ib_d kfree(query); } - return ret ? ret : query->sa_query.id; + return ret ? ret : id; } EXPORT_SYMBOL(ib_sa_mcmember_rec_query); From roland at topspin.com Wed Jun 1 16:00:14 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 01 Jun 2005 16:00:14 -0700 Subject: [openib-general] [PATCH] mthca: fix registration for giant MRs In-Reply-To: <20050601224729.GM25321@esmail.cup.hp.com> (Grant Grundler's message of "Wed, 1 Jun 2005 15:47:29 -0700") References: <52r7fl6bxa.fsf@topspin.com> <20050601224729.GM25321@esmail.cup.hp.com> Message-ID: <52u0kh4r9d.fsf@topspin.com> Grant> There is a limit on some platforms: IO Virtual Address Grant> (IOVA) space You're right -- I wasn't precise in my previous email. All my patch tries to do is remove arbitrary limitations caused by mthca insisting on allocating contiguous memory for no good reason. As you alluded to, some pSeries machines have extremely limited IOMMU resources and the amount of memory that can be registered at any one time may be severely limited. - R. From mshefty at ichips.intel.com Wed Jun 1 16:00:47 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 01 Jun 2005 16:00:47 -0700 Subject: [openib-general] Re: minor issue in SA query code In-Reply-To: <52y89t4rbs.fsf@topspin.com> References: <429E11A8.104@ichips.intel.com> <52y89t4rbs.fsf@topspin.com> Message-ID: <429E3E1F.3070206@ichips.intel.com> Roland Dreier wrote: > Sean> It looks like there might be minor race issue in the SA > Sean> query code. ib_sa_path_rec_get() and > Sean> ib_sa_mcmember_rec_query() both return > Sean> query->sa_query.id. However, if a send completes > Sean> quickly, I think that it's possible that query could have > Sean> been freed. > > I think this patch should fix it -- does this seem right to you? It looks good to me. - Sean From roland at topspin.com Wed Jun 1 16:03:16 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 01 Jun 2005 16:03:16 -0700 Subject: [openib-general] [PATCH] mthca: fix registration for giant MRs In-Reply-To: <20050601224729.GM25321@esmail.cup.hp.com> (Grant Grundler's message of "Wed, 1 Jun 2005 15:47:29 -0700") References: <52r7fl6bxa.fsf@topspin.com> <20050601224729.GM25321@esmail.cup.hp.com> Message-ID: <52oeap4r4b.fsf@topspin.com> Grant> The problem is sga_dma_map* functions do not return an Grant> error. One needs to call dma_mapping_error() to verify the Grant> IO address returned was valid. This is a relatively new Grant> interface so I wouldn't expect that many driver writers are Grant> not aware of it. Most don't need to be aware of it since Grant> they don't attempt to map GBs of physical memory at a time. On ppc64 at least, dma_map_sg() will return 0 if the mapping request fails (even if it fails in the middle). Are other architectures different? - R. From iod00d at hp.com Wed Jun 1 16:21:23 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 1 Jun 2005 16:21:23 -0700 Subject: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050525145727.08DE92283D7@openib.ca.sandia.gov> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> Message-ID: <20050601232123.GN25321@esmail.cup.hp.com> On Wed, May 25, 2005 at 07:57:27AM -0700, mst at openib.org wrote: > gen2/trunk/src/userspace/perftest/rdma_bw.c > Log: > Bandwidth test for RDMA write Michael, I finally had a chance to test rdma_bw....and it didn't work :^( This was running r2519 IB kernel drivers, libibverbs, libmthca. Here is the rdma_bw output on HP IA64 ZX1 (rx2600) machines: grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw local address: LID 0x27, QPN 0x70406, PSN 0xac2e45 RKey 0x2a0434 VAddr 0x6000000000014001 remote address: LID 0x25, QPN 0x60406, PSN 0xf0f7a, RKey 0x240434 VAddr 0x6000000000014001 server read: Address family not supported by protocol 0/45: Couldn't read remote address grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw 10.0.0.51 local address: LID 0x25, QPN 0x60406, PSN 0xf0f7a RKey 0x240434 VAddr 0x6000000000014001 remote address: LID 0x27, QPN 0x70406, PSN 0xac2e45, RKey 0x2a0434 VAddr 0x6000000000014001 Couldn't post send: scnt=0 rdma_lat test does work. I'll post results in a seperate email since there is a bit more discussion around those results. thanks, grant From halr at voltaire.com Wed Jun 1 16:24:18 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 01 Jun 2005 19:24:18 -0400 Subject: [openib-general] user space verbs examples Message-ID: <1117667806.4490.160.camel@hal.voltaire.com> Hi Roland, I can run ibv_devices but ibv_asyncwatch seems to fail: ibv_devices device node GUID ------ ---------------- mthca0 0008f10403960558 ibv_asyncwatch Couldn't get context for mthca0 dmesg shows: Jun 1 19:01:00 hal kernel: UCM: Created struct Jun 1 19:01:00 hal kernel: UCM: Deleted struct Any idea on what is wrong ? Thanks. -- Hal From roland at topspin.com Wed Jun 1 16:34:03 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 01 Jun 2005 16:34:03 -0700 Subject: [openib-general] Re: user space verbs examples In-Reply-To: <1117667806.4490.160.camel@hal.voltaire.com> (Hal Rosenstock's message of "01 Jun 2005 19:24:18 -0400") References: <1117667806.4490.160.camel@hal.voltaire.com> Message-ID: <52fyw14pp0.fsf@topspin.com> Hal> ibv_asyncwatch Couldn't get context for mthca0 Hal> dmesg shows: Jun 1 19:01:00 hal kernel: UCM: Created struct Hal> Jun 1 19:01:00 hal kernel: UCM: Deleted struct Looks like it's coming from UCM, not any of these tests... Hal> Any idea on what is wrong ? Nope. Are all your userspace and kernel versions in sync? If so, then send the output of 'strace -ewrite=all -etrace=write ibv_asyncwatch' - R. From robert.j.woodruff at intel.com Wed Jun 1 16:36:34 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Wed, 1 Jun 2005 16:36:34 -0700 Subject: [openib-general] README file for user-verbs access (increase memlock in /etc/security/limits.conf) In-Reply-To: <52oeap4r4b.fsf@topspin.com> Message-ID: Hi Roland, Now that this code has been added to ib_umem_get, locked = npages + current->mm->locked_vm; lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur >> PAGE_SHIFT; if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) { ret = -ENOMEM; goto out; } If there is a readme (how to) file on the user-mode verbs, might want to tell people that to use user-mode verbs, they will need to get their system administrator to increase the default value of memlock in /etc/security/limits.conf By default, RedHat EL4.0 for example, limits it to 32K, which is not particulatly useful and they will likely get the ENOMEM error returned when trying to register memory. woody From iod00d at hp.com Wed Jun 1 16:40:49 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 1 Jun 2005 16:40:49 -0700 Subject: [openib-general] [PATCH] mthca: fix registration for giant MRs In-Reply-To: <52oeap4r4b.fsf@topspin.com> References: <52r7fl6bxa.fsf@topspin.com> <20050601224729.GM25321@esmail.cup.hp.com> <52oeap4r4b.fsf@topspin.com> Message-ID: <20050601234049.GO25321@esmail.cup.hp.com> On Wed, Jun 01, 2005 at 04:03:16PM -0700, Roland Dreier wrote: ... > On ppc64 at least, dma_map_sg() will return 0 if the mapping request > fails (even if it fails in the middle). Are other architectures > different? Yes. "0" is a valid DMA address. no portable PCI driver should be expecting 0 for a failure. ppc64 can return anything it likes. Many arches (including parisc, alpha, sparc) would just panic when the DMA pool was exhausted. I've had this discussion several years ago: | Resent-Date: Fri, 10 Nov 2000 04:39:25 -0700 | From: "David S. Miller" ... | Subject: Re: [parisc-linux] tulip DMA mapping ... | I've quoted the whole of Grants message below, so you can see the | context. It looks like tulip is treating zero as meaning it | doesn't have anything to pci_unmap... | Thank you for the clarification. | | Jeff, this is in fact a bug, please fix :-) Original problem report is here: http://lists.parisc-linux.org/pipermail/parisc-linux/2000-November/009850.html hth, grant From roland at topspin.com Wed Jun 1 16:39:56 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 01 Jun 2005 16:39:56 -0700 Subject: [openib-general] [PATCH] mthca: fix registration for giant MRs In-Reply-To: <20050601234049.GO25321@esmail.cup.hp.com> (Grant Grundler's message of "Wed, 1 Jun 2005 16:40:49 -0700") References: <52r7fl6bxa.fsf@topspin.com> <20050601224729.GM25321@esmail.cup.hp.com> <52oeap4r4b.fsf@topspin.com> <20050601234049.GO25321@esmail.cup.hp.com> Message-ID: <52acm94pf7.fsf@topspin.com> Grant> Yes. "0" is a valid DMA address. no portable PCI driver Grant> should be expecting 0 for a failure. ppc64 can return Grant> anything it likes. Many arches (including parisc, alpha, Grant> sparc) would just panic when the DMA pool was exhausted. But dma_map_sg() isn't returning a DMA address -- it's returning the number of entries in the sg list it used to do the mapping. So 0 indicates an error, and there's no need to scan through the whole returned list for a bad address. - R. From roland at topspin.com Wed Jun 1 16:40:50 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 01 Jun 2005 16:40:50 -0700 Subject: [openib-general] Re: README file for user-verbs access (increase memlock in /etc/security/limits.conf) In-Reply-To: (Bob Woodruff's message of "Wed, 1 Jun 2005 16:36:34 -0700") References: Message-ID: <524qch4pdp.fsf@topspin.com> Thanks, I'll add that to the (currently unwritten) libibverbs README. - R. From libor at topspin.com Wed Jun 1 17:36:38 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 1 Jun 2005 17:36:38 -0700 Subject: [openib-general] [PATCH] ib_ucm: remove devfs usage In-Reply-To: <200506012152.j51LqlB22501@sbc2.infiniconsys.com>; from bjordan@sbc2.infiniconsys.com on Wed, Jun 01, 2005 at 05:52:47PM -0400 References: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> Message-ID: <20050601173638.A14794@topspin.com> On Wed, Jun 01, 2005 at 05:52:47PM -0400, Bill Jordan wrote: > Libor, > > Here is a patch to remove the devfs usage from ib_ucm. Thanks Bill, applied and commited. > The udev support already exists, but requires a > rule entry of KERNEL="ucm", NAME="infiniband/%k". > Is there an appropriate place that the udev rules > should be documented? Should probably be documented in libibcm/README at a minimum, that would be the natural place to look. Other then that udev support/naming look reasonable as is? -Libor From iod00d at hp.com Wed Jun 1 17:52:28 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 1 Jun 2005 17:52:28 -0700 Subject: [openib-general] [PATCH] rdma_lat-09 and results Message-ID: <20050602005228.GP25321@esmail.cup.hp.com> Michael, Good news: My next cleanup of rdma_lat.c is working and patch is appended. Summary of changes below. Bad News: perf is about ~15 cycles slower since the last time I tested. (Hrm...maybe it's time to cycle power on the TS90 switch again.) Here's with the new rdma_lat.c: grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C local address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 remote address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 Latency typical: 7140 cycles Latency best : 6915 cycles Latency worst : 52915.5 cycles grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ And the "client" side: grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C 10.0.0.51 local address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 remote address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 Latency typical: 7140 cycles Latency best : 6907 cycles Latency worst : 94920 cycles The previous set of rdma_lat results are here: http://openib.org/pipermail/openib-general/2005-May/006721.html I'll guess the previous SVN verion was no older than r2229. I get 7140 to 7151 for the original rdma_lat. Usually 7147.5. I get 7132 to 7155 with my version of rdma_lat. Usually 7140. No statistically significant differences. Both essentially agree on the higher result. Using "-n 10000" gave more consistent results * I use "taskset" to bind the rdma_lat test to a CPU. But it didn't matter which CPU I bound the task to - results where basically the same. I suspect the "stream" mode just does not depend on or generating that many interrupts. diffstat rdma_lat.c-09-diff rdma_lat.c | 395 +++++++++++++++++++++++++++++-------------------------------- 1 files changed, 188 insertions(+), 207 deletions(-) Commit Log entry/Summary of changes: o move device lookup from main() to pp_find_dev() o move sockfd handling code to pp_open_port() o consolidate server/client "key exchange" code path o enumerate return values in main() o fixed nit: pp_*_exch_dest was called twice. Each time it would malloc a new "rem_dest". Code in pp_open_port() now free()'s the first one. Signed-off-by: Grant Grundler thanks, grant Index: rdma_lat.c =================================================================== --- rdma_lat.c (revision 2519) +++ rdma_lat.c (working copy) @@ -103,30 +103,40 @@ static uint16_t pp_get_local_lid(struct return attr.lid; } -static int pp_client_connect(const char *servername, int port) + +static int pp_connect_sock(const char *servername, int port) { struct addrinfo *res, *t; struct addrinfo hints = { + .ai_flags = AI_PASSIVE, /* Server only? */ .ai_family = AF_UNSPEC, .ai_socktype = SOCK_STREAM }; char *service; - int n; int sockfd = -1; + int n; asprintf(&service, "%d", port); n = getaddrinfo(servername, service, &hints, &res); if (n < 0) { - fprintf(stderr, "%s for %s:%d\n", gai_strerror(n), servername, port); + fprintf(stderr, "%s for %s:%d\n", gai_strerror(n), + servername ? servername : "NULL", port); return n; } for (t = res; t; t = t->ai_next) { sockfd = socket(t->ai_family, t->ai_socktype, t->ai_protocol); if (sockfd >= 0) { - if (!connect(sockfd, t->ai_addr, t->ai_addrlen)) - break; + if (servername) { + if (!connect(sockfd, t->ai_addr, t->ai_addrlen)) + break; + } else { + setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &n, sizeof n); + if (!bind(sockfd, t->ai_addr, t->ai_addrlen)) + break; + } + close(sockfd); sockfd = -1; } @@ -134,147 +144,100 @@ static int pp_client_connect(const char freeaddrinfo(res); - if (sockfd < 0) { - fprintf(stderr, "Couldn't connect to %s:%d\n", servername, port); - return sockfd; + if (sockfd < 0) + fprintf(stderr, "Couldn't %s to %s:%d\n", + servername ? "connect" : "bind", + servername ? servername : "NULL", port); + + /* "server" side needs to wait/listen for client to call */ + if (!servername) { + int connfd; + + listen(sockfd, 1); + connfd = accept(sockfd, NULL, 0); + if (connfd < 0) { + perror("server accept"); + fprintf(stderr, "accept() failed\n"); + } + close(sockfd); + return connfd; } + return sockfd; } -struct pingpong_dest * pp_client_exch_dest(int sockfd, - const struct pingpong_dest *my_dest) +#define KEY_MSG_SIZE (sizeof "0000:000000:000000:00000000:0000000000000000") +#define KEY_PRINT_FMT "%04x:%06x:%06x:%08x:%016Lx" + +static int pp_write_keys(int sockfd, const struct pingpong_dest *my_dest) { - struct pingpong_dest *rem_dest = NULL; - char msg[sizeof "0000:000000:000000:00000000:0000000000000000"]; - int parsed; + char msg[KEY_MSG_SIZE]; + + sprintf(msg, KEY_PRINT_FMT, my_dest->lid, my_dest->qpn, + my_dest->psn, my_dest->rkey, my_dest->vaddr); - sprintf(msg, "%04x:%06x:%06x:%08x:%016Lx", my_dest->lid, my_dest->qpn, - my_dest->psn,my_dest->rkey,my_dest->vaddr); if (write(sockfd, msg, sizeof msg) != sizeof msg) { perror("client write"); fprintf(stderr, "Couldn't send local address\n"); - goto out; + return 0; } + return 1; +} + +static struct pingpong_dest * pp_read_keys(int sockfd, const struct pingpong_dest *my_dest) +{ + struct pingpong_dest *rem_dest = NULL; + int parsed; + char msg[KEY_MSG_SIZE]; + if (read(sockfd, msg, sizeof msg) != sizeof msg) { - perror("client read"); + perror("pp_read_keys"); fprintf(stderr, "Couldn't read remote address\n"); - goto out; + return NULL; } rem_dest = malloc(sizeof *rem_dest); if (!rem_dest) - goto out; + return NULL; - parsed = sscanf(msg, "%x:%x:%x:%x:%Lx", &rem_dest->lid, &rem_dest->qpn, - &rem_dest->psn,&rem_dest->rkey,&rem_dest->vaddr); + parsed = sscanf(msg, KEY_PRINT_FMT, &rem_dest->lid, &rem_dest->qpn, + &rem_dest->psn, &rem_dest->rkey, &rem_dest->vaddr); if (parsed != 5) { - fprintf(stderr, "Couldn't parse line <%.*s>\n",(int)sizeof msg, - msg); + fprintf(stderr, "Couldn't parse line <%.*s>\n", + (int)sizeof msg, msg); free(rem_dest); - rem_dest = NULL; - goto out; - } -out: - return rem_dest; -} - -int pp_server_connect(int port) -{ - struct addrinfo *res, *t; - struct addrinfo hints = { - .ai_flags = AI_PASSIVE, - .ai_family = AF_UNSPEC, - .ai_socktype = SOCK_STREAM - }; - char *service; - int sockfd = -1, connfd; - int n; - - asprintf(&service, "%d", port); - n = getaddrinfo(NULL, service, &hints, &res); - - if (n < 0) { - fprintf(stderr, "%s for port %d\n", gai_strerror(n), port); - return n; - } - - for (t = res; t; t = t->ai_next) { - sockfd = socket(t->ai_family, t->ai_socktype, t->ai_protocol); - if (sockfd >= 0) { - n = 1; - - setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &n, sizeof n); - - if (!bind(sockfd, t->ai_addr, t->ai_addrlen)) - break; - close(sockfd); - sockfd = -1; - } - } - - freeaddrinfo(res); - - if (sockfd < 0) { - fprintf(stderr, "Couldn't listen to port %d\n", port); - return sockfd; - } - - listen(sockfd, 1); - connfd = accept(sockfd, NULL, 0); - if (connfd < 0) { - perror("server accept"); - fprintf(stderr, "accept() failed\n"); - close(sockfd); - return connfd; + return NULL; } - close(sockfd); - return connfd; + return rem_dest; } -static struct pingpong_dest *pp_server_exch_dest(int connfd, const struct pingpong_dest *my_dest) +static struct pingpong_dest * pp_exch_dest(int sockfd, const char *servername, + const struct pingpong_dest *my_dest) { - char msg[sizeof "0000:000000:000000:00000000:0000000000000000"]; struct pingpong_dest *rem_dest = NULL; - int parsed; - int n; - - n = read(connfd, msg, sizeof msg); - if (n != sizeof msg) { - perror("server read"); - fprintf(stderr, "%d/%d: Couldn't read remote address\n", n, (int) sizeof msg); - goto out; - } - rem_dest = malloc(sizeof *rem_dest); - if (!rem_dest) - goto out; - - parsed = sscanf(msg, "%x:%x:%x:%x:%Lx", &rem_dest->lid, &rem_dest->qpn, - &rem_dest->psn, &rem_dest->rkey, &rem_dest->vaddr); - if (parsed != 5) { - fprintf(stderr, "Couldn't parse line <%.*s>\n",(int)sizeof msg, - msg); - free(rem_dest); - rem_dest = NULL; - goto out; + if (servername) { + if (!pp_write_keys(sockfd, my_dest)) + goto exch_failed; + rem_dest = pp_read_keys(sockfd, my_dest); + if (!rem_dest) + goto exch_failed; + } else { + rem_dest = pp_read_keys(sockfd, my_dest); + if (!rem_dest) + goto exch_failed; + if (!pp_write_keys(sockfd, my_dest)) + goto exch_failed; } - sprintf(msg, "%04x:%06x:%06x:%08x:%016Lx", my_dest->lid, my_dest->qpn, - my_dest->psn, my_dest->rkey, my_dest->vaddr); - if (write(connfd, msg, sizeof msg) != sizeof msg) { - perror("server write"); - fprintf(stderr, "Couldn't send local address\n"); - free(rem_dest); - rem_dest = NULL; - goto out; - } -out: +exch_failed: return rem_dest; } + static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size, int tx_depth, int port) { @@ -424,6 +387,86 @@ static int pp_connect_ctx(struct pingpon return 0; } +static struct ibv_device * pp_find_dev( const char *ib_devname) +{ + struct dlist *dev_list; + struct ibv_device *ib_dev = NULL; + + dev_list = ibv_get_devices(); + + dlist_start(dev_list); + if (!ib_devname) { + ib_dev = dlist_next(dev_list); + if (!ib_dev) + fprintf(stderr, "No IB devices found\n"); + } else { + dlist_for_each_data(dev_list, ib_dev, struct ibv_device) + if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) + break; + if (!ib_dev) + fprintf(stderr, "IB device %s not found\n", ib_devname); + } + return ib_dev; +} + + +static struct pingpong_dest * pp_open_port(struct pingpong_context *ctx, + const char * servername, int ib_port, int port) +{ + char addr_fmt[] = "%8s address: LID %#04x QPN %#06x PSN %#06x RKey %#08x VAddr %#016Lx\n"; + struct pingpong_dest my_dest; + struct pingpong_dest *rem_dest; + int sockfd; + + + /* Create connection between client and server. + * We do it by exchanging data over a TCP socket connection. */ + + my_dest.lid = pp_get_local_lid(ctx, ib_port); + my_dest.qpn = ctx->qp->qp_num; + my_dest.psn = lrand48() & 0xffffff; + if (!my_dest.lid) { + fprintf(stderr, "Local lid 0x0 detected. Is an SM running?\n"); + return NULL; + } + my_dest.rkey = ctx->mr->rkey; + my_dest.vaddr = (uintptr_t)ctx->buf + ctx->size; + + printf(addr_fmt, "local", my_dest.lid, my_dest.qpn, my_dest.psn, + my_dest.rkey, my_dest.vaddr); + + + sockfd = pp_connect_sock(servername, port); + if (sockfd < 0) { + printf("pp_connect_sock(%s,%d) failed (%d)!\n", + servername, port, sockfd); + return NULL; + } + + rem_dest = pp_exch_dest(sockfd, servername, &my_dest); + if (!rem_dest) + return NULL; + + printf(addr_fmt, "remote", rem_dest->lid, rem_dest->qpn, rem_dest->psn, + rem_dest->rkey, rem_dest->vaddr); + + if (pp_connect_ctx(ctx, ib_port, my_dest.psn, rem_dest)) + return NULL; + + /* An additional handshake is required *after* moving qp to RTR. + * Arbitrarily reuse exch_dest for this purpose. + */ + + free(rem_dest); + rem_dest = pp_exch_dest(sockfd, servername, &my_dest); + + write(sockfd, "done", sizeof "done"); + close(sockfd); + + return rem_dest; +} + + static void usage(const char *argv0) { printf("Usage:\n"); @@ -518,27 +561,26 @@ static void print_report(struct report_o int main(int argc, char *argv[]) { - struct dlist *dev_list; struct ibv_device *ib_dev; struct pingpong_context *ctx; - struct pingpong_dest my_dest; struct pingpong_dest *rem_dest; char *ib_devname = NULL; char *servername = NULL; - int port = 18515; - int ib_port = 1; - int size = 1; - int tx_depth = 50; - int iters = 1000; - int scnt, rcnt, ccnt; - int sockfd; + struct ibv_qp *qp; struct ibv_send_wr *wr; volatile char *poll_buf; volatile char *post_buf; - struct report_options report = {}; - cycles_t *tstamp; + int port = 18515; + int ib_port = 1; + int size = 1; + int iters = 1000; + int tx_depth = 50; + int scnt, rcnt, ccnt; + + static struct report_options report = {}; + static cycles_t *tstamp; /* Parameter parsing. */ while (1) { @@ -578,25 +620,25 @@ int main(int argc, char *argv[]) ib_port = strtol(optarg, NULL, 0); if (ib_port < 0) { usage(argv[0]); - return 1; + return 2; } break; case 's': size = strtol(optarg, NULL, 0); - if (size < 1) { usage(argv[0]); return 1; } + if (size < 1) { usage(argv[0]); return 3; } break; case 't': tx_depth = strtol(optarg, NULL, 0); - if (tx_depth < 1) { usage(argv[0]); return 1; } + if (tx_depth < 1) { usage(argv[0]); return 4; } break; case 'n': iters = strtol(optarg, NULL, 0); if (iters < 2) { usage(argv[0]); - return 1; + return 5; } break; @@ -615,7 +657,7 @@ int main(int argc, char *argv[]) default: usage(argv[0]); - return 1; + return 5; } } @@ -623,90 +665,26 @@ int main(int argc, char *argv[]) servername = strdupa(argv[optind]); else if (optind < argc) { usage(argv[0]); - return 1; + return 6; } - /* Done with parameter parsing. Perform setup. */ + /* + * Done with parameter parsing. Perform setup. + */ srand48(getpid() * time(NULL)); - page_size = sysconf(_SC_PAGESIZE); - dev_list = ibv_get_devices(); - - dlist_start(dev_list); - if (!ib_devname) { - ib_dev = dlist_next(dev_list); - if (!ib_dev) { - fprintf(stderr, "No IB devices found\n"); - return 1; - } - } else { - dlist_for_each_data(dev_list, ib_dev, struct ibv_device) - if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) - break; - if (!ib_dev) { - fprintf(stderr, "IB device %s not found\n", ib_devname); - return 1; - } - } + ib_dev = pp_find_dev(ib_devname); ctx = pp_init_ctx(ib_dev, size, tx_depth, ib_port); if (!ctx) - return 1; - - /* Create connection between client and server. - * We do it by exchanging data over a TCP socket connection. */ - - my_dest.lid = pp_get_local_lid(ctx, ib_port); - my_dest.qpn = ctx->qp->qp_num; - my_dest.psn = lrand48() & 0xffffff; - if (!my_dest.lid) { - fprintf(stderr, "Local lid 0x0 detected. Is an SM running?\n"); - return 1; - } - my_dest.rkey = ctx->mr->rkey; - my_dest.vaddr = (uintptr_t)ctx->buf + ctx->size; - - printf(" local address: LID %#04x, QPN %#06x, PSN %#06x " - "RKey %#08x VAddr %#016Lx\n", - my_dest.lid, my_dest.qpn, my_dest.psn, - my_dest.rkey, my_dest.vaddr); - - if (servername) { - sockfd = pp_client_connect(servername, port); - if (sockfd < 0) - return 1; - rem_dest = pp_client_exch_dest(sockfd, &my_dest); - } else { - sockfd = pp_server_connect(port); - if (sockfd < 0) - return 1; - rem_dest = pp_server_exch_dest(sockfd, &my_dest); - } + return 7; + rem_dest = pp_open_port(ctx, servername, ib_port, port); if (!rem_dest) - return 1; - - printf(" remote address: LID %#04x, QPN %#06x, PSN %#06x, " - "RKey %#08x VAddr %#016Lx\n", - rem_dest->lid, rem_dest->qpn, rem_dest->psn, - rem_dest->rkey, rem_dest->vaddr); - - if (pp_connect_ctx(ctx, ib_port, my_dest.psn, rem_dest)) - return 1; - - /* An additional handshake is required *after* moving qp to RTR. - Arbitrarily reuse exch_dest for this purpose. */ - if (servername) { - rem_dest = pp_client_exch_dest(sockfd, &my_dest); - } else { - rem_dest = pp_server_exch_dest(sockfd, &my_dest); - } - - write(sockfd, "done", sizeof "done"); - close(sockfd); + return 8; wr = &ctx->wr; ctx->list.addr = (uintptr_t) ctx->buf; @@ -726,7 +704,7 @@ int main(int argc, char *argv[]) if (!tstamp) { perror("malloc"); - return 1; + return 9; } /* Done with setup. Start the test. */ @@ -736,8 +714,8 @@ int main(int argc, char *argv[]) /* Wait till buffer changes. */ if (rcnt < iters && !(scnt < 1 && servername)) { ++rcnt; - while (*poll_buf != (char)rcnt) { - } + while (*poll_buf != (char)rcnt) + ; /* Here the data is already in the physical memory. If we wanted to actually use it, we may need a read memory barrier here. */ @@ -751,7 +729,8 @@ int main(int argc, char *argv[]) if (ibv_post_send(qp, wr, &bad_wr)) { fprintf(stderr, "Couldn't post send: scnt=%d\n", scnt); - return 1; + free(tstamp); + return 10; } } @@ -765,7 +744,8 @@ int main(int argc, char *argv[]) if (ne < 0) { fprintf(stderr, "poll CQ failed %d\n", ne); - return 1; + free(tstamp); + return 11; } if (wc.status != IBV_WC_SUCCESS) { fprintf(stderr, "Completion wth error at %s:\n", @@ -774,7 +754,8 @@ int main(int argc, char *argv[]) wc.status, (int) wc.wr_id); fprintf(stderr, "scnt=%d, rcnt=%d, ccnt=%d\n", scnt, rcnt, ccnt); - return 1; + free(tstamp); + return 12; } } } From halr at voltaire.com Wed Jun 1 17:45:45 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 01 Jun 2005 20:45:45 -0400 Subject: [openib-general] Re: opensm: new segv on shutdown In-Reply-To: <1117659061.25491.12.camel@duffman> References: <1117659061.25491.12.camel@duffman> Message-ID: <1117672772.4476.46.camel@hal.voltaire.com> On Wed, 2005-06-01 at 16:51, Tom Duffy wrote: > I am putting together a network with a dumb IB switch, a couple of Linux > OpenIB boxes, a Solaris 10 box, a Solaris Nevada box, etc. I fired up > opensm on one of the Linux nodes, tried to plumb Solaris, no luck. I > then hit control-c on opensm and it crashed. Here is the messages and > then crash. Anything from the Solaris side on what it doesn't like about the OpenIB RMPP ? More to come... -- Hal From iod00d at hp.com Wed Jun 1 18:05:20 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 1 Jun 2005 18:05:20 -0700 Subject: [openib-general] [PATCH] mthca: fix registration for giant MRs In-Reply-To: <52acm94pf7.fsf@topspin.com> References: <52r7fl6bxa.fsf@topspin.com> <20050601224729.GM25321@esmail.cup.hp.com> <52oeap4r4b.fsf@topspin.com> <20050601234049.GO25321@esmail.cup.hp.com> <52acm94pf7.fsf@topspin.com> Message-ID: <20050602010520.GQ25321@esmail.cup.hp.com> On Wed, Jun 01, 2005 at 04:39:56PM -0700, Roland Dreier wrote: > Grant> Yes. "0" is a valid DMA address. no portable PCI driver > Grant> should be expecting 0 for a failure. ppc64 can return > Grant> anything it likes. Many arches (including parisc, alpha, > Grant> sparc) would just panic when the DMA pool was exhausted. > > But dma_map_sg() isn't returning a DMA address -- it's returning the > number of entries in the sg list it used to do the mapping. So 0 > indicates an error, and there's no need to scan through the whole > returned list for a bad address. Of course - you're right. My brain was fixated on dma_map_single(). Documentation/DMA-API.txt does a better job of calling out zero is a failure. I don't know if ia64 DMA support correctly handles this but will find out. I don't recall this being part of the original interface but glad to see it is. thanks, grant From caitlin.bestler at gmail.com Wed Jun 1 18:33:14 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 1 Jun 2005 18:33:14 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050531155812.B12767@topspin.com> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050531155812.B12767@topspin.com> Message-ID: <469958e0050601183329578260@mail.gmail.com> The need for a single unifying event disptaching method was *the* primary feedback received from VIA user when DAPL started. That included both kernel and user-mode applications, in fact one of the primary reasons why the DAT Collaborative was started was that there was no unrestricted API for VIA that could be used in the kernel. As I stated earlier, kDAPL is designed for *applications* and was not really intended for use by the OS itself. As such it abstracts more than just the transport. If you wrip out the EVDs it isn't really "kDAPL" anymore. Having a stripped subset of kDAPL with a clearly different name that is *only* what is needed to abstract transport differences might be a really neat idea, and having it be otherwise compatible with kDAPL would also be good. But I'd suggest understanding why something is present before ripping it out. EVDs are there precisely because dealing with multiple callbacks and completion queues was universally found to be unwieldly and unfriendly to application developers. On 5/31/05, Libor Michalek wrote: > > Well, from a kernel API design philosophy the evd is somewhat odd. > The whole idea behind the event model seems a bit convoluted. First > multiplex a wide variety of events from the provider into a single event > queue, and then have an API so the consumer can tell what type of event > they actually have and can still receive the event notification in the > provider's context. > > This seems to be a lot of work to first hide useful information, but > also not loose the information in case the consumer really does want it. > It appears to be a case of a decent userspace idea that doesn't make > much sense in the kernel. Why is it there? I imagine it's to abstract a > variety of OS kernels, which was one of the goals of the design. > > Also, I realize it's just an implementation detail, but I've got a > number of issues with ATS. > > > -Libor > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From caitlin.bestler at gmail.com Wed Jun 1 18:37:24 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 1 Jun 2005 18:37:24 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050601100410.GA16265@lst.de> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> Message-ID: <469958e005060118376562dd75@mail.gmail.com> On 6/1/05, Christoph Hellwig wrote: > On Wed, Jun 01, 2005 at 09:36:09AM +0200, Christoph Hellwig wrote: > > For (1) doing a proper RDMA stack should solve thing, and the discussion > > how to do it is already ongoing on this list. Once we have proper RMDA > > stack that part of KDAPL isn't needed at all anymore. The second is > > more interesting and there's indeed the need for some higher level > > helpers than the API at the level of the current OpenIB code offers. But > > having a separate layer, with different data structures, provider > > registration and a totally different API for that is utter nonsense. > > Instead the higher level helpers should operate on the same > > datastructures as the RDMA stack, or build new ones ontop of that. > > In addition some of the abstractions don't make much sense, the event > > handling has already been mentioned. > > That beeing said, one of the first thing you should get rid of if you > want to be able to take code from kdapl to the generic rdma code is way > it deals with handles. The kdapl code gives up language type safety > just to add its own bandaided type system below. Please always pass the > real struct pointer around, or at least the generic struct type which > might be embedded into a bigger structure, and get rid of all the magic > number checking and magic macros that expect special members at the > beginning of all objects behind these handles. > That is a good suggestion, and is fully compatible with DAT defnitions. In fact the first version of the reference implementation did use struct pointers as the DAT handles. The indirection method was added after the fact to reduce problems with bad pointers in sloppy user-mode applications, and is specifically something that any DAT implementation is free to revert back to the struct pointer. I fully agree that having the indirected handled is a layer of safety that is not called for in a kernel (or any embedded application for that matter). DAT Consumers are not supposed to rely upon such safety mechanisms, and the DAT handle is opaque to them. So changing the representation of the DAT Handle is *totally* legitimate. From caitlin.bestler at gmail.com Wed Jun 1 18:42:40 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 1 Jun 2005 18:42:40 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050531155812.B12767@topspin.com> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050531155812.B12767@topspin.com> Message-ID: <469958e005060118427c050e25@mail.gmail.com> On 5/31/05, Libor Michalek wrote: > > Also, I realize it's just an implementation detail, but I've got a > number of issues with ATS. > ATS was a workaround the fact that OS naming systems could not be relied upon to deal with IPv6 format addresses (i.e. GIDs). That is an artifact of DAT being distributed outside of the OS. Working in full integration with the kernel I would see no reason why ATS could not be replaced by any other solution that provided the same answers. I suspect being integrated with subnet management and the kernel that there are in fact a lot of new options that were not available in the context where ATS was designed. ATS is not visible to the application. There's a reason for that. Any solution that can translate "dat_ep_connect" to the correct remote endpoint works and will not break existing DAT applications as long as the new solution can accept at least the same data as ATS does now (or equivalent). From Thomas.Talpey at netapp.com Wed Jun 1 19:39:23 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Wed, 01 Jun 2005 22:39:23 -0400 Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117662945.25491.23.camel@duffman> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> <1117657154.25491.5.camel@duffman> <6.2.1.2.2.20050601170353.057b0d60@exnane01.nane.netapp.com> <429E2821.6020602@ichips.intel.com> <1117662945.25491.23.camel@duffman> Message-ID: <6.2.1.2.2.20050601222644.05f7b510@exnane01.nane.netapp.com> At 05:55 PM 6/1/2005, Tom Duffy wrote: >I was looking at some of the ULPs. SDP sets the timeout to 20, without >explanation. DAPL has a #define (set to 20 saying it is 4 sec in a >comment). simple example? Set to 20, no reason. To me, it ends up >just being this opaque value. Maybe some pre#defines in CM would make >it clearer. Constant values are never correct in networking. The main reason for a timeout is because there isn't a cancel. Cancel is hard (impossible) because the operation is proceeding partly in hardware. The most portable thing for an upper layer to do with a timeout is to specify a "large" value. How big is large is open to debate. I think it is less of an upper layer issue and more of a user environment one. If there's a human with a ^C key, timeouts will tend to be smaller. If there is kernel mediation (e.g. an NFS mount point), timeouts can be long. Tom. From mst at mellanox.co.il Wed Jun 1 22:29:34 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 2 Jun 2005 08:29:34 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602005228.GP25321@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> Message-ID: <20050602052934.GA30946@mellanox.co.il> Quoting r. Grant Grundler : > Subject: [PATCH] rdma_lat-09 and results > > Michael, > > Good news: > My next cleanup of rdma_lat.c is working and patch is appended. > Summary of changes below. > > Bad News: > perf is about ~15 cycles slower since the last time I tested. > (Hrm...maybe it's time to cycle power on the TS90 switch again.) > > > Here's with the new rdma_lat.c: > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C > local address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > remote address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > Latency typical: 7140 cycles > Latency best : 6915 cycles > Latency worst : 52915.5 cycles > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ > > And the "client" side: > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C 10.0.0.51 > local address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > remote address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > Latency typical: 7140 cycles > Latency best : 6907 cycles > Latency worst : 94920 cycles > > > The previous set of rdma_lat results are here: > http://openib.org/pipermail/openib-general/2005-May/006721.html > > I'll guess the previous SVN verion was no older than r2229. > > > I get 7140 to 7151 for the original rdma_lat. Usually 7147.5. > I get 7132 to 7155 with my version of rdma_lat. Usually 7140. > No statistically significant differences. > Both essentially agree on the higher result. > Using "-n 10000" gave more consistent results * I changed the timestamping strategy. I used to: post tstamp poll post tstamp poll post tstamp poll post tstamp poll This meant that tstamp instruction was out of the data path, while we did polling. On the negative side, although the average (and likely median) delta between tstamps was a reliable measurement of round trip time (since there was one tstamp each roundtrip), min/max values were not measuring anything reliably: if I start polling late, two tstamps can be closer than what the wire allows for. So I changed that to: tstamp post poll tstamp post poll tstamp post poll tstamp post poll And now, on the plus side, the mix/max deltas are actually pessimistic about roundtrip times, on the minus side, we are calling tstamp on detapath, slowing it down. ~15 cycles is a bit high: of course tstamp needs to prevent instructions from being reordered across it, and so it should take on the order of the pipeline depth to perform, but then maybe its a microcode thing. I'm not against going back to the previous measurement, but we'd have to give up the min/max reporting since its an artefact. What do you say? -- MST From xma at us.ibm.com Wed Jun 1 22:47:10 2005 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 1 Jun 2005 22:47:10 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602052934.GA30946@mellanox.co.il> Message-ID: ./rdma_lat libibverbs: Warning: no userspace device-specific driver found for uverbs0 driver search path: /usr/local/lib/infiniband No IB devices found I have one node working, one node giving this error. What am I missing? I've checked uverbs0, it's the same with the working node. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Wed Jun 1 23:32:10 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 2 Jun 2005 09:32:10 +0300 Subject: [openib-general] Re: Re: [PATCH] rdma_lat-09 and results In-Reply-To: References: <20050602052934.GA30946@mellanox.co.il> Message-ID: <20050602063210.GC30946@mellanox.co.il> Quoting r. Shirley Ma : > Subject: Re: Re: [PATCH] rdma_lat-09 and results > > > ./rdma_lat > libibverbs: Warning: no userspace device-specific driver found for uverbs0 > driver search path: /usr/local/lib/infiniband > No IB devices found > > I have one node working, one node giving this error. > What am I missing? I've checked uverbs0, it's the same with the working node. > > Thanks > Shirley Ma > IBM Linux Technology Center > 15300 SW Koll Parkway > Beaverton, OR 97006-6063 > Phone(Fax): (503) 578-7638 > > libibverbs cant find the libmthca plugin. -- MST From glebn at voltaire.com Wed Jun 1 23:49:09 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Thu, 2 Jun 2005 09:49:09 +0300 Subject: [openib-general] user space verbs examples In-Reply-To: <1117667806.4490.160.camel@hal.voltaire.com> References: <1117667806.4490.160.camel@hal.voltaire.com> Message-ID: <20050602064909.GN24231@minantech.com> On Wed, Jun 01, 2005 at 07:24:18PM -0400, Hal Rosenstock wrote: > Hi Roland, > > I can run ibv_devices but ibv_asyncwatch seems to fail: > > ibv_devices > device node GUID > ------ ---------------- > mthca0 0008f10403960558 > > ibv_asyncwatch > Couldn't get context for mthca0 > > dmesg shows: > Jun 1 19:01:00 hal kernel: UCM: Created struct > Jun 1 19:01:00 hal kernel: UCM: Deleted struct > > Any idea on what is wrong ? Thanks. > Have you created /dev/infiniband/uverbs0 device? AFAIR I've got such error before configuring udev. -- Gleb. From glebn at voltaire.com Thu Jun 2 00:01:43 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Thu, 2 Jun 2005 10:01:43 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602052934.GA30946@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602052934.GA30946@mellanox.co.il> Message-ID: <20050602070143.GO24231@minantech.com> On Thu, Jun 02, 2005 at 08:29:34AM +0300, Michael S. Tsirkin wrote: > Quoting r. Grant Grundler : > > Subject: [PATCH] rdma_lat-09 and results > > > > Michael, > > > > Good news: > > My next cleanup of rdma_lat.c is working and patch is appended. > > Summary of changes below. > > > > Bad News: > > perf is about ~15 cycles slower since the last time I tested. > > (Hrm...maybe it's time to cycle power on the TS90 switch again.) > > > > > > Here's with the new rdma_lat.c: > > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C > > local address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > > remote address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > > Latency typical: 7140 cycles > > Latency best : 6915 cycles > > Latency worst : 52915.5 cycles > > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ > > > > And the "client" side: > > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C 10.0.0.51 > > local address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > > remote address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > > Latency typical: 7140 cycles > > Latency best : 6907 cycles > > Latency worst : 94920 cycles > > > > > > The previous set of rdma_lat results are here: > > http://openib.org/pipermail/openib-general/2005-May/006721.html > > > > I'll guess the previous SVN verion was no older than r2229. > > > > > > I get 7140 to 7151 for the original rdma_lat. Usually 7147.5. > > I get 7132 to 7155 with my version of rdma_lat. Usually 7140. > > No statistically significant differences. > > Both essentially agree on the higher result. > > Using "-n 10000" gave more consistent results * > > I changed the timestamping strategy. I used to: > > post > tstamp > poll > post > tstamp > poll > post > tstamp > poll > post > tstamp > poll > > This meant that tstamp instruction was out of the data path, > while we did polling. > On the negative side, although the average (and likely median) delta > between tstamps was a reliable measurement of round trip time > (since there was one tstamp each roundtrip), > min/max values were not measuring anything reliably: if I start polling > late, two tstamps can be closer than what the wire allows for. > > So I changed that to: > > > tstamp > post > poll > tstamp > post > poll > tstamp > post > poll > tstamp > post > poll > > And now, on the plus side, the mix/max deltas are actually pessimistic about > roundtrip times, on the minus side, we are calling tstamp on detapath, > slowing it down. ~15 cycles is a bit high: of course tstamp > needs to prevent instructions from being reordered across it, and so > it should take on the order of the pipeline depth to perform, but then > maybe its a microcode thing. This is what I recently saw on linux-kernel: ### begin quote of Andi Kleen > RDTSC on older Intel CPUs takes something like 6 cycles. On P4's it > takes much more, since it's decoded to a microcode MSR access. It actually seems to flush the trace cache, because Intel figured out that out of order RDTSC is probably not too useful (which is right) and the only way to ensure that on Netburst seems to stop the trace cache in its track. That can be pretty slow, we're talking 1000+ cycles here. ### end quote > > I'm not against going back to the previous measurement, but we'd have to > give up the min/max reporting since its an artefact. > What do you say? > > -- > MST > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Gleb. From mst at mellanox.co.il Thu Jun 2 00:12:03 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 2 Jun 2005 10:12:03 +0300 Subject: [openib-general] Re: Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602070143.GO24231@minantech.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602052934.GA30946@mellanox.co.il> <20050602070143.GO24231@minantech.com> Message-ID: <20050602071203.GA23899@mellanox.co.il> Quoting r. Gleb Natapov : > Subject: Re: Re: [PATCH] rdma_lat-09 and results > > On Thu, Jun 02, 2005 at 08:29:34AM +0300, Michael S. Tsirkin wrote: > > Quoting r. Grant Grundler : > > > Subject: [PATCH] rdma_lat-09 and results > > > > > > Michael, > > > > > > Good news: > > > My next cleanup of rdma_lat.c is working and patch is appended. > > > Summary of changes below. > > > > > > Bad News: > > > perf is about ~15 cycles slower since the last time I tested. > > > (Hrm...maybe it's time to cycle power on the TS90 switch again.) > > > > > > > > > Here's with the new rdma_lat.c: > > > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C > > > local address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > > > remote address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > > > Latency typical: 7140 cycles > > > Latency best : 6915 cycles > > > Latency worst : 52915.5 cycles > > > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ > > > > > > And the "client" side: > > > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C 10.0.0.51 > > > local address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > > > remote address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > > > Latency typical: 7140 cycles > > > Latency best : 6907 cycles > > > Latency worst : 94920 cycles > > > > > > > > > The previous set of rdma_lat results are here: > > > http://openib.org/pipermail/openib-general/2005-May/006721.html > > > > > > I'll guess the previous SVN verion was no older than r2229. > > > > > > > > > I get 7140 to 7151 for the original rdma_lat. Usually 7147.5. > > > I get 7132 to 7155 with my version of rdma_lat. Usually 7140. > > > No statistically significant differences. > > > Both essentially agree on the higher result. > > > Using "-n 10000" gave more consistent results * > > > > I changed the timestamping strategy. I used to: > > > > post > > tstamp > > poll > > post > > tstamp > > poll > > post > > tstamp > > poll > > post > > tstamp > > poll > > > > This meant that tstamp instruction was out of the data path, > > while we did polling. > > On the negative side, although the average (and likely median) delta > > between tstamps was a reliable measurement of round trip time > > (since there was one tstamp each roundtrip), > > min/max values were not measuring anything reliably: if I start polling > > late, two tstamps can be closer than what the wire allows for. > > > > So I changed that to: > > > > > > tstamp > > post > > poll > > tstamp > > post > > poll > > tstamp > > post > > poll > > tstamp > > post > > poll > > > > And now, on the plus side, the mix/max deltas are actually pessimistic about > > roundtrip times, on the minus side, we are calling tstamp on detapath, > > slowing it down. ~15 cycles is a bit high: of course tstamp > > needs to prevent instructions from being reordered across it, and so > > it should take on the order of the pipeline depth to perform, but then > > maybe its a microcode thing. > > > This is what I recently saw on linux-kernel: > > ### begin quote of Andi Kleen > > > RDTSC on older Intel CPUs takes something like 6 cycles. On P4's it > > takes much more, since it's decoded to a microcode MSR access. > > It actually seems to flush the trace cache, because Intel figured > out that out of order RDTSC is probably not too useful (which is right) > and the only way to ensure that on Netburst seems to stop the trace > cache in its track. That can be pretty slow, we're talking 1000+ cycles > here. > ### end quote OK, but I imagine thats probably a worst case. I certainly see nothing like a 1 usec here, and my systems are P4 based Xeons. > > I'm not against going back to the previous measurement, but we'd have to > > give up the min/max reporting since its an artefact. > > What do you say? -- MST From xma at us.ibm.com Thu Jun 2 00:41:30 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 2 Jun 2005 00:41:30 -0700 Subject: [openib-general] Re: Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602063210.GC30946@mellanox.co.il> Message-ID: > libibverbs cant find the libmthca plugin. ibv_pingpong, ibv_asyncwatch & ibv_devices work OK. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Thu Jun 2 00:53:10 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 2 Jun 2005 10:53:10 +0300 Subject: [openib-general] Re: Re: [PATCH] rdma_lat-09 and results In-Reply-To: References: <20050602063210.GC30946@mellanox.co.il> Message-ID: <20050602075310.GB23899@mellanox.co.il> Quoting r. Shirley Ma : > Subject: Re: Re: [PATCH] rdma_lat-09 and results > > > > libibverbs cant find the libmthca plugin. do ls /usr/local/lib/infiniband thats where its looking > > ibv_pingpong, ibv_asyncwatch & ibv_devices work OK. > > Shirley Ma Maybe you have multiple copies of libibverbs, so that ibv_pingpong uses a different copy? ldd `which ibv_pingpong` ldd `which rdma_lat` -- MST From xma at us.ibm.com Thu Jun 2 01:14:40 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 2 Jun 2005 01:14:40 -0700 Subject: [openib-general] Re: Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602075310.GB23899@mellanox.co.il> Message-ID: >Maybe you have multiple copies of libibverbs, so that > ibv_pingpong uses a different copy? Found the problem. They use the same copy, which is out of date. ibv_pingpong still works, rdma_lat doesn't. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From itamar at mellanox.co.il Thu Jun 2 04:24:44 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 2 Jun 2005 14:24:44 +0300 Subject: [openib-general] station hang with the last patch to infiniband/core/sa_query.c Message-ID: <91DB792C7985D411BEC300B40080D29CC35C1C@mtvex01.mtv.mtl.com> Hi Roland, Today i have tryied svn rev 2522 on my stations. One of them hang when i tried to ping the othere one. when i reverted the file infiniband/core/sa_query.c to rev 2319 (befoe your last patch) ping is working again. my system is x86 daul xeon 2G with 1GB memory I am running with some CONFIG_DEBUG_* (the .config file attached) Both gen2 stations are connected with Anafa2 switch . opensm is Mellanox IBGD 1.7.0 on another station. Itamar -------------- next part -------------- A non-text attachment was scrubbed... Name: .config Type: application/octet-stream Size: 56682 bytes Desc: not available URL: From hch at lst.de Thu Jun 2 04:08:47 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 2 Jun 2005 13:08:47 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <469958e0050601183329578260@mail.gmail.com> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050531155812.B12767@topspin.com> <469958e0050601183329578260@mail.gmail.com> Message-ID: <20050602110847.GA8060@lst.de> On Wed, Jun 01, 2005 at 06:33:14PM -0700, Caitlin Bestler wrote: > As I stated earlier, kDAPL is designed for *applications* > and was not really intended for use by the OS itself. > As such it abstracts more than just the transport. I'm not sure what you call kernel applications, but I'm pretty sure we're not interested to support anything that comes close to this term in linux kernelspace. From caitlin.bestler at gmail.com Thu Jun 2 05:22:26 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 2 Jun 2005 05:22:26 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050602110847.GA8060@lst.de> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050531155812.B12767@topspin.com> <469958e0050601183329578260@mail.gmail.com> <20050602110847.GA8060@lst.de> Message-ID: <469958e00506020522169d9759@mail.gmail.com> I hadn't heard that insmod was being removed from Linux. In fact the DAPL Plugfest successfully used kernel daemons and kdapltest to demonstrate DAT interoperability across multiple vendors: kernel to kernel, kernel to user and user to user. These are existing applications already deployed. I fail to see how you believe breaking them "improves" Linux. Software like the NFS over RDMA or SDP implementations are developed as kernel daemons outside of the official kernel first. Others develop special interest drivers and daemons that will never be of sufficient general interest to be incorporated in "the kernel". That's why insmod exists. the ability to add prototype or specialty daemon in the kernel is an ability that virtually every operating system provides. And there are KDAPL applications runnign that way *today*. Whatever runs inside the kernel source tree can be totally unrelated to that, but it shouldn't export existing DAT symbols if it changes their meaning. And existing device drivers and DAT Provider libraries should not be broken either (assuming the OS services that they *are* relying on are still there. If those changed, the developers of those modules are responsible for udpating them). On 6/2/05, Christoph Hellwig wrote: > On Wed, Jun 01, 2005 at 06:33:14PM -0700, Caitlin Bestler wrote: > > As I stated earlier, kDAPL is designed for *applications* > > and was not really intended for use by the OS itself. > > As such it abstracts more than just the transport. > > I'm not sure what you call kernel applications, but I'm pretty sure > we're not interested to support anything that comes close to this term > in linux kernelspace. > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hch at lst.de Thu Jun 2 05:56:22 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 2 Jun 2005 14:56:22 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <469958e00506020522169d9759@mail.gmail.com> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050531155812.B12767@topspin.com> <469958e0050601183329578260@mail.gmail.com> <20050602110847.GA8060@lst.de> <469958e00506020522169d9759@mail.gmail.com> Message-ID: <20050602125622.GA9787@lst.de> On Thu, Jun 02, 2005 at 05:22:26AM -0700, Caitlin Bestler wrote: > I hadn't heard that insmod was being removed from Linux. No one claimed that. > In fact the DAPL Plugfest successfully used kernel > daemons and kdapltest to demonstrate DAT interoperability > across multiple vendors: kernel to kernel, kernel to user > and user to user. And that matters for this discussion exactly how? > These are existing applications already deployed. I fail to > see how you believe breaking them "improves" Linux. Again, what application. Also who says we want to "improve" Linux to make it easier to use for everyone? We're certainly not interested in supporting freeloaders. > Software like the NFS over RDMA or SDP implementations > are developed as kernel daemons outside of the official > kernel first. Others develop special interest drivers and > daemons that will never be of sufficient general interest > to be incorporated in "the kernel". Okay, after these rants we come closer to what you actually mean with "kernel applications". We're certainly interested in supporting filesystems or networking protocol, but like all other in-kernel APIs we don't care at all about out of tree projects. Especially as filesystems and network protocols are inherently part of the bigger kernel picture and benefit from integration with other parts of the kernel. Not even mentioning license implications as I very strongly doubt you can claim either of it is not a derived work. From itamar at mellanox.co.il Thu Jun 2 06:28:43 2005 From: itamar at mellanox.co.il (Itamar) Date: Thu, 2 Jun 2005 16:28:43 +0300 Subject: [openib-general] [patch][kdapl] enable kdapltest -T P Message-ID: <20050602132843.GA3760@mellanox.co.il> in order to enable kdapltest -T P i needed to remark attr that are not set by openib gen2 1) ia_attr.max_evd_qlen 2) ia_attr.max_rdma_read_per_ep_in 3) ia_attr.max_rdma_read_per_ep_out also there was bug in kdapltest memory registration (file dapl_bpool.c) and there was bug where we free memory and then we try to print part of the memory i can run now kdapltest -T P for both RR and RW enable kdapltest -T P fix bug in memory registration Signed-off-by: Itamar Rabenstein Index: test/dapl_performance_util.c =================================================================== --- test/dapl_performance_util.c (revision 2522) +++ test/dapl_performance_util.c (working copy) @@ -76,17 +76,20 @@ } pipeline_len = DT_min ( - DT_min (test_ptr->cmd->num_iterations, - test_ptr->cmd->pipeline_len), - DT_min (test_ptr->ia_attr.max_dto_per_ep, - test_ptr->ia_attr.max_evd_qlen)); + DT_min (test_ptr->cmd->num_iterations, + test_ptr->cmd->pipeline_len), + test_ptr->ia_attr.max_dto_per_ep); +/* DT_min (test_ptr->ia_attr.max_dto_per_ep, */ +/* test_ptr->ia_attr.max_evd_qlen) */ +/* ); */ if ( RDMA_READ == test_ptr->cmd->op.transfer_type ) - { - pipeline_len = DT_min (pipeline_len, - test_ptr->ia_attr.max_rdma_read_per_ep_in); - pipeline_len = DT_min (pipeline_len, - test_ptr->ia_attr.max_rdma_read_per_ep_out); + { +/* max_rdma_read_per_ep_in max_rdma_read_per_ep_out are not set by gen2 */ +/* pipeline_len = DT_min (pipeline_len, */ +/* test_ptr->ia_attr.max_rdma_read_per_ep_in); */ +/* pipeline_len = DT_min (pipeline_len, */ +/* test_ptr->ia_attr.max_rdma_read_per_ep_out); */ } test_ptr->reqt_evd_length = pipeline_len; @@ -203,7 +206,7 @@ DAT_HANDLE_NULL, /* rmr */ DT_PERF_SYNC_BUFF_SIZE, 2, /* 2 RMIs */ - 255, /* FIXME should query for this */ + 256, /* FIXME should query for this */ FALSE, FALSE); if ( !test_ptr->ep_context.bp ) @@ -406,8 +409,8 @@ } } + DT_Tdep_PT_Debug (1, (phead,"Test[" F64x "]: cleanup is done\n", test_ptr->base_port)); DT_MemListFree (test_ptr->pt_ptr, test_ptr); - DT_Tdep_PT_Debug (1, (phead,"Test[" F64x "]: cleanup is done\n", test_ptr->base_port)); return status; } Index: test/dapl_bpool.c =================================================================== --- test/dapl_bpool.c (revision 2522) +++ test/dapl_bpool.c (working copy) @@ -388,7 +388,7 @@ u64 DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index) { - return ( bpool_ptr->reg_addr + index * bpool_ptr->seg_size ); + return (virt_to_phys(DT_Bpool_GetBuffer(bpool_ptr,index))); } /*****************************************************************************/ -- Itamar From jlentini at netapp.com Thu Jun 2 07:17:57 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 10:17:57 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117576362.1373.16.camel@duffman> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1117576362.1373.16.camel@duffman> Message-ID: On Tue, 31 May 2005, Tom Duffy wrote: > On Tue, 2005-05-31 at 14:17 -0400, James Lentini wrote: >> Here's the specification's exact description: >> >> timeout: Duration of time, in microseconds, that a consumer waits for >> connection establishment. The value of DAT_TIMEOUT_INFINITE >> represents no timeout, indefinite wait. Values must be >> positive. > > Let me make sure I got this right: timeout is in µs (10^-6 seconds), not > ms (10^-3 seconds). If so, I am off by 3 orders of magnitude in my > calculation. Right? Correct, the value was intended to be in microseconds. >> My perspective is that we are not implementing this API for a real >> time operating system and therefore should take a fuzzy view of time. > > Trust me, it is going to fuzzy what with the mechanism IB uses to encode > timeouts. > > BTW, what do you think would be a good test case to make sure the new > code is working as intended? dapltest could be updated to allow the timeout value to be specified on the command line. From jlentini at netapp.com Thu Jun 2 07:36:50 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 10:36:50 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers insteadof own In-Reply-To: <1117576517.1373.19.camel@duffman> References: <1117576517.1373.19.camel@duffman> Message-ID: On Tue, 31 May 2005, Tom Duffy wrote: > On Tue, 2005-05-31 at 14:34 -0700, Sean Hefty wrote: >>> Sean, >>> >>> Is there any way of requesting an infinite number of retries? >> >> There is not, but nothing prevents a user from simply re-issuing a request >> after it times out. > > Infinite retries inside the kernel does not sound like a good idea. How > would you break it? At least we should have some sort of exponential > backoff to prevent flooding the network. We want a value that the consumer can pass as the timeout that will give the protocol enough time to connect regardless of network topology. If the value is a fixed period of time, say 500 us, there will be some network configuration that it won't work on. The infinite value allows a user to give the CM protocol all the time it needs to work. If it fails for some other reason (like the remote node is down) then the connection attempt should fail. So I think we should do a few things: - rename this constant - define its value as 0 - upon seeing it, assign a CM timeout value that allows for a the protocol to complete We don't need to do all that on this first pass though. Let's just get the timeout value working in the general case and then we can worry about DAT_TIMEOUT_INFINITE and what it means. james From mst at mellanox.co.il Thu Jun 2 07:43:05 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 2 Jun 2005 17:43:05 +0300 Subject: [openib-general] Re: [PATCH] mthca: fix registration for giant MRs In-Reply-To: <52r7fl6bxa.fsf@topspin.com> References: <52r7fl6bxa.fsf@topspin.com> Message-ID: <20050602144305.GM23899@mellanox.co.il> Quoting r. Roland Dreier : > Subject: [PATCH] mthca: fix registration for giant MRs > > Here's a patch that allows mthca to break up registration of giant > userspace MRs into multiple firmware commands. The net effect of this > is that there should no longer be any limit (beyond physical memory) > on the size of MRs that userspace can register. > > Arlin -- could you give this a shot and see if it works for you? > > MST -- can you eyeball this patch and see if it looks OK? > > If no one sees problems with this, I'll commit it soon. > > - R. Looks good. A couple of comments: ------ 1) --- mthca/mthca_mr.c (revision 2468) +++ mthca/mthca_mr.c (working copy) +struct mthca_mtt { + int order; + u32 first_seg; + int fmr; +}; + And later in mthca_alloc_mtt: + mtt->first_seg = __mthca_alloc_mtt(dev, mtt->order, + fmr ? dev->mr_table.fmr_mtt_buddy : + &dev->mr_table.mtt_buddy); I'd like to suggest we keep passing struct mthca_buddy *to mthca_alloc_mtt, instead of passing around and keeping in memory the binary fmr flag, since all that this flag does is select the right allocator, and callers of mthca_alloc_mtt/mthca_free_mtt already know which it is. mthca_alloc_mtt(dev, order, dev->mr_tabel.fmr_mtt_buddy) is in my opinion clearer than mthca_alloc_mtt(dev, order, 1) When I implemented fmr support I took pains to ensure on memfree at least fmr and mr mtts are exactly the same, and I have plans to make it so on all 64 bit OSes, and also to use that to speed up allocations, so if possible lets not break that with the fmr flag. ------ 2) + for (i = MTHCA_MTT_SEG_SIZE / 8; i < size; i <<= 1) + ++mtt->order; At some point it'd be nice to support zero-sized regions, but meanwhile, isnt this better written as mtt->order = fls(max(size, MTHCA_MTT_SEG_SIZE / 8) - 1); ------ 3) > static int mthca_dereg_mr(struct ib_mr *mr) > --- mthca/mthca_provider.h (revision 2470) > +++ mthca/mthca_provider.h (working copy) > @@ -62,18 +62,18 @@ struct mthca_ucontext { > struct mthca_user_db_table *db_tab; > }; > > +struct mthca_mtt; > + > struct mthca_mr { > - struct ib_mr ibmr; > - int order; > - u32 first_seg; > + struct ib_mr ibmr; > + struct mthca_mtt *mtt; > }; Why dont we keep mthca_mtt by instance in struct mthca_mr, like this: struct mthca_mr { struct ib_mr ibmr; struct mthca_mtt mtt; }; Saves keeping around an extra small chunk of memory that we need to kfree/kmalloc/check. mthca_alloc_mtt could get struct mthca_mtt* and use that. -- MST From jlentini at netapp.com Thu Jun 2 08:01:07 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 11:01:07 -0400 (EDT) Subject: [openib-general] RE: [PATCH][kdapl] fix fatal bug in triger the evd upcall In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35BFD@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35BFD@mtvex01.mtv.mtl.com> Message-ID: Sorry about that. Should be fixed now. On Wed, 1 Jun 2005, Itamar Rabenstein wrote: itamar> Hi James, itamar> itamar> you replied with "Committed in revision 2514." itamar> But I am checking and you have committed my patch with a change. itamar> Please whenever you decide to change a patch please reply with a reason or itamar> at least say that you have change it so other can make a review for it. itamar> itamar> any way why did you change the patch? as I said in my mail a return value itamar> from dapl_evd_dequeue can be itamar> queue empty. I don't think we should print "dapl_evd_dequeue failed" for itamar> empty queue it is a good return value. itamar> itamar> Itamar itamar> itamar> > -----Original Message----- itamar> > From: James Lentini [mailto:jlentini at netapp.com] itamar> > Sent: Tuesday, May 31, 2005 6:32 PM itamar> > To: Itamar itamar> > Cc: James.Lentini at netapp.com; openib-general at openib.org itamar> > Subject: Re: [PATCH][kdapl] fix fatal bug in triger the evd upcall itamar> > itamar> > itamar> > itamar> > Committed in revision 2514. itamar> > itamar> > On Sun, 29 May 2005, Itamar wrote: itamar> > itamar> > itamar> Hi James, itamar> > itamar> itamar> > itamar> This patch fix a fatal bug that exist in current itamar> > lastet bits in kdapl (svn rev 2507) itamar> > itamar> As you can see we need to triger the upcall when itamar> > dapl_evd_dequeue return with good status itamar> > itamar> and quit the method when dapl_evd_dequeue return with itamar> > non zero status which mean queue is empty. itamar> > itamar> In the current bits no kdapltest can run even the itamar> > simple quit test. itamar> > itamar> itamar> > itamar> Please in the future before you commit changes to the itamar> > svn run a simple regression. itamar> > itamar> Any way with this patch the code is working again. itamar> > itamar> itamar> > itamar> fix fatal bug in triger the evd upcall itamar> > itamar> Signed-off-by: Itamar Rabenstein itamar> > itamar> itamar> > itamar> Index: dapl_cno_util.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- dapl_cno_util.c (revision 2509) itamar> > itamar> +++ dapl_cno_util.c (working copy) itamar> > itamar> @@ -115,12 +115,8 @@ itamar> > itamar> itamar> > itamar> for (;;) { itamar> > itamar> status = itamar> > dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); itamar> > itamar> - if (DAT_SUCCESS == status) { itamar> > itamar> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, itamar> > itamar> - "dapl_evd_dequeue itamar> > failed: %x\n", status); itamar> > itamar> + if (DAT_SUCCESS != status) itamar> > itamar> return; itamar> > itamar> - } itamar> > itamar> - itamar> > itamar> itamar> > cno->cno_upcall.upcall_func(cno->cno_upcall.instance_data, itamar> > itamar> &event, FALSE); itamar> > itamar> } itamar> > itamar> -- itamar> > itamar> Itamar itamar> > itamar> itamar> > itamar> From jlentini at netapp.com Thu Jun 2 08:16:53 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 11:16:53 -0400 (EDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1117649626.29566.2.camel@duffman> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> <1117649626.29566.2.camel@duffman> Message-ID: On Wed, 1 Jun 2005, Tom Duffy wrote: > On Wed, 2005-06-01 at 12:04 +0200, Christoph Hellwig wrote: >> That beeing said, one of the first thing you should get rid of if you >> want to be able to take code from kdapl to the generic rdma code is way >> it deals with handles. The kdapl code gives up language type safety >> just to add its own bandaided type system below. Please always pass the >> real struct pointer around, or at least the generic struct type which >> might be embedded into a bigger structure, and get rid of all the magic >> number checking and magic macros that expect special members at the >> beginning of all objects behind these handles. > > I absolutely agree with this and was planning on implementing said > changes in the near future. > > James, can I get a "hell yeah" or a "no freaking way" from you before I > go off and start writing patches? > > -tduffy This could be an improvement. We just need to be careful that we don't expose anything transport specific. Off the top of my head, I can think of one way to do this: - in dat.h, create a public structure for each object type: struct dat_ep { struct dat_provider *provider; }; - in the transport provider (dapl.h) have a private structure that contains the public one: struct dapl_ep { struct dat_ep dat_ep; ... }; Is that the sort of thing you were thinking of? I'll do some research here and find out why the original mechanism was chosen. From tduffy at sun.com Thu Jun 2 08:27:14 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 02 Jun 2005 08:27:14 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> <1117649626.29566.2.camel@duffman> Message-ID: <1117726034.15225.1.camel@duffman> On Thu, 2005-06-02 at 11:16 -0400, James Lentini wrote: > - in dat.h, create a public structure for each object type: > > struct dat_ep { > struct dat_provider *provider; > }; > > - in the transport provider (dapl.h) have a private structure that > contains the public one: > > struct dapl_ep { > struct dat_ep dat_ep; > ... > }; > > Is that the sort of thing you were thinking of? Yes, exactly. Then you can create small inline functions that would cast back and forth. The other "issue" right now is that dapl has a header struct that needs to come first in all the structs. So, that would need to be changed. > I'll do some research here and find out why the original mechanism was > chosen. Cool. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Thu Jun 2 08:36:44 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 11:36:44 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <20050601183544.GA24939@lst.de> References: <1117650224.29566.6.camel@duffman> <20050601183544.GA24939@lst.de> Message-ID: On Wed, 1 Jun 2005, Christoph Hellwig wrote: hch> On Wed, Jun 01, 2005 at 11:23:44AM -0700, Tom Duffy wrote: hch> > This patch cleans up dat/ a bit, removing unneeded parenthesis, hch> > gratuitous comments, and a few whitespace cleanups. Please apply. hch> hch> btw, I think most of the code under dat/ should just go away ASAP. hch> Just have a simple linked list of providers with trivial hch> registration/unregisteration/find_get/put functions like all the other hch> subsystems We have registration/unregistration functions for providers: dat_registry_add_provider dat_registry_remove_provider Consumer's use dat_ia_openv dat_ia_close dat_registry_list_providers Is this what you mean? hch> and kill all the dictionaty mess The dictionary is implemented with a simple linked list. We could look into collapsing the code in api.c, dr.c and dictionary.c further. hch> and provider vs provider_info braindamage. What do you mean by provider vs provider_info? Are you suggesting adding the information in the dat_provider_info structure to the dat_provider structure? hch> hch> _______________________________________________ hch> openib-general mailing list hch> openib-general at openib.org hch> http://openib.org/mailman/listinfo/openib-general hch> hch> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general hch> From jlentini at netapp.com Thu Jun 2 08:40:16 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 11:40:16 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117654831.29566.21.camel@duffman> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1117576362.1373.16.camel@duffman> <1117654831.29566.21.camel@duffman> Message-ID: On Wed, 1 Jun 2005, Tom Duffy wrote: > On Tue, 2005-05-31 at 14:52 -0700, Tom Duffy wrote: >> On Tue, 2005-05-31 at 14:17 -0400, James Lentini wrote: >>> Here's the specification's exact description: >>> >>> timeout: Duration of time, in microseconds, that a consumer waits for >>> connection establishment. The value of DAT_TIMEOUT_INFINITE >>> represents no timeout, indefinite wait. Values must be >>> positive. >> >> Let me make sure I got this right: timeout is in µs (10^-6 seconds), not >> ms (10^-3 seconds). If so, I am off by 3 orders of magnitude in my >> calculation. Right? > > This is from DT_fft_connect() in test/dapltest/test/dapl_fft_util.c: > > /* attempt to connect, timeout = 10 secs */ > rc = dat_ep_connect (conn->ep_handle, conn->remote_netaddr, > SERVER_PORT_NUMBER, 10*1000, 0, (void *)0, > DAT_QOS_BEST_EFFORT, DAT_CONNECT_DEFAULT_FLAG); > DT_assert_dat (phead, rc == DAT_SUCCESS); > > leading me to believe we are talking about milliseconds. Or is this a > bug? That is a bug. From jlentini at netapp.com Thu Jun 2 08:46:33 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 11:46:33 -0400 (EDT) Subject: [openib-general] [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> References: <1117655324.19908.3.camel@duffman> <6.2.1.2.2.20050601160713.051749f0@exnane01.nane.netapp.com> Message-ID: On Wed, 1 Jun 2005, Talpey, Thomas wrote: > At 03:48 PM 6/1/2005, Tom Duffy wrote: >> This patch takes into account some of the feedback by Sean, it sets the >> retry to 0, so it will do the timeout right. It also stops the >> conversion of ms to ib at 63 since 64 overflows. I have tested this >> version using kdapltest in quit, performance, and transaction modes. >> >> James, do you think it good enough to apply? > > Do any of these tests intentionally trigger timer events? I don't think > they do... We don't have any tests that do that. > > BTW :-) surely you can think of an even more colorful name than this! > >> static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) > > Shouldn't this routine live in an IB-specific area, not core DAPL? It is in IB-specific code. From iod00d at hp.com Thu Jun 2 08:59:17 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 2 Jun 2005 08:59:17 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602052934.GA30946@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602052934.GA30946@mellanox.co.il> Message-ID: <20050602155917.GA29040@esmail.cup.hp.com> On Thu, Jun 02, 2005 at 08:29:34AM +0300, Michael S. Tsirkin wrote: ... > I changed the timestamping strategy. I used to: ... > This meant that tstamp instruction was out of the data path, > while we did polling. > On the negative side, although the average (and likely median) delta > between tstamps was a reliable measurement of round trip time > (since there was one tstamp each roundtrip), > min/max values were not measuring anything reliably: if I start polling > late, two tstamps can be closer than what the wire allows for. ok. ... > And now, on the plus side, the mix/max deltas are actually pessimistic about > roundtrip times, on the minus side, we are calling tstamp on datapath, > slowing it down. ~15 cycles is a bit high: of course tstamp > needs to prevent instructions from being reordered across it, and so > it should take on the order of the pipeline depth to perform, but then > maybe its a microcode thing. Yes, that makes sense now. get_cycles() on ia64 is expected to cost 36 cycles and we are reporting roughly 1/2 of that. > I'm not against going back to the previous measurement, but we'd have to > give up the min/max reporting since its an artefact. > What do you say? Overall, 15 cycles isn't a big deal when the total is around 7000. I was looking for differences thinking it would be something in the SW stack (libibverbs or libmthca). But changes in the test are obviously going to show up too and that's ok if they can be explained. min/max seem like useful numbers if someone wants to get an idea of best/worst case performance...but maybe something like standard deviation is a more useful metric to extract. We should seperate "start up" times from "streaming" times. I will re-run ibv_pingpong and collect similar measure for non-stream Request-Response traffic. Two more ideas: 1) call get_cycles() twice in a row and substract the difference from each measurement *after* all measurements are taken. 2) report 1/2 the cost of get_cycles() and leave the measurements alone. thanks, grant From roland at topspin.com Thu Jun 2 09:05:14 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 02 Jun 2005 09:05:14 -0700 Subject: [openib-general] Re: station hang with the last patch to infiniband/core/sa_query.c In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C1C@mtvex01.mtv.mtl.com> (Itamar Rabenstein's message of "Thu, 2 Jun 2005 14:24:44 +0300") References: <91DB792C7985D411BEC300B40080D29CC35C1C@mtvex01.mtv.mtl.com> Message-ID: <524qcg3ft1.fsf@topspin.com> Thanks, I found a problem with that patch and checked in a fix. Can you let me know if it works better now? - R. From roland at topspin.com Thu Jun 2 09:07:55 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 02 Jun 2005 09:07:55 -0700 Subject: [openib-general] Re: [PATCH] mthca: fix registration for giant MRs In-Reply-To: <20050602144305.GM23899@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 2 Jun 2005 17:43:05 +0300") References: <52r7fl6bxa.fsf@topspin.com> <20050602144305.GM23899@mellanox.co.il> Message-ID: <52zmu82144.fsf@topspin.com> Michael> I'd like to suggest we keep passing struct mthca_buddy Michael> *to mthca_alloc_mtt, instead of passing around and Michael> keeping in memory the binary fmr flag, since all that Michael> this flag does is select the right allocator, and callers Michael> of mthca_alloc_mtt/mthca_free_mtt already know which it Michael> is. Makes sense, I'll add that change. Michael> + for (i = MTHCA_MTT_SEG_SIZE / 8; i < size; i <<= 1) ++mtt->order; Michael> At some point it'd be nice to support zero-sized regions, Michael> but meanwhile, isnt this better written as mtt->order = fls(max(size, MTHCA_MTT_SEG_SIZE / 8) - 1); Again I'll do that. Michael> Why dont we keep mthca_mtt by instance in struct Michael> mthca_mr, like this: struct mthca_mr { struct ib_mr ibmr; Michael> struct mthca_mtt mtt; }; Michael> Saves keeping around an extra small chunk of memory that Michael> we need to kfree/kmalloc/check. mthca_alloc_mtt could Michael> get struct mthca_mtt* and use that. My reasoning was that when we support shared memory regions we'll be able to share MTTs just by adding a reference cound to the mtt structure. - R. From jlentini at netapp.com Thu Jun 2 09:31:36 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 12:31:36 -0400 (EDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1117726034.15225.1.camel@duffman> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> <1117649626.29566.2.camel@duffman> <1117726034.15225.1.camel@duffman> Message-ID: On Thu, 2 Jun 2005, Tom Duffy wrote: > On Thu, 2005-06-02 at 11:16 -0400, James Lentini wrote: >> - in dat.h, create a public structure for each object type: >> >> struct dat_ep { >> struct dat_provider *provider; >> }; >> >> - in the transport provider (dapl.h) have a private structure that >> contains the public one: >> >> struct dapl_ep { >> struct dat_ep dat_ep; >> ... >> }; >> >> Is that the sort of thing you were thinking of? > > Yes, exactly. Then you can create small inline functions that would > cast back and forth. > > The other "issue" right now is that dapl has a header struct that needs > to come first in all the structs. So, that would need to be changed. I think that issue will go away if we implement the above proposal. The dapl objects all place the dapl_header structure first so that the struct dat_provider pointer will be the first item in memory. This allows the dat_* functions to cast the user's handle to a struct dat_provider pointer and then call the appropriate provider function. >> I'll do some research here and find out why the original mechanism was >> chosen. > > Cool. > > -tduffy > > From jim.ryan at intel.com Thu Jun 2 09:48:14 2005 From: jim.ryan at intel.com (Ryan, Jim) Date: Thu, 2 Jun 2005 09:48:14 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA:Convergence on common RDMAAPIs and ULPs for Linux Message-ID: Please see some comments below which I'm offering on my own but I hope speaking fairly and responsibly for the "steering committee" (more correctly the Board) of OpenIB. Jim Ryan, Chairman, OpenIB ________________________________ From: openib-general-bounces at openib.org [mailto:openib-general-bounces at openib.org] On Behalf Of Venkata Jagana Sent: Tuesday, May 31, 2005 4:44 PM To: Grant Grundler Cc: Sukanta ganguly; rdma-developers at lists.sourceforge.net; openib-general at openib.org; rdma-developers-admin at lists.sourceforge.net Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA:Convergence on common RDMAAPIs and ULPs for Linux > I've been advocating rdmaconsortium folks submit patches > against openib.org for several reasons: Probably, you meant openrdma.org opensource project but not a standards setting body (i.e. RDMA consortium - http://www.rdmaconsortium.org/home ) :) > 1) start with a code base that works > 2) start with a code base that is already upstream > 3) get advice/guidance from people who know how to collaborate > in an open source environment. > > I thought (2) was the most important...but now I have to wonder > if it's really (3). You are mistaken. I know people in the OpenRDMA community have worked with the opensource projects before and they know how to play and collaborate in an open source environment. The early part of the work in openrdma is in fact, a true example of that effort (which you may disagree with but having worked with several other opensource projects and with OpenIB, we have solved the issues which other projects including OpenIB have faced) and the next phase of work which is of course the code development, a key aspect of broader community effort. I think we are diverging from the real issue - the fundamental differences in the views of each community in how we can solve this common problem of supporting multiple RDMA fabrics, which is what we need to focus on. > > > Just having OpenIB subsume control of anything iWARP or impose only > > DAPL for all RDMA infrastructure because it just happens to be there today > > seems rather stifling. Just stating that some OpenIB steering group is > > somehow empowered to decide this for Linux is also rather strange. > > AFAICT the openib.org steering group doesn't control the content > of the svn.openib.org source tree. It manages things like web content, > overall charter, etc .... Don't agree. If you have read the email thread on this discussion, you would find that steering committee need to decide whether openIB should work on including the support for iWARP. Not that I am supporting this idea -:) In the opensource world, developers should/will have the freedom to add what they want to do but of course, the acceptance of their contributions into mainline is completely a different matter. [Ryan, Jim] The Board of OpenIB is considering a proposal to formally support iWARP in addition to InfiniBand. I believe formally expanding our charter would signal developers and industry participants that OpenIB is undertaking this work as a complement to what we're doing with InfiniBand. A couple of additional points. First, no one is saying OpenIB should "subsume control of anything iWARP" - that's quite a stretch from OpenIB entertaining code submissions for iWARP as part of our Linux development activities. Second, the statement the "steering committee" doesn't control the content of the source tree is correct in this arena. The steering committee maintains the web site and other administrative activities, including, potentially, sending the message to the industry of our interest and support of iWARP. It does not control the activity of the open community - I don't think we could if we wanted to, and we don't. To be as clear as possible, if the steering committee were to vote against this proposed expanded charter, the open community participants may well proceed with these activities on their own. .....hope that helps, Jim Thanks Venkat -------------- next part -------------- An HTML attachment was scrubbed... URL: From caitlin.bestler at gmail.com Thu Jun 2 09:49:51 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 2 Jun 2005 09:49:51 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1117726034.15225.1.camel@duffman> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> <1117649626.29566.2.camel@duffman> <1117726034.15225.1.camel@duffman> Message-ID: <469958e00506020949356275b9@mail.gmail.com> On 6/2/05, Tom Duffy wrote: > On Thu, 2005-06-02 at 11:16 -0400, James Lentini wrote: > > - in dat.h, create a public structure for each object type: > > > > struct dat_ep { > > struct dat_provider *provider; > > }; > > > > - in the transport provider (dapl.h) have a private structure that > > contains the public one: > > > > struct dapl_ep { > > struct dat_ep dat_ep; > > ... > > }; > > > > Is that the sort of thing you were thinking of? > > Yes, exactly. Then you can create small inline functions that would > cast back and forth. > They were actually #defines in the DAT 1.0 headers, but I agree that inline functions are better. At the time the #defines were adopted there was still a concern about compilers that did not implement inlining efficiently. I believe that concern is no longer valid in general, and it is especially not relevant when gcc is known to be available. > The other "issue" right now is that dapl has a header struct that needs > to come first in all the structs. So, that would need to be changed. > That is what enables the use of the method table. It allows the in-line function to know the actual method to call for 'dat_ep_post_send' on this specific EP without knowing at compile time which Provider code is being invoked. The DAT 1.0 headers included some #define options to let that be hard-coded in environments where it was known that there was only a single provider. From mst at mellanox.co.il Thu Jun 2 09:50:53 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 2 Jun 2005 19:50:53 +0300 Subject: [openib-general] Re: [PATCH] mthca: fix registration for giant MRs In-Reply-To: <52zmu82144.fsf@topspin.com> References: <52r7fl6bxa.fsf@topspin.com> <20050602144305.GM23899@mellanox.co.il> <52zmu82144.fsf@topspin.com> Message-ID: <20050602165053.GA28229@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] mthca: fix registration for giant MRs > > Michael> Why dont we keep mthca_mtt by instance in struct > Michael> mthca_mr, like this: struct mthca_mr { struct ib_mr ibmr; > Michael> struct mthca_mtt mtt; }; > > Michael> Saves keeping around an extra small chunk of memory that > Michael> we need to kfree/kmalloc/check. mthca_alloc_mtt could > Michael> get struct mthca_mtt* and use that. > > My reasoning was that when we support shared memory regions we'll be > able to share MTTs just by adding a reference cound to the mtt > structure. > > - R. > Good idea. -- MST From caitlin.bestler at gmail.com Thu Jun 2 09:56:51 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 2 Jun 2005 09:56:51 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C15@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C15@mtvex01.mtv.mtl.com> Message-ID: <469958e0050602095659d486ab@mail.gmail.com> Why is it that you believe that the DAT registry does not support plug and play? The interface was most specifically designed to allow that. When a device is plugged in, the driver is loaded by existing OS mechanisms. It can then load the provider code (if needed). Whenever the provider module is loaded it registers itself with the registry, and it unregisters itself before it is unloaded. Keep in mind that loading/unloading DAT Provider is *not* synonymous with loading/unloading drivers. In fact I believe the intent is to have a single provider that supports multiple devices. Such a provider would simply register itself when it was loaded. I could see adding new interfaces to track which devices are loaded/unloaded for use *by* the DAT provider, but I do not see how that changes the function of the DAT registry, which is to support multiple DAT Providers. On 6/1/05, Itamar Rabenstein wrote: > I think that if we want to support pci hot plug we need to change the kdapl > IF > to be like something like the gen2 IF. > a link lins of the providers with register/unregister client functions > which will register a CB function. > we only need to have 4 functions > register_client , unregister_client register_provider , unregister_provider > > Itamar > > > -----Original Message----- > > From: Christoph Hellwig [mailto:hch at lst.de] > > Sent: Wednesday, June 01, 2005 9:36 PM > > To: Tom Duffy > > Cc: openib-general at openib.org > > Subject: Re: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more > > > > > > On Wed, Jun 01, 2005 at 11:23:44AM -0700, Tom Duffy wrote: > > > This patch cleans up dat/ a bit, removing unneeded parenthesis, > > > gratuitous comments, and a few whitespace cleanups. Please apply. > > > > btw, I think most of the code under dat/ should just go away ASAP. > > Just have a simple linked list of providers with trivial > > registration/unregisteration/find_get/put functions like all the other > > subsystems and kill all the dictionaty mess and provider vs > > provider_info braindamage. > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From iod00d at hp.com Thu Jun 2 10:07:34 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 2 Jun 2005 10:07:34 -0700 Subject: [openib-general] [PATCH] rdma_lat-09 and results In-Reply-To: <20050602005228.GP25321@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> Message-ID: <20050602170734.GB29040@esmail.cup.hp.com> On Wed, Jun 01, 2005 at 05:52:28PM -0700, Grant Grundler wrote: > Here's with the new rdma_lat.c: Here are some pfmon31 perf results for the new rdma_lat. The first set is just L3 Cache misses. The second set shows all cache misses. The sample rate on the second set was a bit high - note the median cycle count was ~60 cycles higher than normal. Unfortunately pfmon doesn't look up symbols in shared libraries. Shared libraries live in 0x2... address space on ia64. I'll see if I can dig up the top hitters by hand. 0x20000000002d7021 in particular is "of interest". And PLEASE, if you reply, please delete quoted text you are not responding to from your reply. I'm getting tired of wading through 5 pages of quotes to get to a 3 line comment. grant ====================== FIRST SET - L3 CACHE MISSES =============== grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ taskset 1 ./rdma_lat -C -n 100000 local address: LID 0x2d QPN 0x60406 PSN 0xd2cfc8 RKey 0x240434 VAddr 0x6000000000014001 remote address: LID 0x2b QPN 0x60406 PSN 0x91ea3b RKey 0x240434 VAddr 0x6000000000014001 Latency typical: 7140 cycles Latency best : 6881 cycles Latency worst : 293444 cycles grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ pfmon31 --us-c --cpu-list=1 --resolve-addr --smpl-module=dear-hist-itanium2 -e data_ear_cache_lat64 --long-smpl-periods=20 -- ./rdma_lat -C 10.0.0.51 -n 100000 added event set 0 local address: LID 0x2b QPN 0x60406 PSN 0x91ea3b RKey 0x240434 VAddr 0x6000000000014001 remote address: LID 0x2d QPN 0x60406 PSN 0xd2cfc8 RKey 0x240434 VAddr 0x6000000000014001 Latency typical: 7136.5 cycles Latency best : 6870 cycles Latency worst : 293396 cycles # total_samples 1937 # instruction addr view # sorted by count # showing per per distinct value # %L2 : percentage of L1 misses that hit L2 # %L3 : percentage of L1 misses that hit L3 # %RAM : percentage of L1 misses that hit memory # L2 : 5 cycles load latency # L3 : 14 cycles load latency # sampling period: 20 # #count %self %cum %L2 %L3 %RAM instruction addr 630 32.52% 32.52% 0.00% 0.00% 100.00% 0x200000000032ba80 0x200000000032ba80 612 31.60% 64.12% 0.00% 0.00% 100.00% 0x4000000000003f10 main+0x1a60 568 29.32% 93.44% 0.00% 0.00% 100.00% 0x4000000000003f60 main+0x1ab0 60 3.10% 96.54% 0.00% 0.00% 100.00% 0x4000000000003fd1 main+0x1b21 36 1.86% 98.40% 0.00% 0.00% 100.00% 0x40000000000040c0 main+0x1c10 10 0.52% 98.92% 0.00% 0.00% 100.00% 0x2000000000141081 0x2000000000141081 8 0.41% 99.33% 0.00% 0.00% 100.00% 0x40000000000040b0 main+0x1c00 8 0.41% 99.74% 0.00% 0.00% 100.00% 0x2000000000141061 0x2000000000141061 3 0.15% 99.90% 0.00% 0.00% 100.00% 0x20000000002d7021 0x20000000002d7021 1 0.05% 99.95% 0.00% 0.00% 100.00% 0x20000000001316f0 0x20000000001316f0 1 0.05% 100.00% 0.00% 0.00% 100.00% 0x4000000000002470 cycles_compare # level 0 : counts=0 avg_cycles=0.0ms 0.00% # level 1 : counts=0 avg_cycles=0.0ms 0.00% # level 2 : counts=1937 avg_cycles=0.0ms 100.00% approx cost: 0.0s ====================== SECOND SET - ALL CACHE LEVELS =============== grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ taskset 1 ./rdma_lat -C -n 100000 local address: LID 0x2d QPN 0x80406 PSN 0xbb769a RKey 0x300434 VAddr 0x6000000000014001 remote address: LID 0x2b QPN 0x80406 PSN 0x60ce3b RKey 0x300434 VAddr 0x6000000000014001 Latency typical: 7203.5 cycles Latency best : 6881.5 cycles Latency worst : 297164 cycles grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ pfmon31 --us-c --cpu-list=1 --resolve-addr --smpl-module=dear-hist-itanium2 -e data_ear_cache_lat4 --long-smpl-periods=20 -- ./rdma_lat -C 10.0.0.51 -n 100000 added event set 0 local address: LID 0x2b QPN 0x80406 PSN 0x60ce3b RKey 0x300434 VAddr 0x6000000000014001 remote address: LID 0x2d QPN 0x80406 PSN 0xbb769a RKey 0x300434 VAddr 0x6000000000014001 Latency typical: 7196.5 cycles Latency best : 6896.5 cycles Latency worst : 297009 cycles # total_samples 231428 # instruction addr view # sorted by count # showing per per distinct value # %L2 : percentage of L1 misses that hit L2 # %L3 : percentage of L1 misses that hit L3 # %RAM : percentage of L1 misses that hit memory # L2 : 5 cycles load latency # L3 : 14 cycles load latency # sampling period: 20 # #count %self %cum %L2 %L3 %RAM instruction addr 55,598 24.02% 24.02% 99.94% 0.06% 0.00% 0x4000000000003f71 main+0x1ac1 55,461 23.96% 47.99% 0.00% 100.00% 0.00% 0x200000000032ba41 0x200000000032ba41 54,671 23.62% 71.61% 99.00% 0.09% 0.91% 0x4000000000003f60 main+0x1ab0 54,427 23.52% 95.13% 0.00% 0.00% 100.00% 0x20000000002d7021 0x20000000002d7021 1,327 0.57% 95.70% 74.91% 23.66% 1.43% 0x2000000000141061 0x2000000000141061 1,283 0.55% 96.26% 75.45% 21.75% 2.81% 0x2000000000141081 0x2000000000141081 808 0.35% 96.61% 93.56% 6.19% 0.25% 0x4000000000002470 cycles_compare 744 0.32% 96.93% 34.81% 0.00% 65.19% 0x200000000032ba80 0x200000000032ba80 710 0.31% 97.23% 0.00% 89.01% 10.99% 0x4000000000002471 cycles_compare+0x1 545 0.24% 97.47% 4.04% 10.09% 85.87% 0x4000000000003f10 main+0x1a60 522 0.23% 97.70% 100.00% 0.00% 0.00% 0x200000000032da70 0x200000000032da70 507 0.22% 97.92% 99.80% 0.20% 0.00% 0x40000000000048f0 main+0x2440 500 0.22% 98.13% 0.00% 0.00% 100.00% 0x4000000000003fd1 main+0x1b21 490 0.21% 98.34% 98.57% 1.43% 0.00% 0x200000000032d4c0 0x200000000032d4c0 485 0.21% 98.55% 0.00% 0.00% 100.00% 0x200000000032bac1 0x200000000032bac1 452 0.20% 98.75% 99.78% 0.22% 0.00% 0x200000000032bdb0 0x200000000032bdb0 448 0.19% 98.94% 99.78% 0.22% 0.00% 0x200000000032dd20 0x200000000032dd20 431 0.19% 99.13% 0.00% 100.00% 0.00% 0x200000000032b5d1 0x200000000032b5d1 418 0.18% 99.31% 77.51% 22.49% 0.00% 0x200000000032b8d0 0x200000000032b8d0 334 0.14% 99.45% 99.10% 0.90% 0.00% 0x4000000000004931 main+0x2481 258 0.11% 99.56% 98.45% 1.55% 0.00% 0x20000000001410d1 0x20000000001410d1 220 0.10% 99.66% 98.18% 1.82% 0.00% 0x20000000001410b0 0x20000000001410b0 91 0.04% 99.70% 42.86% 19.78% 37.36% 0x40000000000040c0 main+0x1c10 83 0.04% 99.73% 100.00% 0.00% 0.00% 0x200000000032bcd1 0x200000000032bcd1 74 0.03% 99.77% 1.35% 98.65% 0.00% 0x4000000000003f90 main+0x1ae0 55 0.02% 99.79% 100.00% 0.00% 0.00% 0x2000000000141100 0x2000000000141100 44 0.02% 99.81% 20.45% 79.55% 0.00% 0x20000000000ae0b0 0x20000000000ae0b0 37 0.02% 99.82% 24.32% 70.27% 5.41% 0x20000000000ae0e0 0x20000000000ae0e0 25 0.01% 99.84% 4.00% 0.00% 96.00% 0x40000000000040b0 main+0x1c00 21 0.01% 99.84% 0.00% 100.00% 0.00% 0x200000000032b9d1 0x200000000032b9d1 21 0.01% 99.85% 14.29% 85.71% 0.00% 0x4000000000004950 main+0x24a0 20 0.01% 99.86% 100.00% 0.00% 0.00% 0x200000000032d571 0x200000000032d571 15 0.01% 99.87% 33.33% 53.33% 13.33% 0x20000000000231d0 0x20000000000231d0 14 0.01% 99.88% 100.00% 0.00% 0.00% 0x200000000032fb40 0x200000000032fb40 11 0.00% 99.88% 0.00% 100.00% 0.00% 0x2000000000141111 0x2000000000141111 10 0.00% 99.88% 70.00% 20.00% 10.00% 0x2000000000011020 0x2000000000011020 8 0.00% 99.89% 75.00% 25.00% 0.00% 0x2000000000010f90 0x2000000000010f90 8 0.00% 99.89% 62.50% 37.50% 0.00% 0x200000000032d500 0x200000000032d500 6 0.00% 99.89% 66.67% 16.67% 16.67% 0x2000000000011060 0x2000000000011060 5 0.00% 99.90% 0.00% 100.00% 0.00% 0x2000000000021570 0x2000000000021570 5 0.00% 99.90% 20.00% 60.00% 20.00% 0x2000000000014b21 0x2000000000014b21 5 0.00% 99.90% 100.00% 0.00% 0.00% 0x2000000000021880 0x2000000000021880 5 0.00% 99.90% 20.00% 80.00% 0.00% 0x2000000000010fd1 0x2000000000010fd1 5 0.00% 99.90% 100.00% 0.00% 0.00% 0x2000000000010ad0 0x2000000000010ad0 5 0.00% 99.91% 100.00% 0.00% 0.00% 0x200000000032b8f1 0x200000000032b8f1 4 0.00% 99.91% 0.00% 25.00% 75.00% 0x2000000000018c20 0x2000000000018c20 4 0.00% 99.91% 100.00% 0.00% 0.00% 0x2000000000021520 0x2000000000021520 4 0.00% 99.91% 25.00% 75.00% 0.00% 0x2000000000010a81 0x2000000000010a81 4 0.00% 99.91% 0.00% 100.00% 0.00% 0x4000000000004941 main+0x2491 3 0.00% 99.91% 66.67% 33.33% 0.00% 0x2000000000014b11 0x2000000000014b11 3 0.00% 99.92% 33.33% 33.33% 33.33% 0x2000000000017de0 0x2000000000017de0 3 0.00% 99.92% 66.67% 33.33% 0.00% 0x2000000000018421 0x2000000000018421 3 0.00% 99.92% 66.67% 33.33% 0.00% 0x200000000012f791 0x200000000012f791 3 0.00% 99.92% 0.00% 33.33% 66.67% 0x200000000012f850 0x200000000012f850 3 0.00% 99.92% 0.00% 100.00% 0.00% 0x2000000000010a41 0x2000000000010a41 3 0.00% 99.92% 66.67% 33.33% 0.00% 0x20000000000110c0 0x20000000000110c0 3 0.00% 99.92% 0.00% 0.00% 100.00% 0x20000000001dfa00 0x20000000001dfa00 3 0.00% 99.93% 100.00% 0.00% 0.00% 0x200000000032d740 0x200000000032d740 3 0.00% 99.93% 0.00% 100.00% 0.00% 0x200000000032fad0 0x200000000032fad0 2 0.00% 99.93% 0.00% 0.00% 100.00% 0x20000000000231d1 0x20000000000231d1 2 0.00% 99.93% 50.00% 50.00% 0.00% 0x2000000000010770 0x2000000000010770 2 0.00% 99.93% 0.00% 100.00% 0.00% 0x2000000000015020 0x2000000000015020 2 0.00% 99.93% 50.00% 50.00% 0.00% 0x2000000000014610 0x2000000000014610 2 0.00% 99.93% 0.00% 100.00% 0.00% 0x2000000000021850 0x2000000000021850 2 0.00% 99.93% 100.00% 0.00% 0.00% 0x2000000000014a20 0x2000000000014a20 2 0.00% 99.93% 100.00% 0.00% 0.00% 0x2000000000011090 0x2000000000011090 2 0.00% 99.93% 100.00% 0.00% 0.00% 0x2000000000014ba0 0x2000000000014ba0 2 0.00% 99.93% 0.00% 50.00% 50.00% 0x2000000000010f10 0x2000000000010f10 2 0.00% 99.94% 50.00% 50.00% 0.00% 0x2000000000018451 0x2000000000018451 2 0.00% 99.94% 0.00% 0.00% 100.00% 0x2000000000010f20 0x2000000000010f20 2 0.00% 99.94% 0.00% 100.00% 0.00% 0x20000000000126d0 0x20000000000126d0 2 0.00% 99.94% 100.00% 0.00% 0.00% 0x200000000012f781 0x200000000012f781 2 0.00% 99.94% 0.00% 100.00% 0.00% 0x20000000001906f1 0x20000000001906f1 2 0.00% 99.94% 100.00% 0.00% 0.00% 0x20000000000109e0 0x20000000000109e0 2 0.00% 99.94% 0.00% 0.00% 100.00% 0x20000000001deba0 0x20000000001deba0 2 0.00% 99.94% 50.00% 0.00% 50.00% 0x20000000002a4481 0x20000000002a4481 2 0.00% 99.94% 0.00% 100.00% 0.00% 0x2000000000190780 0x2000000000190780 2 0.00% 99.94% 0.00% 50.00% 50.00% 0x2000000000017d41 0x2000000000017d41 2 0.00% 99.94% 0.00% 100.00% 0.00% 0x20000000002b1ff1 0x20000000002b1ff1 2 0.00% 99.94% 100.00% 0.00% 0.00% 0x200000000013c7d1 0x200000000013c7d1 2 0.00% 99.95% 50.00% 50.00% 0.00% 0x200000000032d640 0x200000000032d640 2 0.00% 99.95% 0.00% 100.00% 0.00% 0x200000000032bc61 0x200000000032bc61 2 0.00% 99.95% 50.00% 50.00% 0.00% 0x200000000032a030 0x200000000032a030 2 0.00% 99.95% 100.00% 0.00% 0.00% 0x200000000032a450 0x200000000032a450 2 0.00% 99.95% 100.00% 0.00% 0.00% 0x200000000032fb70 0x200000000032fb70 2 0.00% 99.95% 100.00% 0.00% 0.00% 0x200000000032fac1 0x200000000032fac1 2 0.00% 99.95% 50.00% 50.00% 0.00% 0x2000000000083b30 0x2000000000083b30 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x2000000000005330 0x2000000000005330 1 0.00% 99.95% 0.00% 100.00% 0.00% 0x200000000000c3d0 0x200000000000c3d0 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x200000000000c530 0x200000000000c530 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x20000000000165b0 0x20000000000165b0 1 0.00% 99.95% 0.00% 100.00% 0.00% 0x20000000000246d1 0x20000000000246d1 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x20000000000059f0 0x20000000000059f0 1 0.00% 99.95% 0.00% 100.00% 0.00% 0x20000000000062e0 0x20000000000062e0 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x20000000000216f0 0x20000000000216f0 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x2000000000010e30 0x2000000000010e30 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000014d81 0x2000000000014d81 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000021800 0x2000000000021800 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000006400 0x2000000000006400 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x2000000000011f40 0x2000000000011f40 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000014d71 0x2000000000014d71 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x2000000000018441 0x2000000000018441 1 0.00% 99.96% 0.00% 0.00% 100.00% 0x2000000000019070 0x2000000000019070 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x20000000001356e1 0x20000000001356e1 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x20000000000107c1 0x20000000000107c1 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x2000000000010a10 0x2000000000010a10 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000010e00 0x2000000000010e00 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000010a11 0x2000000000010a11 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000018ce1 0x2000000000018ce1 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000010f21 0x2000000000010f21 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000018c30 0x2000000000018c30 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x200000000000e6b0 0x200000000000e6b0 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x20000000000171d1 0x20000000000171d1 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x2000000000010f50 0x2000000000010f50 1 0.00% 99.96% 0.00% 0.00% 100.00% 0x2000000000044741 0x2000000000044741 1 0.00% 99.96% 0.00% 0.00% 100.00% 0x20000000002d83a1 0x20000000002d83a1 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x20000000002a4d10 0x20000000002a4d10 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x20000000000841e1 0x20000000000841e1 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x20000000002a4e00 0x20000000002a4e00 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000017e81 0x2000000000017e81 1 0.00% 99.97% 0.00% 0.00% 100.00% 0x20000000002a3f81 0x20000000002a3f81 1 0.00% 99.97% 0.00% 0.00% 100.00% 0x2000000000017d20 0x2000000000017d20 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000000109f0 0x20000000000109f0 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000002a6ca0 0x20000000002a6ca0 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x2000000000017f70 0x2000000000017f70 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a7b11 0x20000000002a7b11 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x2000000000017d21 0x2000000000017d21 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002acb90 0x20000000002acb90 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a6770 0x20000000002a6770 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x2000000000132290 0x2000000000132290 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000002a4cd0 0x20000000002a4cd0 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000002a43a1 0x20000000002a43a1 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000000109e1 0x20000000000109e1 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a4321 0x20000000002a4321 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x2000000000084190 0x2000000000084190 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x2000000000011b41 0x2000000000011b41 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x200000000010e4d1 0x200000000010e4d1 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a4461 0x20000000002a4461 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x200000000013a1f0 0x200000000013a1f0 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a4201 0x20000000002a4201 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002b27a0 0x20000000002b27a0 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000002a3da1 0x20000000002a3da1 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a4301 0x20000000002a4301 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x2000000000132860 0x2000000000132860 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4021 0x20000000002a4021 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002b1ff0 0x20000000002b1ff0 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4741 0x20000000002a4741 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a3f01 0x20000000002a3f01 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x200000000013cb70 0x200000000013cb70 1 0.00% 99.98% 0.00% 0.00% 100.00% 0x20000000001906b0 0x20000000001906b0 1 0.00% 99.98% 0.00% 0.00% 100.00% 0x200000000012f7b1 0x200000000012f7b1 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x20000000001321c1 0x20000000001321c1 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x20000000001314a0 0x20000000001314a0 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4210 0x20000000002a4210 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x2000000000132240 0x2000000000132240 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4430 0x20000000002a4430 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x200000000013ce11 0x200000000013ce11 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000001308d1 0x20000000001308d1 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000001d0ea1 0x20000000001d0ea1 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002b1b70 0x20000000002b1b70 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x2000000000131a20 0x2000000000131a20 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x20000000001318b0 0x20000000001318b0 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002b1e00 0x20000000002b1e00 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4810 0x20000000002a4810 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x2000000000017e60 0x2000000000017e60 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x4000000000002600 main+0x150 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x20000000001317b1 0x20000000001317b1 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x2000000000010df0 0x2000000000010df0 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x2000000000045c80 0x2000000000045c80 1 0.00% 99.99% 0.00% 0.00% 100.00% 0x200000000012f701 0x200000000012f701 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x4000000000001290 _init+0x50 1 0.00% 99.99% 0.00% 0.00% 100.00% 0x20000000000e0560 0x20000000000e0560 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x2000000000017e20 0x2000000000017e20 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x4000000000003650 main+0x11a0 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x20000000000e51f1 0x20000000000e51f1 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x2000000000017d00 0x2000000000017d00 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x20000000000b3660 0x20000000000b3660 1 0.00% 99.99% 0.00% 0.00% 100.00% 0x20000000000424e1 0x20000000000424e1 1 0.00% 99.99% 0.00% 0.00% 100.00% 0x200000000012d051 0x200000000012d051 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032ba40 0x200000000032ba40 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032be50 0x200000000032be50 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x200000000032d8b0 0x200000000032d8b0 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x200000000032a201 0x200000000032a201 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x200000000032be10 0x200000000032be10 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x4000000000003f81 main+0x1ad1 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032ba50 0x200000000032ba50 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032bc01 0x200000000032bc01 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x4000000000003f91 main+0x1ae1 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032bc60 0x200000000032bc60 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000032b570 0x200000000032b570 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000032a441 0x200000000032a441 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000032a210 0x200000000032a210 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000013f311 0x200000000013f311 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000012faf0 0x200000000012faf0 1 0.00% 100.00% 0.00% 100.00% 0.00% 0x200000000012c181 0x200000000012c181 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x20000000000ae341 0x20000000000ae341 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x20000000000ae340 0x20000000000ae340 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x20000000000db490 0x20000000000db490 1 0.00% 100.00% 0.00% 100.00% 0.00% 0x20000000000ef5d0 0x20000000000ef5d0 1 0.00% 100.00% 0.00% 0.00% 100.00% 0x2000000000019620 0x2000000000019620 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x20000000000198d1 0x20000000000198d1 # level 0 : counts=116599 avg_cycles=0.0ms 50.38% # level 1 : counts=57736 avg_cycles=0.0ms 24.95% # level 2 : counts=57093 avg_cycles=0.0ms 24.67% approx cost: 0.0s From tduffy at sun.com Thu Jun 2 10:20:16 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 02 Jun 2005 10:20:16 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1117576362.1373.16.camel@duffman> Message-ID: <1117732816.15084.16.camel@duffman> On Thu, 2005-06-02 at 10:17 -0400, James Lentini wrote: > > On Tue, 31 May 2005, Tom Duffy wrote: > > > On Tue, 2005-05-31 at 14:17 -0400, James Lentini wrote: > >> Here's the specification's exact description: > >> > >> timeout: Duration of time, in microseconds, that a consumer waits for > >> connection establishment. The value of DAT_TIMEOUT_INFINITE > >> represents no timeout, indefinite wait. Values must be > >> positive. > > > > Let me make sure I got this right: timeout is in µs (10^-6 seconds), not > > ms (10^-3 seconds). If so, I am off by 3 orders of magnitude in my > > calculation. Right? > > Correct, the value was intended to be in microseconds. OK, so this should be the conversion function: /* * approximately transforms microseconds to 4.096us*2^x * 63(+8) is max return */ static inline u8 dapl_convert_us_to_kookyib(unsigned long us) { unsigned long ms = us/1000UL, converged = 2; u8 i; if (2 > ms) return 8; for (i = 1; i < 63; i++) { if (converged >= ms) break; converged = 2*converged; } return i+8; } -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mshefty at ichips.intel.com Thu Jun 2 10:24:06 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 02 Jun 2005 10:24:06 -0700 Subject: [openib-general] separate CQs for QP 0/1? Message-ID: <429F40B6.8020602@ichips.intel.com> I haven't been able to determine the cause of the MAD crashes, so I was going to separate the CQs for QP 0 and 1 to give me more context/state information when a completion occurs. Is this a change that we want permanently? If so, I will cleanup the change for submission, otherwise, I'll just do it quickly to help identify the cause of the issue. - Sean From tduffy at sun.com Thu Jun 2 10:31:35 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 02 Jun 2005 10:31:35 -0700 Subject: [openib-general] Re: opensm: new segv on shutdown In-Reply-To: <1117672772.4476.46.camel@hal.voltaire.com> References: <1117659061.25491.12.camel@duffman> <1117672772.4476.46.camel@hal.voltaire.com> Message-ID: <1117733495.17989.6.camel@duffman> On Wed, 2005-06-01 at 20:45 -0400, Hal Rosenstock wrote: > On Wed, 2005-06-01 at 16:51, Tom Duffy wrote: > > I am putting together a network with a dumb IB switch, a couple of Linux > > OpenIB boxes, a Solaris 10 box, a Solaris Nevada box, etc. I fired up > > opensm on one of the Linux nodes, tried to plumb Solaris, no luck. I > > then hit control-c on opensm and it crashed. Here is the messages and > > then crash. > > Anything from the Solaris side on what it doesn't like about the OpenIB > RMPP ? I am not seeing any errors coming from Solaris, I will have to enable debug and try again. Clearly OpenSM is able to find the Solaris nodes (there are 3 solaris, 2 linux): [root at flopteron2 ~]# ibhosts Hca : 0x0002c90109766e40 ports 2 "MT23108 InfiniHost Mellanox Technologies" Hca : 0x0002c90109765630 ports 2 "MT23108 InfiniHost Mellanox Technologies" Hca : 0x0002c901097624c0 ports 2 "MT23108 InfiniHost Mellanox Technologies" Hca : 0x0002c90109765710 ports 2 "MT23108 InfiniHost Mellanox Technologies" Hca : 0x0002c9010a99e030 ports 2 "MT25208 InfiniHostEx Mellanox Technologies" [root at flopteron2 ~]# ibnetdiscover # # Topology file: generated on Thu Jun 2 10:20:58 2005 # switchguids=0x617000000000d Switch 8 "S-000617000000000d" # Agilent and RedSwitch High Performance 8 Port 4x IBA Switch port 0 lid 2 [8] "H-0002c90109766e40"[2] [6] "H-0002c90109766e40"[1] [5] "H-0002c90109765630"[1] [4] "H-0002c90109765710"[1] [3] "H-0002c901097624c0"[1] [2] "H-00109765710"[2] [7] "H-0002c9010a99e030"[1] hcaguids=0x2c90109766e40 Hca 2 "H-0002c90109766e40" # MT23108 InfiniHost Mellanox Technologies [2] "S-000617000000000d"[8] # lid 0 lmc 0 [1] "S-000617000000000d"[6] # lid 0 lmc 0 hcaguids=0x2c90109765630 Hca 2 "H-0002c90109765630" # MT23108 InfiniHost Mellanox Technologies [1] "S-000617000000000d"[5] # lid 4 lmc 0 hcaguids=0x2c901097624c0 Hca 2 "H-0002c901097624c0" # MT23108 InfiniHost Mellanox Technologies [1] "S-000617000000000d"[3] # lid 5 lmc 0 hcaguids=0x2c90109765710 Hca 2 "H-0002c90109765710" # MT23108 InfiniHost Mellanox Technologies [1] "S-000617000000000d"[4] # lid 18 lmc 0 [2] "S-000617000000000d"[2] # lid 3 lmc 0 hcaguids=0x2c9010a99e030 Hca 2 "H-0002c9010a99e030" # MT25208 InfiniHostEx Mellanox Technologies [1] "S-000617000000000d"[7] # lid 1 lmc 0 --- When I pop open ibsmgui (the graphical IB browser on Solaris), I get an error: sa_access_retrieve failed sa_access_retrieve failed: -18 from the application. This is the relevant opensm log generated by this query: Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 13802 QP1 MADs received. Jun 02 10:22:57 [44808960] -> SA MAD dump: base_ver................0x1 mgmt_class..............0x3 class_ver...............0x2 method..................0x12 (SubnAdmGetTable) status..................0x0 resv....................0x0 trans_id................0x976563100000010 attr_id.................0x31 (ServiceRecord) resv1...................0x0 attr_mod................0xFFFFFFFF rmpp_version............0x0 rmpp_type...............0x0 rmpp_flags..............0x0 rmpp_status.............0x0 seg_num.................0x0 payload_len/new_win.....0x0 sm_key..................0x0000000000000000 attr_offset.............0x0 resv2...................0x0 comp_mask...............0x0000000000000000 Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_process: [ Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_SERVICE_RECORD. Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_process: ] -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Thu Jun 2 10:37:49 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 02 Jun 2005 10:37:49 -0700 Subject: [openib-general] separate CQs for QP 0/1? In-Reply-To: <429F40B6.8020602@ichips.intel.com> (Sean Hefty's message of "Thu, 02 Jun 2005 10:24:06 -0700") References: <429F40B6.8020602@ichips.intel.com> Message-ID: <52hdgg1wya.fsf@topspin.com> Sean> I haven't been able to determine the cause of the MAD Sean> crashes, so I was going to separate the CQs for QP 0 and 1 Sean> to give me more context/state information when a completion Sean> occurs. Sean> Is this a change that we want permanently? If so, I will Sean> cleanup the change for submission, otherwise, I'll just do Sean> it quickly to help identify the cause of the issue. I don't see a reason to care either way, so as far as I see, we may as well stick with what we've got. On the other hand, if splitting the CQs is an improvement then we should do it. - R. From mshefty at ichips.intel.com Thu Jun 2 10:42:54 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 02 Jun 2005 10:42:54 -0700 Subject: [openib-general] separate CQs for QP 0/1? In-Reply-To: <52hdgg1wya.fsf@topspin.com> References: <429F40B6.8020602@ichips.intel.com> <52hdgg1wya.fsf@topspin.com> Message-ID: <429F451E.6040209@ichips.intel.com> Roland Dreier wrote: > Sean> I haven't been able to determine the cause of the MAD > Sean> crashes, so I was going to separate the CQs for QP 0 and 1 > Sean> to give me more context/state information when a completion > Sean> occurs. > > Sean> Is this a change that we want permanently? If so, I will > Sean> cleanup the change for submission, otherwise, I'll just do > Sean> it quickly to help identify the cause of the issue. > > I don't see a reason to care either way, so as far as I see, we may as > well stick with what we've got. On the other hand, if splitting the > CQs is an improvement then we should do it. Separately would likely increase the number of interrupts seen by the system, but allows separate completion processing of SMI and GSI traffic... I don't see a strong reason for the change, and am only making the change to give myself more context when a completion event occurs at this point. (So I'll know exactly which queue I should be accessing.) - Sean From roland at topspin.com Thu Jun 2 10:59:13 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 02 Jun 2005 10:59:13 -0700 Subject: [openib-general] Re: [PATCH] mthca: fix registration for giant MRs In-Reply-To: <20050602144305.GM23899@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 2 Jun 2005 17:43:05 +0300") References: <52r7fl6bxa.fsf@topspin.com> <20050602144305.GM23899@mellanox.co.il> Message-ID: <52psv4zlla.fsf@topspin.com> OK, I respun the patch. I didn't change the calculation of mtt->order to use fls, because the formula fls(max(size, MTHCA_MTT_SEG_SIZE / 8) - 1) should really be fls(max(size, MTHCA_MTT_SEG_SIZE / 8)) - fls(MTHCA_MTT_SEG_SIZE / 8) or something like that, and I ended up confusing myself and so I decided to stick with a loop that I know is correct. Anyway, here's the new patch. --- mthca/mthca_dev.h (revision 2470) +++ mthca/mthca_dev.h (working copy) @@ -387,6 +387,12 @@ void mthca_uar_free(struct mthca_dev *de int mthca_pd_alloc(struct mthca_dev *dev, int privileged, struct mthca_pd *pd); void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd); +struct mthca_mtt *mthca_alloc_mtt(struct mthca_dev *dev, int size); +void mthca_free_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt); +int mthca_write_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt, + int start_index, u64 *buffer_list, int list_len); +int mthca_mr_alloc(struct mthca_dev *dev, u32 pd, int buffer_size_shift, + u64 iova, u64 total_size, u32 access, struct mthca_mr *mr); int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, u32 access, struct mthca_mr *mr); int mthca_mr_alloc_phys(struct mthca_dev *dev, u32 pd, --- mthca/mthca_provider.c (revision 2490) +++ mthca/mthca_provider.c (working copy) @@ -787,13 +787,13 @@ static struct ib_mr *mthca_reg_phys_mr(s static struct ib_mr *mthca_reg_user_mr(struct ib_pd *pd, struct ib_umem *region, int acc, const void __user *udata, int udatalen) { + struct mthca_dev *dev = to_mdev(pd->device); struct ib_umem_chunk *chunk; - int npages = 0; - u64 *page_list; struct mthca_mr *mr; - int shift; + u64 *pages; + int shift, n, len; int i, j, k; - int err; + int err = 0; shift = ffs(region->page_size) - 1; @@ -801,37 +801,68 @@ static struct ib_mr *mthca_reg_user_mr(s if (!mr) return ERR_PTR(-ENOMEM); + n = 0; list_for_each_entry(chunk, ®ion->chunk_list, list) - npages += chunk->nents; + n += chunk->nents; - page_list = kmalloc(npages * sizeof *page_list, GFP_KERNEL); - if (!page_list) { - kfree(mr); - return ERR_PTR(-ENOMEM); + mr->mtt = mthca_alloc_mtt(dev, n); + if (IS_ERR(mr->mtt)) { + err = PTR_ERR(mr->mtt); + goto err; + } + + pages = (u64 *) __get_free_page(GFP_KERNEL); + if (!pages) { + err = -ENOMEM; + goto err_mtt; } - i = 0; + i = n = 0; list_for_each_entry(chunk, ®ion->chunk_list, list) - for (j = 0; j < chunk->nmap; ++j) - for (k = 0; k < sg_dma_len(&chunk->page_list[j]) >> shift; ++k) - page_list[i++] = sg_dma_address(&chunk->page_list[j]) + + for (j = 0; j < chunk->nmap; ++j) { + len = sg_dma_len(&chunk->page_list[j]) >> shift; + for (k = 0; k < len; ++k) { + pages[i++] = sg_dma_address(&chunk->page_list[j]) + region->page_size * k; + /* + * Be friendly to WRITE_MTT command + * and leave two empty slots for the + * index and reserved fields of the + * mailbox. + */ + if (i == PAGE_SIZE / sizeof (u64) - 2) { + err = mthca_write_mtt(dev, mr->mtt, + n, pages, i); + if (err) + goto mtt_done; + n += i; + i = 0; + } + } + } - err = mthca_mr_alloc_phys(to_mdev(pd->device), - to_mpd(pd)->pd_num, - page_list, shift, npages, - region->virt_base, region->length, - convert_access(acc), mr); + if (i) + err = mthca_write_mtt(dev, mr->mtt, n, pages, i); +mtt_done: + free_page((unsigned long) pages); + if (err) + goto err_mtt; - if (err) { - kfree(page_list); - kfree(mr); - return ERR_PTR(err); - } + err = mthca_mr_alloc(dev, to_mpd(pd)->pd_num, shift, region->virt_base, + region->length, convert_access(acc), mr); + + if (err) + goto err_mtt; - kfree(page_list); return &mr->ibmr; + +err_mtt: + mthca_free_mtt(dev, mr->mtt); + +err: + kfree(mr); + return ERR_PTR(err); } static int mthca_dereg_mr(struct ib_mr *mr) --- mthca/mthca_provider.h (revision 2470) +++ mthca/mthca_provider.h (working copy) @@ -62,18 +62,18 @@ struct mthca_ucontext { struct mthca_user_db_table *db_tab; }; +struct mthca_mtt; + struct mthca_mr { - struct ib_mr ibmr; - int order; - u32 first_seg; + struct ib_mr ibmr; + struct mthca_mtt *mtt; }; struct mthca_fmr { - struct ib_fmr ibmr; + struct ib_fmr ibmr; struct ib_fmr_attr attr; - int order; - u32 first_seg; - int maps; + struct mthca_mtt *mtt; + int maps; union { struct { struct mthca_mpt_entry __iomem *mpt; --- mthca/mthca_mr.c (revision 2468) +++ mthca/mthca_mr.c (working copy) @@ -40,6 +40,12 @@ #include "mthca_cmd.h" #include "mthca_memfree.h" +struct mthca_mtt { + struct mthca_buddy *buddy; + int order; + u32 first_seg; +}; + /* * Must be packed because mtt_seg is 64 bits but only aligned to 32 bits. */ @@ -173,8 +179,8 @@ static void __devexit mthca_buddy_cleanu kfree(buddy->bits); } -static u32 mthca_alloc_mtt(struct mthca_dev *dev, int order, - struct mthca_buddy *buddy) +static u32 mthca_alloc_mtt_range(struct mthca_dev *dev, int order, + struct mthca_buddy *buddy) { u32 seg = mthca_buddy_alloc(buddy, order); @@ -191,14 +197,101 @@ static u32 mthca_alloc_mtt(struct mthca_ return seg; } -static void mthca_free_mtt(struct mthca_dev *dev, u32 seg, int order, - struct mthca_buddy* buddy) +static struct mthca_mtt *__mthca_alloc_mtt(struct mthca_dev *dev, int size, + struct mthca_buddy *buddy) { - mthca_buddy_free(buddy, seg, order); + struct mthca_mtt *mtt; + int i; + + if (size <= 0) + return ERR_PTR(-EINVAL); + + mtt = kmalloc(sizeof *mtt, GFP_KERNEL); + if (!mtt) + return ERR_PTR(-ENOMEM); + + mtt->buddy = buddy; + mtt->order = 0; + for (i = MTHCA_MTT_SEG_SIZE / 8; i < size; i <<= 1) + ++mtt->order; + + mtt->first_seg = mthca_alloc_mtt_range(dev, mtt->order, buddy); + if (mtt->first_seg == -1) { + kfree(mtt); + return ERR_PTR(-ENOMEM); + } + + return mtt; +} + +struct mthca_mtt *mthca_alloc_mtt(struct mthca_dev *dev, int size) +{ + return __mthca_alloc_mtt(dev, size, &dev->mr_table.mtt_buddy); +} + +void mthca_free_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt) +{ + if (!mtt) + return; + + mthca_buddy_free(mtt->buddy, mtt->first_seg, mtt->order); if (mthca_is_memfree(dev)) - mthca_table_put_range(dev, dev->mr_table.mtt_table, seg, - seg + (1 << order) - 1); + mthca_table_put_range(dev, dev->mr_table.mtt_table, + mtt->first_seg, + mtt->first_seg + (1 << mtt->order) - 1); + + kfree(mtt); +} + +int mthca_write_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt, + int start_index, u64 *buffer_list, int list_len) +{ + u64 *mtt_entry; + int err = 0; + u8 status; + int i; + + mtt_entry = (u64 *) __get_free_page(GFP_KERNEL); + if (!mtt_entry) + return -ENOMEM; + + while (list_len > 0) { + mtt_entry[0] = cpu_to_be64(dev->mr_table.mtt_base + + mtt->first_seg * MTHCA_MTT_SEG_SIZE + + start_index * 8); + mtt_entry[1] = 0; + for (i = 0; i < list_len && i < PAGE_SIZE / 8 - 2; ++i) + mtt_entry[i + 2] = cpu_to_be64(buffer_list[i] | + MTHCA_MTT_FLAG_PRESENT); + + /* + * If we have an odd number of entries to write, add + * one more dummy entry for firmware efficiency. + */ + if (i & 1) + mtt_entry[i + 2] = 0; + + err = mthca_WRITE_MTT(dev, mtt_entry, (i + 1) & ~1, &status); + if (err) { + mthca_warn(dev, "WRITE_MTT failed (%d)\n", err); + goto out; + } + if (status) { + mthca_warn(dev, "WRITE_MTT returned status 0x%02x\n", + status); + err = -EINVAL; + goto out; + } + + list_len -= i; + start_index += i; + buffer_list += i; + } + +out: + free_page((unsigned long) mtt_entry); + return err; } static inline u32 tavor_hw_index_to_key(u32 ind) @@ -237,18 +330,20 @@ static inline u32 key_to_hw_index(struct return tavor_key_to_hw_index(key); } -int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, - u32 access, struct mthca_mr *mr) +int mthca_mr_alloc(struct mthca_dev *dev, u32 pd, int buffer_size_shift, + u64 iova, u64 total_size, u32 access, struct mthca_mr *mr) { void *mailbox = NULL; struct mthca_mpt_entry *mpt_entry; u32 key; + int i; int err; u8 status; might_sleep(); - mr->order = -1; + WARN_ON(buffer_size_shift >= 32); + key = mthca_alloc(&dev->mr_table.mpt_alloc); if (key == -1) return -ENOMEM; @@ -270,18 +365,36 @@ int mthca_mr_alloc_notrans(struct mthca_ mpt_entry->flags = cpu_to_be32(MTHCA_MPT_FLAG_SW_OWNS | MTHCA_MPT_FLAG_MIO | - MTHCA_MPT_FLAG_PHYSICAL | MTHCA_MPT_FLAG_REGION | access); - mpt_entry->page_size = 0; + if (!mr->mtt) + mpt_entry->flags |= cpu_to_be32(MTHCA_MPT_FLAG_PHYSICAL); + + mpt_entry->page_size = cpu_to_be32(buffer_size_shift - 12); mpt_entry->key = cpu_to_be32(key); mpt_entry->pd = cpu_to_be32(pd); - mpt_entry->start = 0; - mpt_entry->length = ~0ULL; + mpt_entry->start = cpu_to_be64(iova); + mpt_entry->length = cpu_to_be64(total_size); memset(&mpt_entry->lkey, 0, sizeof *mpt_entry - offsetof(struct mthca_mpt_entry, lkey)); + if (mr->mtt) + mpt_entry->mtt_seg = + cpu_to_be64(dev->mr_table.mtt_base + + mr->mtt->first_seg * MTHCA_MTT_SEG_SIZE); + + if (0) { + mthca_dbg(dev, "Dumping MPT entry %08x:\n", mr->ibmr.lkey); + for (i = 0; i < sizeof (struct mthca_mpt_entry) / 4; ++i) { + if (i % 4 == 0) + printk("[%02x] ", i * 4); + printk(" %08x", be32_to_cpu(((u32 *) mpt_entry)[i])); + if ((i + 1) % 4 == 0) + printk("\n"); + } + } + err = mthca_SW2HW_MPT(dev, mpt_entry, key & (dev->limits.num_mpts - 1), &status); @@ -308,150 +421,41 @@ err_out_mpt_free: return err; } +int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, + u32 access, struct mthca_mr *mr) +{ + mr->mtt = NULL; + return mthca_mr_alloc(dev, pd, 12, 0, ~0ULL, access, mr); +} + int mthca_mr_alloc_phys(struct mthca_dev *dev, u32 pd, u64 *buffer_list, int buffer_size_shift, int list_len, u64 iova, u64 total_size, u32 access, struct mthca_mr *mr) { - void *mailbox; - u64 *mtt_entry; - struct mthca_mpt_entry *mpt_entry; - u32 key; - int err = -ENOMEM; - u8 status; - int i; - - might_sleep(); - WARN_ON(buffer_size_shift >= 32); - - key = mthca_alloc(&dev->mr_table.mpt_alloc); - if (key == -1) - return -ENOMEM; - mr->ibmr.rkey = mr->ibmr.lkey = hw_index_to_key(dev, key); - - if (mthca_is_memfree(dev)) { - err = mthca_table_get(dev, dev->mr_table.mpt_table, key); - if (err) - goto err_out_mpt_free; - } - - for (i = MTHCA_MTT_SEG_SIZE / 8, mr->order = 0; - i < list_len; - i <<= 1, ++mr->order) - ; /* nothing */ - - mr->first_seg = mthca_alloc_mtt(dev, mr->order, - &dev->mr_table.mtt_buddy); - if (mr->first_seg == -1) - goto err_out_table; - - /* - * If list_len is odd, we add one more dummy entry for - * firmware efficiency. - */ - mailbox = kmalloc(max(sizeof *mpt_entry, - (size_t) 8 * (list_len + (list_len & 1) + 2)) + - MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) - goto err_out_free_mtt; - - mtt_entry = MAILBOX_ALIGN(mailbox); - - mtt_entry[0] = cpu_to_be64(dev->mr_table.mtt_base + - mr->first_seg * MTHCA_MTT_SEG_SIZE); - mtt_entry[1] = 0; - for (i = 0; i < list_len; ++i) - mtt_entry[i + 2] = cpu_to_be64(buffer_list[i] | - MTHCA_MTT_FLAG_PRESENT); - if (list_len & 1) { - mtt_entry[i + 2] = 0; - ++list_len; - } + int err; - if (0) { - mthca_dbg(dev, "Dumping MPT entry\n"); - for (i = 0; i < list_len + 2; ++i) - printk(KERN_ERR "[%2d] %016llx\n", - i, (unsigned long long) be64_to_cpu(mtt_entry[i])); - } + mr->mtt = mthca_alloc_mtt(dev, list_len); + if (IS_ERR(mr->mtt)) + return PTR_ERR(mr->mtt); - err = mthca_WRITE_MTT(dev, mtt_entry, list_len, &status); + err = mthca_write_mtt(dev, mr->mtt, 0, buffer_list, list_len); if (err) { - mthca_warn(dev, "WRITE_MTT failed (%d)\n", err); - goto err_out_mailbox_free; - } - if (status) { - mthca_warn(dev, "WRITE_MTT returned status 0x%02x\n", - status); - err = -EINVAL; - goto err_out_mailbox_free; - } - - mpt_entry = MAILBOX_ALIGN(mailbox); - - mpt_entry->flags = cpu_to_be32(MTHCA_MPT_FLAG_SW_OWNS | - MTHCA_MPT_FLAG_MIO | - MTHCA_MPT_FLAG_REGION | - access); - - mpt_entry->page_size = cpu_to_be32(buffer_size_shift - 12); - mpt_entry->key = cpu_to_be32(key); - mpt_entry->pd = cpu_to_be32(pd); - mpt_entry->start = cpu_to_be64(iova); - mpt_entry->length = cpu_to_be64(total_size); - memset(&mpt_entry->lkey, 0, - sizeof *mpt_entry - offsetof(struct mthca_mpt_entry, lkey)); - mpt_entry->mtt_seg = cpu_to_be64(dev->mr_table.mtt_base + - mr->first_seg * MTHCA_MTT_SEG_SIZE); - - if (0) { - mthca_dbg(dev, "Dumping MPT entry %08x:\n", mr->ibmr.lkey); - for (i = 0; i < sizeof (struct mthca_mpt_entry) / 4; ++i) { - if (i % 4 == 0) - printk("[%02x] ", i * 4); - printk(" %08x", be32_to_cpu(((u32 *) mpt_entry)[i])); - if ((i + 1) % 4 == 0) - printk("\n"); - } + mthca_free_mtt(dev, mr->mtt); + return err; } - err = mthca_SW2HW_MPT(dev, mpt_entry, - key & (dev->limits.num_mpts - 1), - &status); + err = mthca_mr_alloc(dev, pd, buffer_size_shift, iova, + total_size, access, mr); if (err) - mthca_warn(dev, "SW2HW_MPT failed (%d)\n", err); - else if (status) { - mthca_warn(dev, "SW2HW_MPT returned status 0x%02x\n", - status); - err = -EINVAL; - } - - kfree(mailbox); - return err; - -err_out_mailbox_free: - kfree(mailbox); - -err_out_free_mtt: - mthca_free_mtt(dev, mr->first_seg, mr->order, &dev->mr_table.mtt_buddy); - -err_out_table: - if (mthca_is_memfree(dev)) - mthca_table_put(dev, dev->mr_table.mpt_table, key); + mthca_free_mtt(dev, mr->mtt); -err_out_mpt_free: - mthca_free(&dev->mr_table.mpt_alloc, key); return err; } /* Free mr or fmr */ -static void mthca_free_region(struct mthca_dev *dev, u32 lkey, int order, - u32 first_seg, struct mthca_buddy *buddy) +static void mthca_free_region(struct mthca_dev *dev, u32 lkey) { - if (order >= 0) - mthca_free_mtt(dev, first_seg, order, buddy); - if (mthca_is_memfree(dev)) mthca_table_put(dev, dev->mr_table.mpt_table, arbel_key_to_hw_index(lkey)); @@ -476,8 +480,8 @@ void mthca_free_mr(struct mthca_dev *dev mthca_warn(dev, "HW2SW_MPT returned status 0x%02x\n", status); - mthca_free_region(dev, mr->ibmr.lkey, mr->order, mr->first_seg, - &dev->mr_table.mtt_buddy); + mthca_free_region(dev, mr->ibmr.lkey); + mthca_free_mtt(dev, mr->mtt); } int mthca_fmr_alloc(struct mthca_dev *dev, u32 pd, @@ -522,21 +526,15 @@ int mthca_fmr_alloc(struct mthca_dev *de mr->mem.tavor.mpt = dev->mr_table.tavor_fmr.mpt_base + sizeof *(mr->mem.tavor.mpt) * idx; - for (i = MTHCA_MTT_SEG_SIZE / 8, mr->order = 0; - i < list_len; - i <<= 1, ++mr->order) - ; /* nothing */ - - mr->first_seg = mthca_alloc_mtt(dev, mr->order, - dev->mr_table.fmr_mtt_buddy); - if (mr->first_seg == -1) + mr->mtt = __mthca_alloc_mtt(dev, list_len, dev->mr_table.fmr_mtt_buddy); + if (IS_ERR(mr->mtt)) goto err_out_table; - mtt_seg = mr->first_seg * MTHCA_MTT_SEG_SIZE; + mtt_seg = mr->mtt->first_seg * MTHCA_MTT_SEG_SIZE; if (mthca_is_memfree(dev)) { mr->mem.arbel.mtts = mthca_table_find(dev->mr_table.mtt_table, - mr->first_seg); + mr->mtt->first_seg); BUG_ON(!mr->mem.arbel.mtts); } else mr->mem.tavor.mtts = dev->mr_table.tavor_fmr.mtt_base + mtt_seg; @@ -592,8 +590,7 @@ err_out_mailbox_free: kfree(mailbox); err_out_free_mtt: - mthca_free_mtt(dev, mr->first_seg, mr->order, - dev->mr_table.fmr_mtt_buddy); + mthca_free_mtt(dev, mr->mtt); err_out_table: if (mthca_is_memfree(dev)) @@ -609,8 +606,9 @@ int mthca_free_fmr(struct mthca_dev *dev if (fmr->maps) return -EBUSY; - mthca_free_region(dev, fmr->ibmr.lkey, fmr->order, fmr->first_seg, - dev->mr_table.fmr_mtt_buddy); + mthca_free_region(dev, fmr->ibmr.lkey); + mthca_free_mtt(dev, fmr->mtt); + return 0; } @@ -826,7 +824,8 @@ int __devinit mthca_init_mr_table(struct if (dev->limits.reserved_mtts) { i = fls(dev->limits.reserved_mtts - 1); - if (mthca_alloc_mtt(dev, i, dev->mr_table.fmr_mtt_buddy) == -1) { + if (mthca_alloc_mtt_range(dev, i, + dev->mr_table.fmr_mtt_buddy) == -1) { mthca_warn(dev, "MTT table of order %d is too small.\n", dev->mr_table.fmr_mtt_buddy->max_order); err = -ENOMEM; From jjengla at sandia.gov Thu Jun 2 10:59:58 2005 From: jjengla at sandia.gov (Josh England) Date: Thu, 02 Jun 2005 10:59:58 -0700 Subject: [openib-general] kdaptest wedges server Message-ID: <1117735198.5613.56.camel@localhost> Hi, The kdapltest program wedges consistently wedges a node. I just tried it with Revision 2523, and it runs fine the first couple of times. The third time took a while, and the fourth time killed the server. The server is running: kdapltest -T S -D mthca0a The client runs this several times in succession: kdapltest -T T -D mthca0a -s client SR 256 server SR 256 -JE From halr at voltaire.com Thu Jun 2 11:08:48 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 Jun 2005 14:08:48 -0400 Subject: [openib-general] Re: opensm: new segv on shutdown In-Reply-To: <1117733495.17989.6.camel@duffman> References: <1117659061.25491.12.camel@duffman> <1117672772.4476.46.camel@hal.voltaire.com> <1117733495.17989.6.camel@duffman> Message-ID: <1117734193.4476.12.camel@hal.voltaire.com> On Thu, 2005-06-02 at 13:31, Tom Duffy wrote: > On Wed, 2005-06-01 at 20:45 -0400, Hal Rosenstock wrote: > > On Wed, 2005-06-01 at 16:51, Tom Duffy wrote: > > > I am putting together a network with a dumb IB switch, a couple of Linux > > > OpenIB boxes, a Solaris 10 box, a Solaris Nevada box, etc. I fired up > > > opensm on one of the Linux nodes, tried to plumb Solaris, no luck. I > > > then hit control-c on opensm and it crashed. Here is the messages and > > > then crash. > > > > Anything from the Solaris side on what it doesn't like about the OpenIB > > RMPP ? > > I am not seeing any errors coming from Solaris, I will have to enable > debug and try again. Clearly OpenSM is able to find the Solaris nodes > (there are 3 solaris, 2 linux): > > [root at flopteron2 ~]# ibhosts > Hca : 0x0002c90109766e40 ports 2 "MT23108 InfiniHost Mellanox Technologies" > Hca : 0x0002c90109765630 ports 2 "MT23108 InfiniHost Mellanox Technologies" > Hca : 0x0002c901097624c0 ports 2 "MT23108 InfiniHost Mellanox Technologies" > Hca : 0x0002c90109765710 ports 2 "MT23108 InfiniHost Mellanox Technologies" > Hca : 0x0002c9010a99e030 ports 2 "MT25208 InfiniHostEx Mellanox Technologies" > > [root at flopteron2 ~]# ibnetdiscover > # > # Topology file: generated on Thu Jun 2 10:20:58 2005 > # > switchguids=0x617000000000d > Switch 8 "S-000617000000000d" # Agilent and RedSwitch High Performance 8 Port 4x IBA Switch port 0 lid 2 > [8] "H-0002c90109766e40"[2] [6] "H-0002c90109766e40"[1] > [5] "H-0002c90109765630"[1] > [4] "H-0002c90109765710"[1] > [3] "H-0002c901097624c0"[1] > [2] "H-00109765710"[2] > [7] "H-0002c9010a99e030"[1] > > hcaguids=0x2c90109766e40 > Hca 2 "H-0002c90109766e40" # MT23108 InfiniHost Mellanox Technologies > [2] "S-000617000000000d"[8] # lid 0 lmc 0 > [1] "S-000617000000000d"[6] # lid 0 lmc 0 > > hcaguids=0x2c90109765630 > Hca 2 "H-0002c90109765630" # MT23108 InfiniHost Mellanox Technologies > [1] "S-000617000000000d"[5] # lid 4 lmc 0 > > hcaguids=0x2c901097624c0 > Hca 2 "H-0002c901097624c0" # MT23108 InfiniHost Mellanox Technologies > [1] "S-000617000000000d"[3] # lid 5 lmc 0 > > hcaguids=0x2c90109765710 > Hca 2 "H-0002c90109765710" # MT23108 InfiniHost Mellanox Technologies > [1] "S-000617000000000d"[4] # lid 18 lmc 0 > [2] "S-000617000000000d"[2] # lid 3 lmc 0 > > hcaguids=0x2c9010a99e030 > Hca 2 "H-0002c9010a99e030" # MT25208 InfiniHostEx Mellanox Technologies > [1] "S-000617000000000d"[7] # lid 1 lmc 0 > > --- > > When I pop open ibsmgui (the graphical IB browser on Solaris), I get an > error: > > sa_access_retrieve failed > sa_access_retrieve failed: -18 > > from the application. This is the relevant opensm log generated by this > query: > > Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 13802 QP1 MADs received. > Jun 02 10:22:57 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976563100000010 > attr_id.................0x31 (ServiceRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000000000 > > > Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_SERVICE_RECORD. > Jun 02 10:22:57 [44808960] -> __osm_sa_mad_ctrl_process: ] Something is not working correctly in the OpenIB RMPP space in terms of Solaris RMPP. That is the cause of this (and the root cause of the osm segv reported yesterday). -- Hal From jlentini at netapp.com Thu Jun 2 11:19:21 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 14:19:21 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <1117650224.29566.6.camel@duffman> References: <1117650224.29566.6.camel@duffman> Message-ID: I committed everything in revision 2526, except changing the name of dat_dictionary_key_is_equal to dat_dict_key_is_equal. I assumed that you did this to fit as much of the function declartion on one line as possible. I think I achieved the result you wanted by giving the parameters shorter names. Let me know if there was something else you were after. james On Wed, 1 Jun 2005, Tom Duffy wrote: tduffy> This patch cleans up dat/ a bit, removing unneeded parenthesis, tduffy> gratuitous comments, and a few whitespace cleanups. Please apply. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> api.c | 684 ++++++++++++++++++----------------------------------------- tduffy> core.c | 24 -- tduffy> dictionary.c | 90 +------ tduffy> dr.c | 67 ----- tduffy> dr.h | 6 tduffy> 5 files changed, 242 insertions(+), 629 deletions(-) tduffy> tduffy> Index: linux-kernel-new/dat/dictionary.c tduffy> =================================================================== tduffy> --- linux-kernel-new/dat/dictionary.c (revision 2518) tduffy> +++ linux-kernel-new/dat/dictionary.c (working copy) tduffy> @@ -33,28 +33,11 @@ tduffy> tduffy> #include "dictionary.h" tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Function Declarations * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, tduffy> struct dat_provider_info *new_key); tduffy> tduffy> -static boolean_t tduffy> -dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, tduffy> - const struct dat_provider_info *key_b); tduffy> - tduffy> -/********************************************************************* tduffy> - * * tduffy> - * External Functions * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_destroy tduffy> - ***********************************************************************/ tduffy> +static boolean_t dat_dict_key_is_equal(const struct dat_provider_info *key_a, tduffy> + const struct dat_provider_info *key_b); tduffy> tduffy> void dat_dictionary_destroy(struct dat_dictionary *dictionary) tduffy> { tduffy> @@ -71,10 +54,6 @@ void dat_dictionary_destroy(struct dat_d tduffy> } tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_size tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_dictionary_size(const struct dat_dictionary *dictionary, int *size) tduffy> { tduffy> BUG_ON(NULL == dictionary); tduffy> @@ -85,10 +64,6 @@ u32 dat_dictionary_size(const struct dat tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_entry_create tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry) tduffy> { tduffy> struct dat_dictionary_entry *entry; tduffy> @@ -105,23 +80,15 @@ u32 dat_dictionary_entry_create(struct d tduffy> tduffy> *p_entry = entry; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_entry_destroy tduffy> - ***********************************************************************/ tduffy> - tduffy> void dat_dictionary_entry_destroy(const struct dat_dictionary_entry *entry) tduffy> { tduffy> kfree(entry); tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_insert tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_dictionary_insert(struct dat_dictionary *dictionary, tduffy> const struct dat_provider_info *key, tduffy> struct dat_dictionary_entry *entry, void *data) tduffy> @@ -148,14 +115,10 @@ u32 dat_dictionary_insert(struct dat_dic tduffy> tduffy> dictionary->size++; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_search tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_dictionary_search(struct dat_dictionary *dictionary, tduffy> const struct dat_provider_info *key, void **p_data) tduffy> { tduffy> @@ -168,7 +131,7 @@ u32 dat_dictionary_search(struct dat_dic tduffy> status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); tduffy> tduffy> list_for_each_entry(cur_entry, &dictionary->list, list) { tduffy> - if (TRUE == dat_dictionary_key_is_equal(&cur_entry->key, key)) { tduffy> + if (TRUE == dat_dict_key_is_equal(&cur_entry->key, key)) { tduffy> if (NULL != p_data) tduffy> *p_data = cur_entry->data; tduffy> tduffy> @@ -177,14 +140,10 @@ u32 dat_dictionary_search(struct dat_dic tduffy> } tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_enumerate tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, void *array[], tduffy> int array_size) tduffy> { tduffy> @@ -206,14 +165,10 @@ u32 dat_dictionary_enumerate(struct dat_ tduffy> list_for_each_entry(cur_entry, &dictionary->list, list) tduffy> array[i++] = cur_entry->data; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_remove tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_dictionary_remove(struct dat_dictionary *dictionary, tduffy> const struct dat_provider_info *key, tduffy> struct dat_dictionary_entry **p_entry, void **p_data) tduffy> @@ -231,7 +186,7 @@ u32 dat_dictionary_remove(struct dat_dic tduffy> list_for_each_safe(cur_list, next_list, &dictionary->list) { tduffy> cur_entry = list_entry(cur_list, struct dat_dictionary_entry, tduffy> list); tduffy> - if (TRUE == dat_dictionary_key_is_equal(&cur_entry->key, key)) { tduffy> + if (TRUE == dat_dict_key_is_equal(&cur_entry->key, key)) { tduffy> if (NULL != p_data) tduffy> *p_data = cur_entry->data; tduffy> tduffy> @@ -246,22 +201,12 @@ u32 dat_dictionary_remove(struct dat_dic tduffy> } tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Internal Function Definitions * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_key_create tduffy> - ***********************************************************************/ tduffy> - tduffy> -u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, tduffy> - struct dat_provider_info *new_key) tduffy> +static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, tduffy> + struct dat_provider_info *new_key) tduffy> { tduffy> BUG_ON(NULL == old_key); tduffy> BUG_ON(NULL == new_key); tduffy> @@ -274,12 +219,8 @@ u32 dat_dictionary_key_dup(const struct tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dictionary_key_is_equal tduffy> - ***********************************************************************/ tduffy> - tduffy> -boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, tduffy> - const struct dat_provider_info *key_b) tduffy> +static boolean_t dat_dict_key_is_equal(const struct dat_provider_info *key_a, tduffy> + const struct dat_provider_info *key_b) tduffy> { tduffy> BUG_ON(NULL == key_a); tduffy> BUG_ON(NULL == key_b); tduffy> @@ -288,9 +229,8 @@ boolean_t dat_dictionary_key_is_equal(co tduffy> (!strncmp(key_a->ia_name, key_b->ia_name, strlen(key_a->ia_name))) tduffy> && (key_a->dat_version_major == key_b->dat_version_major) tduffy> && (key_a->dat_version_minor == key_b->dat_version_minor) tduffy> - && (key_a->is_thread_safe == key_b->is_thread_safe)) { tduffy> + && (key_a->is_thread_safe == key_b->is_thread_safe)) tduffy> return TRUE; tduffy> - } else { tduffy> + else tduffy> return FALSE; tduffy> - } tduffy> } tduffy> Index: linux-kernel-new/dat/dr.c tduffy> =================================================================== tduffy> --- linux-kernel-new/dat/dr.c (revision 2518) tduffy> +++ linux-kernel-new/dat/dr.c (working copy) tduffy> @@ -35,47 +35,21 @@ tduffy> #include "dictionary.h" tduffy> #include "dr.h" tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Strucutres * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> struct dat_dr_entry { tduffy> int ref_count; tduffy> struct dat_provider_info info; tduffy> DAT_IA_OPEN_FUNC ia_open_func; tduffy> }; tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Global Variables * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> static spinlock_t g_dr_lock = SPIN_LOCK_UNLOCKED; tduffy> static struct dat_dictionary g_dr_dictionary = tduffy> DAT_DICTIONARY_INIT(g_dr_dictionary); tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * External Functions * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> -/************************************************************************ tduffy> - * Function: dat_dr_fini tduffy> - ************************************************************************/ tduffy> - tduffy> void dat_dr_fini(void) tduffy> { tduffy> dat_dictionary_destroy(&g_dr_dictionary); tduffy> } tduffy> tduffy> -/************************************************************************ tduffy> - * Function: dat_dr_insert tduffy> - ************************************************************************/ tduffy> - tduffy> u32 dat_dr_insert(const struct dat_provider_info *info, tduffy> const DAT_IA_OPEN_FUNC ia_open_func) tduffy> { tduffy> @@ -107,7 +81,7 @@ u32 dat_dr_insert(const struct dat_provi tduffy> tduffy> spin_unlock_irqrestore(&g_dr_lock, flags); tduffy> tduffy> - bail: tduffy> +bail: tduffy> if (DAT_SUCCESS != status) { tduffy> if (NULL != data) tduffy> kfree(data); tduffy> @@ -119,10 +93,6 @@ u32 dat_dr_insert(const struct dat_provi tduffy> return status; tduffy> } tduffy> tduffy> -/************************************************************************ tduffy> - * Function: dat_dr_remove tduffy> - ************************************************************************/ tduffy> - tduffy> u32 dat_dr_remove(const struct dat_provider_info *info) tduffy> { tduffy> struct dat_dr_entry *data; tduffy> @@ -150,7 +120,7 @@ u32 dat_dr_remove(const struct dat_provi tduffy> tduffy> kfree(data); tduffy> tduffy> - bail: tduffy> +bail: tduffy> spin_unlock_irqrestore(&g_dr_lock, flags); tduffy> tduffy> if (NULL != dict_entry) tduffy> @@ -159,10 +129,6 @@ u32 dat_dr_remove(const struct dat_provi tduffy> return status; tduffy> } tduffy> tduffy> -/************************************************************************ tduffy> - * Function: dat_dr_provider_open tduffy> - ************************************************************************/ tduffy> - tduffy> u32 dat_dr_provider_open(const struct dat_provider_info *info, tduffy> DAT_IA_OPEN_FUNC *p_ia_open_func) tduffy> { tduffy> @@ -184,10 +150,6 @@ u32 dat_dr_provider_open(const struct da tduffy> return status; tduffy> } tduffy> tduffy> -/************************************************************************ tduffy> - * Function: dat_dr_provider_close tduffy> - ************************************************************************/ tduffy> - tduffy> u32 dat_dr_provider_close(const struct dat_provider_info *info) tduffy> { tduffy> struct dat_dr_entry *data; tduffy> @@ -206,19 +168,11 @@ u32 dat_dr_provider_close(const struct d tduffy> return status; tduffy> } tduffy> tduffy> -/************************************************************************ tduffy> - * Function: dat_dr_size tduffy> - ************************************************************************/ tduffy> - tduffy> u32 dat_dr_size(int *size) tduffy> { tduffy> return dat_dictionary_size(&g_dr_dictionary, size); tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_dr_list tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_dr_list(int max_to_return, int *entries_returned, tduffy> struct dat_provider_info *(dat_provider_list[])) tduffy> { tduffy> @@ -227,9 +181,11 @@ u32 dat_dr_list(int max_to_return, int * tduffy> unsigned long flags; tduffy> u32 status = DAT_SUCCESS; tduffy> tduffy> - /* The dictionary size may increase between the call to */ tduffy> - /* dat_dictionary_size() and dat_dictionary_enumerate(). */ tduffy> - /* Therefore we loop until a successful enumeration is made. */ tduffy> + /* tduffy> + * The dictionary size may increase between the call to tduffy> + * dat_dictionary_size() and dat_dictionary_enumerate(). tduffy> + * Therefore we loop until a successful enumeration is made. tduffy> + */ tduffy> *entries_returned = 0; tduffy> for (;;) { tduffy> status = dat_dictionary_size(&g_dr_dictionary, &array_size); tduffy> @@ -256,9 +212,9 @@ u32 dat_dr_list(int max_to_return, int * tduffy> tduffy> spin_unlock_irqrestore(&g_dr_lock, flags); tduffy> tduffy> - if (DAT_SUCCESS == status) { tduffy> + if (DAT_SUCCESS == status) tduffy> break; tduffy> - } else { tduffy> + else { tduffy> kfree(array); tduffy> array = NULL; tduffy> continue; tduffy> @@ -277,10 +233,9 @@ u32 dat_dr_list(int max_to_return, int * tduffy> tduffy> *entries_returned = i; tduffy> tduffy> - bail: tduffy> - if (NULL != array) { tduffy> +bail: tduffy> + if (NULL != array) tduffy> kfree(array); tduffy> - } tduffy> tduffy> return status; tduffy> } tduffy> Index: linux-kernel-new/dat/core.c tduffy> =================================================================== tduffy> --- linux-kernel-new/dat/core.c (revision 2518) tduffy> +++ linux-kernel-new/dat/core.c (working copy) tduffy> @@ -38,26 +38,10 @@ MODULE_LICENSE("Dual BSD/GPL"); tduffy> MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); tduffy> MODULE_AUTHOR("James Lentini"); tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Global Variables * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> static DAT_DBG_MASK g_dbg_mask = DAT_DBG_TYPE_ERROR; tduffy> module_param_named(dbg_mask, g_dbg_mask, int, 0644); tduffy> MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug message types."); tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Function Definitions * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> -/*********************************************************************** tduffy> - * Function: dat_dbg_print tduffy> - ***********************************************************************/ tduffy> - tduffy> void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...) tduffy> { tduffy> static char buf[1024]; tduffy> @@ -72,10 +56,6 @@ void dat_dbg_print(DAT_DBG_TYPE type, co tduffy> } tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_init tduffy> - ***********************************************************************/ tduffy> - tduffy> static int __init dat_init(void) tduffy> { tduffy> dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry started\n"); tduffy> @@ -84,10 +64,6 @@ static int __init dat_init(void) tduffy> tduffy> module_init(dat_init); tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_fini tduffy> - ***********************************************************************/ tduffy> - tduffy> static void __exit dat_fini(void) tduffy> { tduffy> dat_dr_fini(); tduffy> Index: linux-kernel-new/dat/api.c tduffy> =================================================================== tduffy> --- linux-kernel-new/dat/api.c (revision 2518) tduffy> +++ linux-kernel-new/dat/api.c (working copy) tduffy> @@ -34,547 +34,321 @@ tduffy> #include "core.h" tduffy> #include "dr.h" tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Internal Function Definitions * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> -/*********************************************************************** tduffy> - * Function: dat_strerror_major tduffy> - ***********************************************************************/ tduffy> - tduffy> static u32 dat_strerror_major(u32 value, const char **message) tduffy> { tduffy> switch (DAT_GET_TYPE(value)) { tduffy> case DAT_SUCCESS: tduffy> - { tduffy> - *message = "DAT_SUCCESS"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_SUCCESS"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_ABORT: tduffy> - { tduffy> - *message = "DAT_ABORT"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_ABORT"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_CONN_QUAL_IN_USE: tduffy> - { tduffy> - *message = "DAT_CONN_QUAL_IN_USE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_CONN_QUAL_IN_USE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INSUFFICIENT_RESOURCES: tduffy> - { tduffy> - *message = "DAT_INSUFFICIENT_RESOURCES"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INSUFFICIENT_RESOURCES"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INTERNAL_ERROR: tduffy> - { tduffy> - *message = "DAT_INTERNAL_ERROR"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INTERNAL_ERROR"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_PARAMETER: tduffy> - { tduffy> - *message = "DAT_INVALID_PARAMETER"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_PARAMETER"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_LENGTH_ERROR: tduffy> - { tduffy> - *message = "DAT_LENGTH_ERROR"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_LENGTH_ERROR"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_MODEL_NOT_SUPPORTED: tduffy> - { tduffy> - *message = "DAT_MODEL_NOT_SUPPORTED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_MODEL_NOT_SUPPORTED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PROVIDER_NOT_FOUND: tduffy> - { tduffy> - *message = "DAT_PROVIDER_NOT_FOUND"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PROVIDER_NOT_FOUND"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PRIVILEGES_VIOLATION: tduffy> - { tduffy> - *message = "DAT_PRIVILEGES_VIOLATION"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PRIVILEGES_VIOLATION"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PROTECTION_VIOLATION: tduffy> - { tduffy> - *message = "DAT_PROTECTION_VIOLATION"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PROTECTION_VIOLATION"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_QUEUE_EMPTY: tduffy> - { tduffy> - *message = "DAT_QUEUE_EMPTY"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_QUEUE_EMPTY"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_QUEUE_FULL: tduffy> - { tduffy> - *message = "DAT_QUEUE_FULL"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_QUEUE_FULL"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_TIMEOUT_EXPIRED: tduffy> - { tduffy> - *message = "DAT_TIMEOUT_EXPIRED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_TIMEOUT_EXPIRED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PROVIDER_ALREADY_REGISTERED: tduffy> - { tduffy> - *message = "DAT_PROVIDER_ALREADY_REGISTERED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PROVIDER_ALREADY_REGISTERED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PROVIDER_IN_USE: tduffy> - { tduffy> - *message = "DAT_PROVIDER_IN_USE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PROVIDER_IN_USE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ADDRESS: tduffy> - { tduffy> - *message = "DAT_INVALID_ADDRESS"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ADDRESS"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INTERRUPTED_CALL: tduffy> - { tduffy> - *message = "DAT_INTERRUPTED_CALL"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INTERRUPTED_CALL"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_NOT_IMPLEMENTED: tduffy> - { tduffy> - *message = "DAT_NOT_IMPLEMENTED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_NOT_IMPLEMENTED"; tduffy> + return DAT_SUCCESS; tduffy> default: tduffy> - { tduffy> - *message = "unknown error"; tduffy> - return DAT_INVALID_PARAMETER; tduffy> - } tduffy> + *message = "unknown error"; tduffy> + return DAT_INVALID_PARAMETER; tduffy> } tduffy> } tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_strerror_minor tduffy> - ***********************************************************************/ tduffy> static u32 dat_strerror_minor(u32 value, const char **message) tduffy> { tduffy> switch (DAT_GET_SUBTYPE(value)) { tduffy> - tduffy> case DAT_NO_SUBTYPE: tduffy> - { tduffy> - *message = ""; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = ""; tduffy> + return DAT_SUCCESS; tduffy> case DAT_SUB_INTERRUPTED: tduffy> - { tduffy> - *message = "DAT_SUB_INTERRUPTED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_SUB_INTERRUPTED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_RESOURCE_MEMORY: tduffy> - { tduffy> - *message = "DAT_RESOURCE_MEMORY"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_RESOURCE_MEMORY"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_RESOURCE_DEVICE: tduffy> - { tduffy> - *message = "DAT_RESOURCE_DEVICE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_RESOURCE_DEVICE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_RESOURCE_TEP: tduffy> - { tduffy> - *message = "DAT_RESOURCE_TEP"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_RESOURCE_TEP"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_RESOURCE_TEVD: tduffy> - { tduffy> - *message = "DAT_RESOURCE_TEVD"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_RESOURCE_TEVD"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_RESOURCE_PROTECTION_DOMAIN: tduffy> - { tduffy> - *message = "DAT_RESOURCE_PROTECTION_DOMAIN"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_RESOURCE_PROTECTION_DOMAIN"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_RESOURCE_MEMORY_REGION: tduffy> - { tduffy> - *message = "DAT_RESOURCE_MEMORY_REGION"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_RESOURCE_MEMORY_REGION"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_RESOURCE_ERROR_HANDLER: tduffy> - { tduffy> - *message = "DAT_RESOURCE_ERROR_HANDLER"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_RESOURCE_ERROR_HANDLER"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_RESOURCE_CREDITS: tduffy> - { tduffy> - *message = "DAT_RESOURCE_CREDITS"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_RESOURCE_CREDITS"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_IA: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_IA"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_IA"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_EP: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_EP"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_EP"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_LMR: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_LMR"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_LMR"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_RMR: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_RMR"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_RMR"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_PZ: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_PZ"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_PZ"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_PSP: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_PSP"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_PSP"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_RSP: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_RSP"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_RSP"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_CR: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_CR"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_CR"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_CNO: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_CNO"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_CNO"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_EVD_CR: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_EVD_CR"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_EVD_CR"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_EVD_REQUEST: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_EVD_REQUEST"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_EVD_REQUEST"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_EVD_RECV: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_EVD_RECV"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_EVD_RECV"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_EVD_CONN: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_EVD_CONN"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_EVD_CONN"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_HANDLE_EVD_ASYNC: tduffy> - { tduffy> - *message = "DAT_INVALID_HANDLE_EVD_ASYNC"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_HANDLE_EVD_ASYNC"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG1: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG1"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG1"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG2: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG2"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG2"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG3: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG3"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG3"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG4: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG4"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG4"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG5: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG5"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG5"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG6: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG6"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG6"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG7: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG7"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG7"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG8: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG8"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG8"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG9: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG9"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG9"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ARG10: tduffy> - { tduffy> - *message = "DAT_INVALID_ARG10"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ARG10"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_UNCONNECTED: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_UNCONNECTED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_UNCONNECTED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_ACTCONNPENDING: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_ACTCONNPENDING"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_ACTCONNPENDING"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_PASSCONNPENDING: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_PASSCONNPENDING"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_PASSCONNPENDING"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_TENTCONNPENDING: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_TENTCONNPENDING"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_TENTCONNPENDING"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_CONNECTED: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_CONNECTED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_CONNECTED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_DISCONNECTED: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_DISCONNECTED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_DISCONNECTED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_RESERVED: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_RESERVED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_RESERVED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_COMPLPENDING: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_COMPLPENDING"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_COMPLPENDING"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_DISCPENDING: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_DISCPENDING"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_DISCPENDING"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_PROVIDERCONTROL: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_PROVIDERCONTROL"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_PROVIDERCONTROL"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EP_NOTREADY: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EP_NOTREADY"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EP_NOTREADY"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_CNO_IN_USE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_CNO_IN_USE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_CNO_IN_USE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_CNO_DEAD: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_CNO_DEAD"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_CNO_DEAD"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_OPEN: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_OPEN"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_OPEN"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_ENABLED: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_ENABLED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_ENABLED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_DISABLED: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_DISABLED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_DISABLED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_WAITABLE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_WAITABLE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_WAITABLE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_UNWAITABLE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_UNWAITABLE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_UNWAITABLE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_IN_USE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_IN_USE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_IN_USE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_CONFIG_NOTIFY: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_CONFIG_NOTIFY"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_CONFIG_NOTIFY"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_CONFIG_SOLICITED: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_CONFIG_SOLICITED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_CONFIG_SOLICITED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_CONFIG_THRESHOLD: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_CONFIG_THRESHOLD"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_CONFIG_THRESHOLD"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_WAITER: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_WAITER"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_WAITER"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_EVD_ASYNC: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_EVD_ASYNC"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_EVD_ASYNC"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_IA_IN_USE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_IA_IN_USE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_IA_IN_USE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_LMR_IN_USE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_LMR_IN_USE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_LMR_IN_USE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_LMR_FREE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_LMR_FREE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_LMR_FREE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_PZ_IN_USE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_PZ_IN_USE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_PZ_IN_USE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_STATE_PZ_FREE: tduffy> - { tduffy> - *message = "DAT_INVALID_STATE_PZ_FREE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_STATE_PZ_FREE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PRIVILEGES_READ: tduffy> - { tduffy> - *message = "DAT_PRIVILEGES_READ"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PRIVILEGES_READ"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PRIVILEGES_WRITE: tduffy> - { tduffy> - *message = "DAT_PRIVILEGES_WRITE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PRIVILEGES_WRITE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PRIVILEGES_RDMA_READ: tduffy> - { tduffy> - *message = "DAT_PRIVILEGES_RDMA_READ"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PRIVILEGES_RDMA_READ"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PRIVILEGES_RDMA_WRITE: tduffy> - { tduffy> - *message = "DAT_PRIVILEGES_RDMA_WRITE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PRIVILEGES_RDMA_WRITE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PROTECTION_READ: tduffy> - { tduffy> - *message = "DAT_PROTECTION_READ"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PROTECTION_READ"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PROTECTION_WRITE: tduffy> - { tduffy> - *message = "DAT_PROTECTION_WRITE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PROTECTION_WRITE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PROTECTION_RDMA_READ: tduffy> - { tduffy> - *message = "DAT_PROTECTION_RDMA_READ"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PROTECTION_RDMA_READ"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_PROTECTION_RDMA_WRITE: tduffy> - { tduffy> - *message = "DAT_PROTECTION_RDMA_WRITE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_PROTECTION_RDMA_WRITE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ADDRESS_UNSUPPORTED: tduffy> - { tduffy> - *message = "DAT_INVALID_ADDRESS_UNSUPPORTED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ADDRESS_UNSUPPORTED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ADDRESS_UNREACHABLE: tduffy> - { tduffy> - *message = "DAT_INVALID_ADDRESS_UNREACHABLE"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ADDRESS_UNREACHABLE"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_INVALID_ADDRESS_MALFORMED: tduffy> - { tduffy> - *message = "DAT_INVALID_ADDRESS_MALFORMED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_INVALID_ADDRESS_MALFORMED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_NAME_NOT_REGISTERED: tduffy> - { tduffy> - *message = "DAT_NAME_NOT_REGISTERED"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_NAME_NOT_REGISTERED"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_MAJOR_NOT_FOUND: tduffy> - { tduffy> - *message = "DAT_MAJOR_NOT_FOUND"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_MAJOR_NOT_FOUND"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_MINOR_NOT_FOUND: tduffy> - { tduffy> - *message = "DAT_MINOR_NOT_FOUND"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_MINOR_NOT_FOUND"; tduffy> + return DAT_SUCCESS; tduffy> case DAT_THREAD_SAFETY_NOT_FOUND: tduffy> - { tduffy> - *message = "DAT_THREAD_SAFETY_NOT_FOUND"; tduffy> - return DAT_SUCCESS; tduffy> - } tduffy> + *message = "DAT_THREAD_SAFETY_NOT_FOUND"; tduffy> + return DAT_SUCCESS; tduffy> default: tduffy> - { tduffy> - *message = "unknown minor error"; tduffy> - return DAT_INVALID_PARAMETER; tduffy> - } tduffy> + *message = "unknown minor error"; tduffy> + return DAT_INVALID_PARAMETER; tduffy> } tduffy> } tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * External Function Definitions * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> -/*********************************************************************** tduffy> - * Function: dat_ia_openv tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_ia_openv(const char *name, int async_event_qlen, tduffy> DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, tduffy> u32 dat_major, u32 dat_minor, boolean_t thread_safety) tduffy> @@ -589,15 +363,13 @@ u32 dat_ia_openv(const char *name, int a tduffy> __func__, name, async_event_qlen, dat_major, dat_minor, tduffy> thread_safety); tduffy> tduffy> - if (DAT_IS_BAD_POINTER(name)) { tduffy> + if (DAT_IS_BAD_POINTER(name)) tduffy> return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); tduffy> - } tduffy> tduffy> len = strlen(name); tduffy> tduffy> - if (DAT_NAME_MAX_LENGTH < len) { tduffy> + if (DAT_NAME_MAX_LENGTH < len) tduffy> return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); tduffy> - } tduffy> tduffy> strncpy(info.ia_name, name, len); tduffy> info.ia_name[len] = '\0'; tduffy> @@ -615,16 +387,12 @@ u32 dat_ia_openv(const char *name, int a tduffy> return status; tduffy> } tduffy> tduffy> - return ia_open_func( tduffy> - name, async_event_qlen, async_event_handle, ia_handle); tduffy> + return ia_open_func(name, async_event_qlen, async_event_handle, tduffy> + ia_handle); tduffy> } tduffy> tduffy> EXPORT_SYMBOL(dat_ia_openv); tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_ia_close tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags flags) tduffy> { tduffy> struct dat_provider *provider; tduffy> @@ -638,15 +406,15 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle tduffy> provider = DAT_HANDLE_TO_PROVIDER(ia_handle); tduffy> ia_name = provider->device_name; tduffy> tduffy> - if (DAT_SUCCESS != (status = dat_ia_query( tduffy> - ia_handle, NULL, NULL, &provider_attr))) { tduffy> + if (DAT_SUCCESS != (status = dat_ia_query(ia_handle, NULL, NULL, tduffy> + &provider_attr))) tduffy> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, tduffy> "dat_ia_query for IA %s failed\n", ia_name); tduffy> - } else if (DAT_SUCCESS != tduffy> - (status = provider->ia_close_func(ia_handle, flags))) { tduffy> + else if (DAT_SUCCESS != (status = provider->ia_close_func(ia_handle, tduffy> + flags))) tduffy> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, tduffy> "close function for IA %s failed\n", ia_name); tduffy> - } else { tduffy> + else { tduffy> struct dat_provider_info info; tduffy> size_t len; tduffy> tduffy> @@ -662,12 +430,11 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle tduffy> info.is_thread_safe = provider_attr.is_thread_safe; tduffy> tduffy> status = dat_dr_provider_close(&info); tduffy> - if (DAT_SUCCESS != status) { tduffy> + if (DAT_SUCCESS != status) tduffy> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, tduffy> "dynamic registry unable to close " tduffy> "provider for IA %s\n", tduffy> ia_name); tduffy> - } tduffy> } tduffy> tduffy> return status; tduffy> @@ -675,10 +442,6 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle tduffy> tduffy> EXPORT_SYMBOL(dat_ia_close); tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_strerror tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_strerror(u32 value, const char **major_message, tduffy> const char **minor_message) tduffy> { tduffy> @@ -695,23 +458,17 @@ u32 dat_strerror(u32 value, const char * tduffy> * implementation does not follow the suggested implementation. tduffy> */ tduffy> tduffy> - if (DAT_SUCCESS != dat_strerror_major(value, major_message)) { tduffy> + if (DAT_SUCCESS != dat_strerror_major(value, major_message)) tduffy> return DAT_INVALID_PARAMETER; tduffy> - } else if (minor_message != NULL) { tduffy> - if (DAT_SUCCESS != dat_strerror_minor(value, minor_message)) { tduffy> + else if (minor_message != NULL) tduffy> + if (DAT_SUCCESS != dat_strerror_minor(value, minor_message)) tduffy> return DAT_INVALID_PARAMETER; tduffy> - } tduffy> - } tduffy> tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> EXPORT_SYMBOL(dat_strerror); tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_registry_add_provider tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_registry_add_provider(const struct dat_provider *provider, tduffy> const struct dat_provider_info *provider_info) tduffy> { tduffy> @@ -733,10 +490,6 @@ u32 dat_registry_add_provider(const stru tduffy> tduffy> EXPORT_SYMBOL(dat_registry_remove_provider); tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_registry_remove_provider tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_registry_remove_provider(const struct dat_provider *provider, tduffy> const struct dat_provider_info *provider_info) tduffy> { tduffy> @@ -751,10 +504,6 @@ u32 dat_registry_remove_provider(const s tduffy> tduffy> EXPORT_SYMBOL(dat_registry_add_provider); tduffy> tduffy> -/*********************************************************************** tduffy> - * Function: dat_registry_list_providers tduffy> - ***********************************************************************/ tduffy> - tduffy> u32 dat_registry_list_providers(int max_to_return, int *entries_returned, tduffy> struct dat_provider_info *(dat_provider_list[])) tduffy> { tduffy> @@ -779,10 +528,9 @@ u32 dat_registry_list_providers(int max_ tduffy> */ tduffy> (void)dat_dr_size(entries_returned); tduffy> return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); tduffy> - } else { tduffy> + } else tduffy> status = dat_dr_list(max_to_return, entries_returned, tduffy> dat_provider_list); tduffy> - } tduffy> return status; tduffy> } tduffy> tduffy> Index: linux-kernel-new/dat/dr.h tduffy> =================================================================== tduffy> --- linux-kernel-new/dat/dr.h (revision 2518) tduffy> +++ linux-kernel-new/dat/dr.h (working copy) tduffy> @@ -36,12 +36,6 @@ tduffy> tduffy> #include tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Function Declarations * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> extern void dat_dr_fini(void); tduffy> tduffy> extern u32 dat_dr_insert(const struct dat_provider_info *info, tduffy> From itamar at mellanox.co.il Thu Jun 2 12:55:29 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 2 Jun 2005 22:55:29 +0300 Subject: [openib-general] RE: station hang with the last patch to infiniband/core/sa_query. c Message-ID: <91DB792C7985D411BEC300B40080D29CC35C21@mtvex01.mtv.mtl.com> Thanks , ping is working now (;-) Itamar > -----Original Message----- > From: Roland Dreier [mailto:roland at topspin.com] > Sent: Thursday, June 02, 2005 7:05 PM > To: Itamar Rabenstein > Cc: openib-general at openib.org > Subject: Re: station hang with the last patch to > infiniband/core/sa_query.c > > > Thanks, I found a problem with that patch and checked in a fix. Can > you let me know if it works better now? > > - R. > From itamar at mellanox.co.il Thu Jun 2 13:02:36 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 2 Jun 2005 23:02:36 +0300 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more Message-ID: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> Hi , I am not an expert to pci hot plug but as far as know we currently dont have away in kdapl to inform the consumer of kdapl that a provider is going to be unloaded. pci hot plug should enable any low level driver (openib) to inform high level driver(kdapl) that the devide is going down. In openib gen2 this is doen with Ccall back remove_one(). how it is doen in current kdapl ? how the consumer who opened IA will know that this card is going down? Itamar > -----Original Message----- > From: Caitlin Bestler [mailto:caitlin.bestler at gmail.com] > Sent: Thursday, June 02, 2005 7:57 PM > To: Itamar Rabenstein > Cc: Christoph Hellwig; Tom Duffy; openib-general at openib.org > Subject: Re: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more > > > Why is it that you believe that the DAT registry does not support > plug and play? The interface was most specifically designed > to allow that. > > When a device is plugged in, the driver is loaded by existing > OS mechanisms. It can then load the provider code (if needed). > Whenever the provider module is loaded it registers itself with > the registry, and it unregisters itself before it is unloaded. > > Keep in mind that loading/unloading DAT Provider is *not* > synonymous with loading/unloading drivers. In fact I believe > the intent is to have a single provider that supports multiple > devices. Such a provider would simply register itself when > it was loaded. > > I could see adding new interfaces to track which devices > are loaded/unloaded for use *by* the DAT provider, but > I do not see how that changes the function of the DAT > registry, which is to support multiple DAT Providers. > > On 6/1/05, Itamar Rabenstein wrote: > > I think that if we want to support pci hot plug we need to > change the kdapl > > IF > > to be like something like the gen2 IF. > > a link lins of the providers with register/unregister > client functions > > which will register a CB function. > > we only need to have 4 functions > > register_client , unregister_client register_provider , > unregister_provider > > > > Itamar > > > > > -----Original Message----- > > > From: Christoph Hellwig [mailto:hch at lst.de] > > > Sent: Wednesday, June 01, 2005 9:36 PM > > > To: Tom Duffy > > > Cc: openib-general at openib.org > > > Subject: Re: [openib-general] [PATCH] kDAPL: cleanup dat/ > a bit more > > > > > > > > > On Wed, Jun 01, 2005 at 11:23:44AM -0700, Tom Duffy wrote: > > > > This patch cleans up dat/ a bit, removing unneeded parenthesis, > > > > gratuitous comments, and a few whitespace cleanups. > Please apply. > > > > > > btw, I think most of the code under dat/ should just go away ASAP. > > > Just have a simple linked list of providers with trivial > > > registration/unregisteration/find_get/put functions like > all the other > > > subsystems and kill all the dictionaty mess and provider vs > > > provider_info braindamage. > > > > > > _______________________________________________ > > > openib-general mailing list > > > openib-general at openib.org > > > http://openib.org/mailman/listinfo/openib-general > > > > > > To unsubscribe, please visit > > > http://openib.org/mailman/listinfo/openib-general > > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > > From itamar at mellanox.co.il Thu Jun 2 13:06:28 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 2 Jun 2005 23:06:28 +0300 Subject: [openib-general] kdaptest wedges server Message-ID: <91DB792C7985D411BEC300B40080D29CC35C23@mtvex01.mtv.mtl.com> Hi Josh, Can you run it with debug option ( -d ) and tell me what is the last debug print on the server side ? when you say "kill" can you hit ^c or is it hanging the station ? what is system configuration ? (arch , Os ,kernel) Itamar > -----Original Message----- > From: Josh England [mailto:jjengla at sandia.gov] > Sent: Thursday, June 02, 2005 9:00 PM > To: openib-general > Subject: [openib-general] kdaptest wedges server > > > Hi, > > The kdapltest program wedges consistently wedges a node. I just tried > it with Revision 2523, and it runs fine the first couple of > times. The > third time took a while, and the fourth time killed the server. > > The server is running: > kdapltest -T S -D mthca0a > > The client runs this several times in succession: > kdapltest -T T -D mthca0a -s client SR 256 server SR 256 > > -JE > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From sean.hefty at intel.com Thu Jun 2 12:55:34 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 2 Jun 2005 12:55:34 -0700 Subject: [openib-general] UCM accesses internal cm_id state Message-ID: The ucm code in ib_ucm_event_handler() reads the cm_id->state information in response to an event occurrence. The cm_id state can change dynamically in response to an event, and in general should not be accessed. (I.e. the current cm_id state may not match up with the event being reported.) Access to the state requires coordination with the CM code itself, so trying to rely on it to drive external states will eventually lead to problems. Clients of the CM should either maintain their own state information or perform operations based on the occurrence of the reported event. I had originally exposed the state only for debugging purposes. A comment was added that it should only be used for internal/debug use, but I will make a note to remove the state from the externally visible cm_id structure. The kDAPL code was recently converted to be event driven rather than state driven, so you may want to examine that code as an example. - Sean From bjordan at sbc2.infiniconsys.com Thu Jun 2 12:57:06 2005 From: bjordan at sbc2.infiniconsys.com (Bill Jordan) Date: Thu, 2 Jun 2005 15:57:06 -0400 Subject: [openib-general] [PATCH]libibcm: updated README for device node configuration Message-ID: <200506021957.j52Jv6Q19238@sbc2.infiniconsys.com> Patch to add basic content to the CM library README file. Add build instructions, install directory, and device node configuration. Signed-off-by: Bill Jordan Index: README =================================================================== --- README (revision 2523) +++ README (working copy) @@ -0,0 +1,27 @@ +This README is for userspace cm library. + +Building + +To make this directory, run: +./autogen.sh && ./configure && make && make install + +Typically the autogen and configure steps only need be done the first +time unless configure.in or Makefile.am changes. + +Libraries are installed by default at /usr/local/lib/infiniband. + +Device files + +The userspace CM uses a single device file regardless of the number +of adapters or ports present. + +To create the appropriate character device file automatically with +udev, a rule like + + KERNEL="ucm", NAME="infiniband/%k", MODE="0666" + +can be used. This will create the device node named + + /dev/infiniband/ucm + + From jlentini at netapp.com Thu Jun 2 13:18:13 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 16:18:13 -0400 (EDT) Subject: [openib-general] Re: [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117655324.19908.3.camel@duffman> References: <1117655324.19908.3.camel@duffman> Message-ID: I'd recommend these changes (see attached): - use the new microsecond conversion function you sent - keep a default DAT timeout constant (I propose DAT_TIMEOUT_MAX) - set the timeout a little differently. Instead of storing it in the ep, store it directly in the CM param - add a kdat.h file to the patches directory. We can start using this to place symbols that are in the official DAT spec (like DAT_TIMEOUT_INFINITE), but not our linux variant. Did I call dapl_convert_us_to_kookyib() correctly? You were passing a value divided by 1000 to dapl_convert_ms_to_kookyib(), so I assumed that I could get rid of that. What do you think? james On Wed, 1 Jun 2005, Tom Duffy wrote: > This patch takes into account some of the feedback by Sean, it sets the > retry to 0, so it will do the timeout right. It also stops the > conversion of ms to ib at 63 since 64 overflows. I have tested this > version using kdapltest in quit, performance, and transaction modes. > > James, do you think it good enough to apply? > > Signed-off-by: Tom Duffy > > Index: linux-kernel-timer/test/dapltest/test/dapl_performance_util.c > =================================================================== > --- linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (working copy) > @@ -538,7 +538,7 @@ DT_performance_wait ( > pre_ts = DT_Mdep_GetTimeStamp (); > > ret = DT_Tdep_evd_wait ( evd_handle, > - DAT_TIMEOUT_INFINITE, > + MAX_SCHEDULE_TIMEOUT, > &event); > > post_ts = DT_Mdep_GetTimeStamp (); > Index: linux-kernel-timer/test/dapltest/test/dapl_performance_client.c > =================================================================== > --- linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (working copy) > @@ -105,7 +105,7 @@ retry: > ret = dat_ep_connect (test_ptr->ep_context.ep_handle, > test_ptr->remote_ia_addr, > test_ptr->ep_context.port, > - DAT_TIMEOUT_INFINITE, > + MAX_SCHEDULE_TIMEOUT, > 0, > (void *) 0, /* no private data */ > test_ptr->cmd->qos, > Index: linux-kernel-timer/test/dapltest/test/dapl_server.c > =================================================================== > --- linux-kernel-timer/test/dapltest/test/dapl_server.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/test/dapl_server.c (working copy) > @@ -580,7 +580,7 @@ DT_cs_Server (Params_t * params_ptr) > > DT_Mdep_wait_object_wait ( > &pt_ptr->synch_wait_object, > - DAT_TIMEOUT_INFINITE); > + MAX_SCHEDULE_TIMEOUT); > > /* Send the Server_Info */ > DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); > Index: linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c > =================================================================== > --- linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (working copy) > @@ -712,7 +712,7 @@ retry: > ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, > test_ptr->remote_ia_addr, > test_ptr->ep_context[i].ia_port, > - DAT_TIMEOUT_INFINITE, > + MAX_SCHEDULE_TIMEOUT, > strlen (private_data_str), private_data_str, > pt_ptr->Params.ReliabilityLevel, > DAT_CONNECT_DEFAULT_FLAG); > Index: linux-kernel-timer/test/dapltest/test/dapl_test_util.c > =================================================================== > --- linux-kernel-timer/test/dapltest/test/dapl_test_util.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/test/dapl_test_util.c (working copy) > @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p > u32 ret; > struct dat_event event; > > - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); > + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); > if (ret != DAT_SUCCESS) > { > DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", > @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * > u32 ret; > struct dat_event event; > > - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); > + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); > if (ret != DAT_SUCCESS) > { > DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", > @@ -369,7 +369,7 @@ DT_disco_event_wait ( DT_Tdep_Print_Head > u32 ret; > struct dat_event event; > > - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); > + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); > if (ret != DAT_SUCCESS) > { > DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", > @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p > u32 ret; > struct dat_event event; > > - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); > + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); > if (ret != DAT_SUCCESS) > { > DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", > @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p > u32 ret; > struct dat_event event; > > - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); > + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); > if (ret != DAT_SUCCESS) > { > DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", > Index: linux-kernel-timer/test/dapltest/test/dapl_client.c > =================================================================== > --- linux-kernel-timer/test/dapltest/test/dapl_client.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/test/dapl_client.c (working copy) > @@ -256,7 +256,7 @@ retry: > ret = dat_ep_connect (ep_handle, > server_netaddr, > SERVER_PORT_NUMBER, > - DAT_TIMEOUT_INFINITE, > + MAX_SCHEDULE_TIMEOUT, > 0, (void *) 0, /* no private data */ > params_ptr->ReliabilityLevel, > DAT_CONNECT_DEFAULT_FLAG); > Index: linux-kernel-timer/test/dapltest/include/dapl_tdep.h > =================================================================== > --- linux-kernel-timer/test/dapltest/include/dapl_tdep.h (revision 2518) > +++ linux-kernel-timer/test/dapltest/include/dapl_tdep.h (working copy) > @@ -56,7 +56,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_ha > > u32 > DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, > - DAT_TIMEOUT timeout, > + unsigned long timeout, > struct dat_event *event); > u32 > DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, > Index: linux-kernel-timer/test/dapltest/include/dapl_proto.h > =================================================================== > --- linux-kernel-timer/test/dapltest/include/dapl_proto.h (revision 2518) > +++ linux-kernel-timer/test/dapltest/include/dapl_proto.h (working copy) > @@ -185,7 +185,7 @@ void DT_Mdep_Thread_EXIT ( void * thr > int DT_Mdep_wait_object_init ( DT_WAIT_OBJECT *wait_obj); > int DT_Mdep_wait_object_wait ( > DT_WAIT_OBJECT *wait_obj, > - int timeout_val); > + unsigned long timeout_val); > int DT_Mdep_wait_object_wakeup ( DT_WAIT_OBJECT *wait_obj); > int DT_Mdep_wait_object_destroy ( DT_WAIT_OBJECT *wait_obj); > > Index: linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h > =================================================================== > --- linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (revision 2518) > +++ linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (working copy) > @@ -79,7 +79,7 @@ typedef struct > Transaction_Cmd_t *cmd; > struct sockaddr * remote_ia_addr; > DAT_CONN_QUAL base_port; > - DAT_TIMEOUT time_out; > + unsigned long time_out; > int evd_length; > Thread *thread; > > Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c > =================================================================== > --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) > @@ -411,7 +411,7 @@ DT_Mdep_wait_object_init ( > int > DT_Mdep_wait_object_wait ( > DT_WAIT_OBJECT *wait_obj, > - int timeout_val) > + unsigned long timeout_val) > { > int dat_status; > int pthread_status; > Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c > =================================================================== > --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) > @@ -252,13 +252,13 @@ DT_Mdep_wait_object_init ( > int > DT_Mdep_wait_object_wait ( > DT_WAIT_OBJECT *wait_obj, > - int timeout_val) > + unsigned long timeout_val) > { > int expire; > int dat_status; > > dat_status = DAT_SUCCESS; > - if ( DAT_TIMEOUT_INFINITE == timeout_val) > + if ( MAX_SCHEDULE_TIMEOUT == timeout_val) > { > interruptible_sleep_on (wait_obj); > } > Index: linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c > =================================================================== > --- linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2518) > +++ linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) > @@ -213,7 +213,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ > > u32 > DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, > - DAT_TIMEOUT timeout, > + unsigned long timeout, > struct dat_event *dat_event) > { > u32 dat_status; > @@ -239,7 +239,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han > return dat_status; > } > /* wait for an event */ > - if (timeout == DAT_TIMEOUT_INFINITE) > + if (timeout == MAX_SCHEDULE_TIMEOUT) > { > rc = wait_event_interruptible (evd_ptr->wait_object, > evd_ptr->event_next != NULL); > Index: linux-kernel-timer/dat-provider/dapl_ep_connect.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_ep_connect.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_ep_connect.c (working copy) > @@ -35,7 +35,6 @@ > #include "dapl_ep_util.h" > #include "dapl_openib_util.h" > #include "dapl_evd_util.h" > -#include "dapl_timer_util.h" > > /* > * Request a connection be established between the local Endpoint > @@ -44,7 +43,7 @@ > */ > u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, > struct sockaddr *remote_ia_address, > - DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, > + DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, > int private_data_size, const void *private_data, > enum dat_qos qos, enum dat_connect_flags connect_flags) > { > @@ -277,23 +276,13 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han > spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); > if (ep_ptr->param.ep_state == > DAT_EP_STATE_ACTIVE_CONNECTION_PENDING > - && timeout != DAT_TIMEOUT_INFINITE) { > - ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, > - GFP_ATOMIC); > - if (!ep_ptr->cxn_timer) { > - dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, > - DAT_RESOURCE_MEMORY); > - goto bail; > - } > - > - dapl_timer_set(ep_ptr->cxn_timer, > - dapl_ep_timeout, ep_ptr, timeout); > - } > + && timeout != MAX_SCHEDULE_TIMEOUT) > + ep_ptr->timeout = timeout; > spin_unlock_irqrestore(&ep_ptr->header.lock, > ep_ptr->header.flags); > } > > - bail: > +bail: > dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, > "dapl_ep_connect () returns 0x%x\n", dat_status); > > Index: linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (working copy) > @@ -32,7 +32,6 @@ > #include "dapl.h" > #include "dapl_evd_util.h" > #include "dapl_ep_util.h" > -#include "dapl_timer_util.h" > > /* > * dapl_evd_connection_callback > @@ -87,15 +86,6 @@ void dapl_evd_connection_callback(struct > */ > spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); > > - /* > - * If a connection timer has been set up on this EP, cancel it now > - */ > - if (ep_ptr->cxn_timer != NULL) { > - dapl_timer_cancel(ep_ptr->cxn_timer); > - kfree(ep_ptr->cxn_timer); > - ep_ptr->cxn_timer = NULL; > - } > - > switch (event) { > case DAT_CONNECTION_EVENT_ESTABLISHED: > { > Index: linux-kernel-timer/dat-provider/dapl_ep_free.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_ep_free.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_ep_free.c (working copy) > @@ -34,7 +34,6 @@ > #include "dapl_ep_util.h" > #include "dapl_openib_util.h" > #include "dapl_ring_buffer_util.h" > -#include "dapl_timer_util.h" > > /* > * dapl_ep_free > @@ -107,12 +106,6 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle > */ > spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); > > - if (ep_ptr->cxn_timer != NULL) { > - dapl_timer_cancel(ep_ptr->cxn_timer); > - kfree(ep_ptr->cxn_timer); > - ep_ptr->cxn_timer = NULL; > - } > - > /* Remove the EP from the IA */ > dapl_ia_unlink_ep(ia_ptr, ep_ptr); > > Index: linux-kernel-timer/dat-provider/dapl_provider.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_provider.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_provider.c (working copy) > @@ -39,7 +39,6 @@ > #include "dapl_mr_util.h" > #include "dapl_util.h" > #include "dapl_openib_util.h" > -#include "dapl_timer_util.h" > > MODULE_LICENSE("Dual BSD/GPL"); > MODULE_DESCRIPTION("DAT Provider for InfiniBand"); > @@ -393,7 +392,6 @@ module_exit(dapl_fini); > static int __init dapl_init(void) > { > dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "provider started\n"); > - dapl_timer_init(); > return ib_register_client(&dapl_client); > } > > Index: linux-kernel-timer/dat-provider/dapl_openib_cm.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_openib_cm.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_openib_cm.c (working copy) > @@ -291,6 +291,26 @@ static int dapl_cm_passive_cb_handler(st > return ret; > } > > +/* > + * approximately transforms miliseconds to 4.096us*2^x > + * 63(+8) is max return > + */ > +static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) { > + unsigned long converged = 2; > + u8 i; > + > + if (2 > ms) > + return 8; > + > + for (i = 1; i < 63; i++) { > + if (converged >= ms) > + break; > + converged = 2*converged; > + } > + > + return i+8; > +} > + > static void dapl_path_comp_handler(u64 req_id, void *context, int rec_num) > { > struct dapl_cm_id *conn = context; > @@ -324,6 +344,12 @@ static void dapl_path_comp_handler(u64 r > if (conn->dapl_path.mtu > IB_MTU_1024) > conn->dapl_path.mtu = IB_MTU_1024; > > + if (conn->ep && conn->ep->timeout != 0) { > + conn->param.remote_cm_response_timeout = > + dapl_convert_ms_to_kookyib(conn->ep->timeout/1000); > + conn->param.retry_count = 0; > + } > + > conn->param.primary_path = &conn->dapl_path; > conn->param.alternate_path = NULL; > > Index: linux-kernel-timer/dat-provider/dapl_timer_util.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_timer_util.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_timer_util.c (working copy) > @@ -1,313 +0,0 @@ > -/* > - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > - * > - * This Software is licensed under one of the following licenses: > - * > - * 1) under the terms of the "Common Public License 1.0" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/cpl.php. > - * > - * 2) under the terms of the "The BSD License" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/bsd-license.php. > - * > - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > - * copy of which is available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/gpl-license.php. > - * > - * Licensee has the right to choose one of the above licenses. > - * > - * Redistributions of source code must retain the above copyright > - * notice and one of the license notices. > - * > - * Redistributions in binary form must reproduce both the above copyright > - * notice, one of the license notices in the documentation > - * and/or other materials provided with the distribution. > - */ > - > -/* > - * Routines to add and cancel timer records. A timer record > - * is put on the global timer queue. If the timer thread is > - * not running, start it. The timer thread will sleep > - * until a timer event or until a process wakes it up > - * to notice a new timer is available; we use a DAPL_WAIT_OBJ > - * for synchronization. > - * > - * If a timer is cancelled, it is simlpy removed from the > - * queue. The timer may wakeup and notice there is no timer > - * record to awaken at this time, so it will reset for the > - * next entry. When there are no timer records to manage, > - * the timer thread just sleeps until awakened. > - * > - * This file also contains the timer handler thread, > - * embodied in dapl_timer_thread(). > - * > - * $Id$ > - */ > - > -#include "dapl.h" > -#include "dapl_timer_util.h" > - > -struct dapl_timer_head { > - struct dapl_llist_entry *timer_list_head; > - spinlock_t lock; > - unsigned long flags; > - DAPL_OS_WAIT_OBJECT wait_object; > - int thread_id; > -}; > - > -static struct dapl_timer_head g_dapl_timer_head; > - > -static int dapl_timer_thread(void *arg); > - > -void dapl_timer_init(void) > -{ > - /* > - * Set up the timer thread elements. The timer thread isn't > - * started until it is actually needed > - */ > - g_dapl_timer_head.timer_list_head = NULL; > - spin_lock_init(&g_dapl_timer_head.lock); > - dapl_os_wait_object_init(&g_dapl_timer_head.wait_object); > - g_dapl_timer_head.thread_id = -1; > -} > - > -/* > - * dapl_timer_set > - * > - * Set a timer. The timer will invoke the specified function > - * after a number of useconds expires. > - * > - * Input: > - * timer User provided timer structure > - * func Function to invoke when timer expires > - * data Argument passed to func() > - * expires microseconds until timer fires > - * > - * Returns: > - * no return value > - * > - */ > -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), > - void *data, DAPL_OS_TIMEVAL expires) > -{ > - struct dapl_timer_entry *list_ptr; > - DAPL_OS_TIMEVAL cur_time; > - boolean_t wakeup_tmo_thread; > - > - /* > - * Start the timer thread the first time we need a timer > - */ > - if ( -1 == g_dapl_timer_head.thread_id ) { > - g_dapl_timer_head.thread_id = > - kernel_thread(dapl_timer_thread, &g_dapl_timer_head, 0); > - } > - > - dapl_llist_init_entry(&timer->list_entry); > - wakeup_tmo_thread = FALSE; > - dapl_os_get_time(&cur_time); > - timer->expires = cur_time + expires; /* calculate future time */ > - timer->function = func; > - timer->data = data; > - > - /* > - * Put the element on the queue: sorted by wakeup time, eariliest > - * first. > - */ > - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); > - /* > - * Deal with 3 cases due to our list structure: > - * 1) list is empty: become the list head > - * 2) New timer is sooner than list head: become the list head > - * 3) otherwise, sort the timer into the list, no need to wake > - * the timer thread up > - */ > - if (dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head)) { > - /* Case 1: add entry to head of list */ > - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, > - (struct dapl_llist_entry *) > - &timer->list_entry, > - timer); > - wakeup_tmo_thread = TRUE; > - } else { > - list_ptr = (struct dapl_timer_entry *) > - dapl_llist_peek_head(&g_dapl_timer_head.timer_list_head); > - > - if (timer->expires < list_ptr->expires) { > - /* Case 2: add entry to head of list */ > - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, > - (struct dapl_llist_entry *) > - &timer->list_entry, timer); > - wakeup_tmo_thread = TRUE; > - } else { > - /* Case 3: figure out where entry goes in sorted list */ > - list_ptr = > - dapl_llist_next_entry(&g_dapl_timer_head. > - timer_list_head, > - (struct dapl_llist_entry *) > - &list_ptr->list_entry); > - > - while (list_ptr != NULL) { > - if (timer->expires < list_ptr->expires) { > - dapl_llist_add_entry(&g_dapl_timer_head. > - timer_list_head, > - (struct dapl_llist_entry *) > - &list_ptr->list_entry, > - (struct dapl_llist_entry *) > - &timer->list_entry, > - timer); > - break; > - > - } > - list_ptr = > - dapl_llist_next_entry(&g_dapl_timer_head. > - timer_list_head, > - (struct dapl_llist_entry *) > - &list_ptr->list_entry); > - } > - if (list_ptr == NULL) { > - /* entry goes to the end of the list */ > - dapl_llist_add_tail(&g_dapl_timer_head. > - timer_list_head, > - (struct dapl_llist_entry *) > - &timer->list_entry, timer); > - } > - } > - > - } > - spin_unlock_irqrestore(&g_dapl_timer_head.lock, > - g_dapl_timer_head.flags); > - > - if (wakeup_tmo_thread == TRUE) { > - dapl_os_wait_object_wakeup(&g_dapl_timer_head.wait_object); > - } > - > - return DAT_SUCCESS; > -} > - > -/* > - * dapl_timer_cancel > - * > - * Cancel a timer. Simply deletes the timer with no function invocations > - * > - * Input: > - * timer User provided timer structure > - * > - * Returns: > - * no return value > - */ > -void dapl_timer_cancel(struct dapl_timer_entry *timer) > -{ > - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); > - /* > - * make sure the entry has not been removed by another thread > - */ > - if (!dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head) && > - timer->list_entry.list_head == &g_dapl_timer_head.timer_list_head) { > - dapl_llist_remove_entry(&g_dapl_timer_head.timer_list_head, > - (struct dapl_llist_entry *) > - &timer->list_entry); > - } > - /* > - * If this was the first entry on the queue we could awaken the > - * thread and have it reset the list; but it will just wake up > - * and find that the timer entry has been removed, then go back > - * to sleep, so don't bother. > - */ > - spin_unlock_irqrestore(&g_dapl_timer_head.lock, > - g_dapl_timer_head.flags); > -} > - > -/* > - * dapl_timer_thread > - * > - * Core worker thread dealing with all timers. Basic algorithm: > - * - Sleep until work shows up > - * - Take first element of sorted timer list and wake > - * invoke the callback if expired > - * - Sleep for the timeout period if not expired > - * > - * Input: > - * timer_head Timer head structure to manage timer lists > - * > - * Returns: > - * int > - */ > -int dapl_timer_thread(void *arg) > -{ > - struct dapl_timer_entry *list_ptr; > - DAPL_OS_TIMEVAL cur_time; > - struct dapl_timer_head *timer_head; > - u32 dat_status; > - > - timer_head = arg; > - > - for (;;) { > - if (dapl_llist_is_empty(&timer_head->timer_list_head)) { > - dat_status = > - dapl_os_wait_object_wait(&timer_head->wait_object, > - DAT_TIMEOUT_INFINITE); > - } > - > - /* > - * Lock policy: > - * While this thread is accessing the timer list, it holds the > - * lock. Otherwise, it doesn't. > - */ > - spin_lock_irqsave(&timer_head->lock, timer_head->flags); > - while (!dapl_llist_is_empty(&timer_head->timer_list_head)) { > - list_ptr = (struct dapl_timer_entry *) > - dapl_llist_peek_head(&g_dapl_timer_head. > - timer_list_head); > - dapl_os_get_time(&cur_time); > - > - if (list_ptr->expires <= cur_time) { > - /* > - * Remove the entry from the list. Sort out how much > - * time we need to sleep for the next one > - */ > - list_ptr = > - dapl_llist_remove_head(&timer_head-> > - timer_list_head); > - spin_unlock_irqrestore(&timer_head->lock, > - timer_head->flags); > - > - /* > - * Invoke the user callback > - */ > - list_ptr->function(list_ptr->data); > - /* timer structure was allocated by caller, we don't > - * free it here. > - */ > - > - /* reacquire the lock */ > - spin_lock_irqsave(&timer_head->lock, > - timer_head->flags); > - } else { > - spin_unlock_irqrestore(&timer_head->lock, > - timer_head->flags); > - dat_status = > - dapl_os_wait_object_wait(&timer_head-> > - wait_object, > - (DAT_TIMEOUT) > - (list_ptr-> > - expires - > - cur_time)); > - spin_lock_irqsave(&timer_head->lock, > - timer_head->flags); > - } > - } > - /* > - * release the lock before going back to the top to sleep > - */ > - spin_unlock_irqrestore(&timer_head->lock, timer_head->flags); > - > - if (dat_status == DAT_INTERNAL_ERROR) { > - /* > - * XXX What do we do here? > - */ > - } > - } /* for (;;) */ > - > - return 0; > -} > Index: linux-kernel-timer/dat-provider/dapl_timer_util.h > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_timer_util.h (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_timer_util.h (working copy) > @@ -1,42 +0,0 @@ > -/* > - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > - * > - * This Software is licensed under one of the following licenses: > - * > - * 1) under the terms of the "Common Public License 1.0" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/cpl.php. > - * > - * 2) under the terms of the "The BSD License" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/bsd-license.php. > - * > - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > - * copy of which is available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/gpl-license.php. > - * > - * Licensee has the right to choose one of the above licenses. > - * > - * Redistributions of source code must retain the above copyright > - * notice and one of the license notices. > - * > - * Redistributions in binary form must reproduce both the above copyright > - * notice, one of the license notices in the documentation > - * and/or other materials provided with the distribution. > - */ > - > -/* > - * $Id$ > - */ > - > -#ifndef DAPL_TIMER_UTIL_H > -#define DAPL_TIMER_UTIL_H > - > -void dapl_timer_init(void); > - > -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), > - void *data, DAPL_OS_TIMEVAL expires); > - > -void dapl_timer_cancel(struct dapl_timer_entry *timer); > - > -#endif /* DAPL_TIMER_UTIL_H */ > Index: linux-kernel-timer/dat-provider/dapl_ep_util.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_ep_util.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_ep_util.c (working copy) > @@ -168,9 +168,6 @@ void dapl_ep_dealloc(struct dapl_ep *ep_ > if (ep_ptr->send_iov) > kfree(ep_ptr->send_iov); > > - if (ep_ptr->cxn_timer) > - kfree(ep_ptr->cxn_timer); > - > kfree(ep_ptr); > } > > @@ -242,32 +239,6 @@ bail: > } > > /* > - * dapl_ep_timeout > - * > - * If this routine is invoked before a connection occurs, generate an > - * event > - */ > -void dapl_ep_timeout(void *arg) > -{ > - struct dapl_ep *ep_ptr; > - > - dapl_dbg_log(DAPL_DBG_TYPE_CM, "--> dapl_ep_timeout! ep %lx\n", arg); > - > - ep_ptr = (struct dapl_ep *)arg; > - > - /* reset the EP state */ > - ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; > - > - /* Clean up the EP and put the underlying QP into the ERROR state. */ > - dapl_ib_disconnect_clean(ep_ptr, TRUE); > - > - (void)dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. > - connect_evd_handle, > - DAT_CONNECTION_EVENT_TIMED_OUT, > - (DAT_HANDLE) ep_ptr, 0, NULL); > -} > - > -/* > * dapl_ep_state_subtype > * > * Return the INVALID_STATE connection subtype associated with an > Index: linux-kernel-timer/dat-provider/dapl_ep_util.h > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_ep_util.h (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_ep_util.h (working copy) > @@ -52,8 +52,6 @@ extern u32 dapl_ep_post_send_req(DAT_EP_ > DAPL_DTO_TYPE dto_type, > enum ib_wr_opcode op_type); > > -void dapl_ep_timeout(void *arg); > - > enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr); > > #endif /* DAPL_EP_UTIL_H */ > Index: linux-kernel-timer/dat-provider/dapl_util.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_util.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_util.c (working copy) > @@ -33,104 +33,6 @@ > #include "dapl_provider.h" > #include "dapl_util.h" > > -/* > - * dapl_os_get_time > - * > - * Return 64 bit value of current time in microseconds. > - * > - * Input: > - * loc User location to place current time > - * > - * Returns: > - * DAT_SUCCESS > - */ > - > -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *loc) > -{ > - struct timeval tv; > - > - do_gettimeofday(&tv); > - *loc = ((u64) (tv.tv_sec) * 1000000L) + (u64) tv.tv_usec; > - > - return DAT_SUCCESS; > -} > - > -/* > - * Wait object routines > - */ > - > -/* > - * dapl_os_wait_object_init > - * > - * Initialize a wait object > - * > - * Input: > - * wait_obj > - * > - * Returns: > - * DAT_SUCCESS > - * DAT_INTERNAL_ERROR > - */ > -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj) > -{ > - init_waitqueue_head(&wait_obj->wait_queue); > - > - return DAT_SUCCESS; > -} > - > -/* Wait on the supplied wait object, up to the specified time_out. > - * A timeout of DAT_TIMEOUT_INFINITE will wait indefinitely. > - * Timeout should be specified in micro seconds. > - * > - * Functional returns: > - * DAT_SUCCESS -- another thread invoked dapl_os_wait object_wakeup > - * DAT_INVALID_STATE -- someone else is already waiting in this wait > - * object. > - * only one waiter is allowed at a time. > - * DAT_ABORT -- another thread invoked dapl_os_wait_object_destroy > - * DAT_TIMEOUT -- the specified time limit was reached. > - */ > - > -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, > - DAT_TIMEOUT timeout_val) > -{ > - u32 dat_status = DAT_SUCCESS; > - > - if (DAT_TIMEOUT_INFINITE == timeout_val) { > - interruptible_sleep_on(&wait_obj->wait_queue); > - } else { > - int expire; > - > - expire = timeout_val * HZ / 1000000; > - > - while (expire) { > - current->state = TASK_INTERRUPTIBLE; > - expire = schedule_timeout(expire); > - } > - dat_status = DAT_ERROR(DAT_TIMEOUT_EXPIRED, 0); > - } > - return dat_status; > -} > - > -/* > - * dapl_os_wait_object_wakeup > - * > - * Wakeup a thread waiting on a wait object > - * > - * Input: > - * wait_obj > - * > - * Returns: > - * DAT_SUCCESS > - * DAT_INTERNAL_ERROR > - */ > -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj) > -{ > - wake_up_interruptible(&wait_obj->wait_queue); > - > - return DAT_SUCCESS; > -} > - > #ifdef DAPL_DBG > > void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) > Index: linux-kernel-timer/dat-provider/dapl_util.h > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_util.h (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_util.h (working copy) > @@ -107,24 +107,6 @@ static inline int dapl_os_atomic_assign( > */ > > /* > - * The wait object invariant: Presuming a call to dapl_os_wait_object_wait > - * occurs at some point, there will be at least one wakeup after each call > - * to dapl_os_wait_object_signal. I.e. Signals are not ignored, though > - * they may be coallesced. > - */ > -typedef struct dapl_os_wait_object { > - wait_queue_head_t wait_queue; > -} DAPL_OS_WAIT_OBJECT; > - > -/* function prototypes */ > -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj); > - > -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, > - DAT_TIMEOUT timeout_val); > - > -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj); > - > -/* > * Memory Functions > */ > > @@ -154,15 +136,6 @@ static inline char *dapl_os_strdup(const > } > > /* > - * Timer Functions > - */ > - > -typedef u64 DAPL_OS_TIMEVAL; > - > -/* timer function prototype */ > -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *); > - > -/* > * *printf format helper. We use the C string constant concatenation > * ability to define 64 bit formats, which unfortunatly are non standard > * in the C compiler world. > Index: linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c > =================================================================== > --- linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (working copy) > @@ -32,7 +32,6 @@ > #include "dapl.h" > #include "dapl_ep_util.h" > #include "dapl_openib_util.h" > -#include "dapl_timer_util.h" > > /* > * dapl_ep_dup_connect > @@ -63,7 +62,7 @@ > * DAT_MODEL_NOT_SUPPORTED > */ > u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, > - DAT_TIMEOUT timeout, int private_data_size, > + unsigned long timeout, int private_data_size, > const void *private_data, enum dat_qos qos) > { > struct dapl_ep *ep_dup_ptr; > @@ -114,6 +113,6 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep > private_data_size, > private_data, > qos, DAT_CONNECT_DEFAULT_FLAG); > - bail: > +bail: > return dat_status; > } > Index: linux-kernel-timer/dat-provider/dapl.h > =================================================================== > --- linux-kernel-timer/dat-provider/dapl.h (revision 2518) > +++ linux-kernel-timer/dat-provider/dapl.h (working copy) > @@ -236,14 +236,6 @@ struct dapl_evd { > struct dapl_cno *cno_ptr; > }; > > -/* DAPL timer entry, used to queue timeouts */ > -struct dapl_timer_entry { > - struct dapl_llist_entry list_entry; /* link entry on ia struct */ > - DAPL_OS_TIMEVAL expires; > - void (*function) (void *); > - void *data; > -}; > - > /* struct dapl_ep maps to DAT_EP_HANDLE */ > struct dapl_ep { > struct dapl_header header; > @@ -264,8 +256,7 @@ struct dapl_ep { > /* For passive connections we maintain a back pointer to the CR */ > void *cr_ptr; > > - /* pointer to connection timer, if set */ > - struct dapl_timer_entry *cxn_timer; > + unsigned long timeout; > > /* private data container */ > char private_data[DAPL_MAX_PRIVATE_DATA_SIZE]; > @@ -500,14 +491,15 @@ extern u32 dapl_ep_modify(DAT_EP_HANDLE > extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, > struct sockaddr *remote_ia_address, > DAT_CONN_QUAL remote_conn_qual, > - DAT_TIMEOUT timeout, > + unsigned long timeout, > int private_data_size, > const void *private_data, > enum dat_qos quality_of_service, > enum dat_connect_flags connect_flags); > > extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, > - DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, > + DAT_EP_HANDLE ep_dup_handle, > + unsigned long timeout, > int private_data_size, > const void *private_data, > enum dat_qos quality_of_service); > Index: linux-kernel-timer/patches/alt_dat_provider_makefile > =================================================================== > --- linux-kernel-timer/patches/alt_dat_provider_makefile (revision 2518) > +++ linux-kernel-timer/patches/alt_dat_provider_makefile (working copy) > @@ -71,7 +71,6 @@ PROVIDER_MODULES := \ > dapl_set_consumer_context \ > dapl_sp_util \ > dapl_srq \ > - dapl_timer_util \ > dapl_util > > PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) > @@ -97,4 +96,4 @@ default: > endif > > clean: > - rm -f *.o *.ko > + rm -rf *.o *.ko .*.cmd *.mod.* .tmp_versions .*.d .*.o.tmp cscope.out > Index: linux-kernel-timer/dat/dat.h > =================================================================== > --- linux-kernel-timer/dat/dat.h (revision 2518) > +++ linux-kernel-timer/dat/dat.h (working copy) > @@ -383,11 +383,6 @@ enum dat_completion_flags { > DAT_COMPLETION_BARRIER_FENCE_FLAG = 0x08, > }; > > -typedef u32 DAT_TIMEOUT; /* microseconds */ > - > -/* timeout = infinity */ > -#define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) > - > /* dat handles */ > typedef void *DAT_HANDLE; > typedef DAT_HANDLE DAT_CR_HANDLE; > @@ -1139,12 +1134,12 @@ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP > const struct dat_ep_param *); > > typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, > - DAT_CONN_QUAL, DAT_TIMEOUT, int, > + DAT_CONN_QUAL, unsigned long timeout, int, > const void *, enum dat_qos, > enum dat_connect_flags); > > typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, > - DAT_TIMEOUT, int, const void *, > + unsigned long timeout, int, const void *, > enum dat_qos); > > typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); > @@ -1493,7 +1488,7 @@ static inline u32 dat_evd_resize(DAT_EVD > } > > static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, > - DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, > + DAT_CONN_QUAL conn_qual, unsigned long timeout, > int private_data_size, > const void *private_data, enum dat_qos qos, > enum dat_connect_flags flags) > @@ -1533,7 +1528,8 @@ static inline u32 dat_ep_disconnect(DAT_ > } > > static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, > - DAT_TIMEOUT timeout, int private_data_size, > + unsigned long timeout, > + int private_data_size, > const void *private_data, enum dat_qos qos) > { > return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, > -------------- next part -------------- Index: test/dapltest/test/dapl_performance_util.c =================================================================== --- test/dapltest/test/dapl_performance_util.c (revision 2525) +++ test/dapltest/test/dapl_performance_util.c (working copy) @@ -538,7 +538,7 @@ DT_performance_wait ( pre_ts = DT_Mdep_GetTimeStamp (); ret = DT_Tdep_evd_wait ( evd_handle, - DAT_TIMEOUT_INFINITE, + DAT_TIMEOUT_MAX, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: test/dapltest/test/dapl_performance_client.c =================================================================== --- test/dapltest/test/dapl_performance_client.c (revision 2525) +++ test/dapltest/test/dapl_performance_client.c (working copy) @@ -105,7 +105,7 @@ retry: ret = dat_ep_connect (test_ptr->ep_context.ep_handle, test_ptr->remote_ia_addr, test_ptr->ep_context.port, - DAT_TIMEOUT_INFINITE, + DAT_TIMEOUT_MAX, 0, (void *) 0, /* no private data */ test_ptr->cmd->qos, Index: test/dapltest/test/dapl_server.c =================================================================== --- test/dapltest/test/dapl_server.c (revision 2525) +++ test/dapltest/test/dapl_server.c (working copy) @@ -580,7 +580,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Mdep_wait_object_wait ( &pt_ptr->synch_wait_object, - DAT_TIMEOUT_INFINITE); + DAT_TIMEOUT_MAX); /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); Index: test/dapltest/test/dapl_transaction_test.c =================================================================== --- test/dapltest/test/dapl_transaction_test.c (revision 2525) +++ test/dapltest/test/dapl_transaction_test.c (working copy) @@ -710,12 +710,12 @@ DT_Transaction_Main (void *param) retry: ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, - test_ptr->remote_ia_addr, - test_ptr->ep_context[i].ia_port, - DAT_TIMEOUT_INFINITE, + test_ptr->remote_ia_addr, + test_ptr->ep_context[i].ia_port, + DAT_TIMEOUT_MAX, strlen (private_data_str), private_data_str, - pt_ptr->Params.ReliabilityLevel, - DAT_CONNECT_DEFAULT_FLAG); + pt_ptr->Params.ReliabilityLevel, + DAT_CONNECT_DEFAULT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_connect #%d error: %s (0x%x)\n", Index: test/dapltest/test/dapl_test_util.c =================================================================== --- test/dapltest/test/dapl_test_util.c (revision 2525) +++ test/dapltest/test/dapl_test_util.c (working copy) @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -369,7 +369,7 @@ DT_disco_event_wait ( DT_Tdep_Print_Head u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", Index: test/dapltest/test/dapl_client.c =================================================================== --- test/dapltest/test/dapl_client.c (revision 2525) +++ test/dapltest/test/dapl_client.c (working copy) @@ -256,7 +256,7 @@ retry: ret = dat_ep_connect (ep_handle, server_netaddr, SERVER_PORT_NUMBER, - DAT_TIMEOUT_INFINITE, + DAT_TIMEOUT_MAX, 0, (void *) 0, /* no private data */ params_ptr->ReliabilityLevel, DAT_CONNECT_DEFAULT_FLAG); Index: test/dapltest/include/dapl_tdep.h =================================================================== --- test/dapltest/include/dapl_tdep.h (revision 2525) +++ test/dapltest/include/dapl_tdep.h (working copy) @@ -56,7 +56,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_ha u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, - DAT_TIMEOUT timeout, + unsigned long timeout, struct dat_event *event); u32 DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, Index: test/dapltest/include/dapl_proto.h =================================================================== --- test/dapltest/include/dapl_proto.h (revision 2525) +++ test/dapltest/include/dapl_proto.h (working copy) @@ -185,7 +185,7 @@ void DT_Mdep_Thread_EXIT ( void * thr int DT_Mdep_wait_object_init ( DT_WAIT_OBJECT *wait_obj); int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val); + unsigned long timeout_val); int DT_Mdep_wait_object_wakeup ( DT_WAIT_OBJECT *wait_obj); int DT_Mdep_wait_object_destroy ( DT_WAIT_OBJECT *wait_obj); Index: test/dapltest/include/dapl_transaction_test.h =================================================================== --- test/dapltest/include/dapl_transaction_test.h (revision 2525) +++ test/dapltest/include/dapl_transaction_test.h (working copy) @@ -79,7 +79,7 @@ typedef struct Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; - DAT_TIMEOUT time_out; + unsigned long time_out; int evd_length; Thread *thread; Index: test/dapltest/mdep/linux/dapl_mdep_user.c =================================================================== --- test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2525) +++ test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) @@ -411,7 +411,7 @@ DT_Mdep_wait_object_init ( int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val) + unsigned long timeout_val) { int dat_status; int pthread_status; Index: test/dapltest/mdep/linux/dapl_mdep_kernel.c =================================================================== --- test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2525) +++ test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) @@ -252,13 +252,13 @@ DT_Mdep_wait_object_init ( int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val) + unsigned long timeout_val) { int expire; int dat_status; dat_status = DAT_SUCCESS; - if ( DAT_TIMEOUT_INFINITE == timeout_val) + if ( DAT_TIMEOUT_MAX == timeout_val) { interruptible_sleep_on (wait_obj); } Index: test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2525) +++ test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -213,7 +213,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, - DAT_TIMEOUT timeout, + unsigned long timeout, struct dat_event *dat_event) { u32 dat_status; @@ -239,7 +239,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han return dat_status; } /* wait for an event */ - if (timeout == DAT_TIMEOUT_INFINITE) + if (timeout == DAT_TIMEOUT_MAX) { rc = wait_event_interruptible (evd_ptr->wait_object, evd_ptr->event_next != NULL); Index: dat-provider/dapl_ep_connect.c =================================================================== --- dat-provider/dapl_ep_connect.c (revision 2525) +++ dat-provider/dapl_ep_connect.c (working copy) @@ -35,7 +35,6 @@ #include "dapl_ep_util.h" #include "dapl_openib_util.h" #include "dapl_evd_util.h" -#include "dapl_timer_util.h" /* * Request a connection be established between the local Endpoint @@ -44,7 +43,7 @@ */ u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, + DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags connect_flags) { @@ -52,15 +51,14 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, - "dapl_ep_connect (%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", + "dapl_ep_connect(%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", ep_handle, remote_ia_address->sa_data[2], remote_ia_address->sa_data[3], remote_ia_address->sa_data[4], remote_ia_address->sa_data[5], - remote_conn_qual, - timeout, - private_data_size, private_data, qos, connect_flags); + remote_conn_qual, timeout, private_data_size, private_data, + qos, connect_flags); ep_ptr = (struct dapl_ep *)ep_handle; @@ -90,7 +88,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han goto bail; } /* - * If the endpoint needs a QP, associated the QP with it. + * If the endpoint needs a QP, associate the QP with it. * This needs to be done carefully, in order to: * * Avoid allocating under a lock. * * Not step on data structures being altered by @@ -242,11 +240,9 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->param.remote_port_qual = remote_conn_qual; memcpy(ep_ptr->private_data,private_data,private_data_size); - dat_status = dapl_ib_connect(ep_handle, - remote_ia_address, - remote_conn_qual, - private_data_size, - ep_ptr->private_data); + dat_status = dapl_ib_connect(ep_ptr, remote_ia_address, + remote_conn_qual, timeout, + private_data_size, ep_ptr->private_data); if (dat_status != DAT_SUCCESS) { ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; @@ -267,33 +263,9 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han NULL); dat_status = DAT_SUCCESS; } - } else { - /* - * Acquire the lock and recheck the state of the EP; this - * thread could have been descheduled after issuing the connect - * request and the EP is now connected. Set up a timer if - * necessary. - */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->param.ep_state == - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - && timeout != DAT_TIMEOUT_INFINITE) { - ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, - GFP_ATOMIC); - if (!ep_ptr->cxn_timer) { - dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - - dapl_timer_set(ep_ptr->cxn_timer, - dapl_ep_timeout, ep_ptr, timeout); - } - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); - } + } - bail: +bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, "dapl_ep_connect () returns 0x%x\n", dat_status); Index: dat-provider/dapl_evd_connection_callb.c =================================================================== --- dat-provider/dapl_evd_connection_callb.c (revision 2527) +++ dat-provider/dapl_evd_connection_callb.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_evd_util.h" #include "dapl_ep_util.h" -#include "dapl_timer_util.h" /* * Connection callback function for ACTIVE connection requests; callbacks @@ -63,15 +62,6 @@ void dapl_evd_connection_callback(struct */ spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - /* - * If a connection timer has been set up on this EP, cancel it now - */ - if (ep_ptr->cxn_timer != NULL) { - dapl_timer_cancel(ep_ptr->cxn_timer); - kfree(ep_ptr->cxn_timer); - ep_ptr->cxn_timer = NULL; - } - switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: { Index: dat-provider/Makefile =================================================================== --- dat-provider/Makefile (revision 2525) +++ dat-provider/Makefile (working copy) @@ -77,7 +77,6 @@ PROVIDER_MODULES := \ dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ - dapl_timer_util \ dapl_util PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) Index: dat-provider/dapl_ep_free.c =================================================================== --- dat-provider/dapl_ep_free.c (revision 2525) +++ dat-provider/dapl_ep_free.c (working copy) @@ -34,7 +34,6 @@ #include "dapl_ep_util.h" #include "dapl_openib_util.h" #include "dapl_ring_buffer_util.h" -#include "dapl_timer_util.h" /* * dapl_ep_free @@ -107,12 +106,6 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->cxn_timer != NULL) { - dapl_timer_cancel(ep_ptr->cxn_timer); - kfree(ep_ptr->cxn_timer); - ep_ptr->cxn_timer = NULL; - } - /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); Index: dat-provider/dapl_provider.c =================================================================== --- dat-provider/dapl_provider.c (revision 2525) +++ dat-provider/dapl_provider.c (working copy) @@ -39,7 +39,6 @@ #include "dapl_mr_util.h" #include "dapl_util.h" #include "dapl_openib_util.h" -#include "dapl_timer_util.h" MODULE_LICENSE("Dual BSD/GPL"); MODULE_DESCRIPTION("DAT Provider for InfiniBand"); @@ -393,7 +392,6 @@ module_exit(dapl_fini); static int __init dapl_init(void) { dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "provider started\n"); - dapl_timer_init(); return ib_register_client(&dapl_client); } Index: dat-provider/dapl_openib_cm.c =================================================================== --- dat-provider/dapl_openib_cm.c (revision 2525) +++ dat-provider/dapl_openib_cm.c (working copy) @@ -38,7 +38,7 @@ #include "dapl_evd_util.h" #include "dapl_cr_util.h" -#define DAPL_IB_RC_RETRY_COUNT 7 +#define DAPL_IB_RC_RETRY_COUNT 0 #define DAPL_IB_RNR_RETRY_COUNT 6 #define DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ #define DAPL_IB_MAX_CM_RETRIES 4 @@ -291,6 +291,26 @@ static int dapl_cm_passive_cb_handler(st return ret; } +/* + * approximately transforms microseconds to 4.096us*2^x + * 63(+8) is max return + */ +static inline u8 dapl_convert_us_to_kookyib(unsigned long us) { + unsigned long ms = us/1000UL, converged = 2; + u8 i; + + if (2 > ms) + return 8; + + for (i = 1; i < 63; i++) { + if (converged >= ms) + break; + converged = 2*converged; + } + + return i+8; +} + static void dapl_path_comp_handler(u64 req_id, void *context, int rec_num) { struct dapl_cm_id *conn = context; @@ -410,7 +430,7 @@ error: */ u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, + DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data) { struct dapl_ia *ia_ptr; @@ -453,7 +473,8 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han conn->param.private_data_len = private_data_size; conn->param.responder_resources = DAPL_IB_TARGET_MAX; conn->param.initiator_depth = DAPL_IB_INITIATOR_DEPTH; - conn->param.remote_cm_response_timeout = DAPL_IB_CM_RESPONSE_TIMEOUT; + conn->param.remote_cm_response_timeout = + dapl_convert_us_to_kookyib(timeout); conn->param.retry_count = DAPL_IB_RC_RETRY_COUNT; conn->param.rnr_retry_count = DAPL_IB_RNR_RETRY_COUNT; conn->param.local_cm_response_timeout = DAPL_IB_CM_RESPONSE_TIMEOUT; Index: dat-provider/dapl_openib_util.h =================================================================== --- dat-provider/dapl_openib_util.h (revision 2525) +++ dat-provider/dapl_openib_util.h (working copy) @@ -61,7 +61,7 @@ u32 dapl_ib_qp_modify(struct dapl_ia *ia u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, + DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data); u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags); Index: dat-provider/dapl_ep_util.c =================================================================== --- dat-provider/dapl_ep_util.c (revision 2525) +++ dat-provider/dapl_ep_util.c (working copy) @@ -168,9 +168,6 @@ void dapl_ep_dealloc(struct dapl_ep *ep_ if (ep_ptr->send_iov) kfree(ep_ptr->send_iov); - if (ep_ptr->cxn_timer) - kfree(ep_ptr->cxn_timer); - kfree(ep_ptr); } @@ -242,32 +239,6 @@ bail: } /* - * dapl_ep_timeout - * - * If this routine is invoked before a connection occurs, generate an - * event - */ -void dapl_ep_timeout(void *arg) -{ - struct dapl_ep *ep_ptr; - - dapl_dbg_log(DAPL_DBG_TYPE_CM, "--> dapl_ep_timeout! ep %lx\n", arg); - - ep_ptr = (struct dapl_ep *)arg; - - /* reset the EP state */ - ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - - /* Clean up the EP and put the underlying QP into the ERROR state. */ - dapl_ib_disconnect_clean(ep_ptr, TRUE); - - (void)dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. - connect_evd_handle, - DAT_CONNECTION_EVENT_TIMED_OUT, - (DAT_HANDLE) ep_ptr, 0, NULL); -} - -/* * dapl_ep_state_subtype * * Return the INVALID_STATE connection subtype associated with an Index: dat-provider/dapl_ep_util.h =================================================================== --- dat-provider/dapl_ep_util.h (revision 2525) +++ dat-provider/dapl_ep_util.h (working copy) @@ -52,8 +52,6 @@ extern u32 dapl_ep_post_send_req(DAT_EP_ DAPL_DTO_TYPE dto_type, enum ib_wr_opcode op_type); -void dapl_ep_timeout(void *arg); - enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr); #endif /* DAPL_EP_UTIL_H */ Index: dat-provider/dapl_util.c =================================================================== --- dat-provider/dapl_util.c (revision 2525) +++ dat-provider/dapl_util.c (working copy) @@ -33,104 +33,6 @@ #include "dapl_provider.h" #include "dapl_util.h" -/* - * dapl_os_get_time - * - * Return 64 bit value of current time in microseconds. - * - * Input: - * loc User location to place current time - * - * Returns: - * DAT_SUCCESS - */ - -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *loc) -{ - struct timeval tv; - - do_gettimeofday(&tv); - *loc = ((u64) (tv.tv_sec) * 1000000L) + (u64) tv.tv_usec; - - return DAT_SUCCESS; -} - -/* - * Wait object routines - */ - -/* - * dapl_os_wait_object_init - * - * Initialize a wait object - * - * Input: - * wait_obj - * - * Returns: - * DAT_SUCCESS - * DAT_INTERNAL_ERROR - */ -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj) -{ - init_waitqueue_head(&wait_obj->wait_queue); - - return DAT_SUCCESS; -} - -/* Wait on the supplied wait object, up to the specified time_out. - * A timeout of DAT_TIMEOUT_INFINITE will wait indefinitely. - * Timeout should be specified in micro seconds. - * - * Functional returns: - * DAT_SUCCESS -- another thread invoked dapl_os_wait object_wakeup - * DAT_INVALID_STATE -- someone else is already waiting in this wait - * object. - * only one waiter is allowed at a time. - * DAT_ABORT -- another thread invoked dapl_os_wait_object_destroy - * DAT_TIMEOUT -- the specified time limit was reached. - */ - -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, - DAT_TIMEOUT timeout_val) -{ - u32 dat_status = DAT_SUCCESS; - - if (DAT_TIMEOUT_INFINITE == timeout_val) { - interruptible_sleep_on(&wait_obj->wait_queue); - } else { - int expire; - - expire = timeout_val * HZ / 1000000; - - while (expire) { - current->state = TASK_INTERRUPTIBLE; - expire = schedule_timeout(expire); - } - dat_status = DAT_ERROR(DAT_TIMEOUT_EXPIRED, 0); - } - return dat_status; -} - -/* - * dapl_os_wait_object_wakeup - * - * Wakeup a thread waiting on a wait object - * - * Input: - * wait_obj - * - * Returns: - * DAT_SUCCESS - * DAT_INTERNAL_ERROR - */ -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj) -{ - wake_up_interruptible(&wait_obj->wait_queue); - - return DAT_SUCCESS; -} - #ifdef DAPL_DBG void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) Index: dat-provider/dapl_util.h =================================================================== --- dat-provider/dapl_util.h (revision 2525) +++ dat-provider/dapl_util.h (working copy) @@ -107,24 +107,6 @@ static inline int dapl_os_atomic_assign( */ /* - * The wait object invariant: Presuming a call to dapl_os_wait_object_wait - * occurs at some point, there will be at least one wakeup after each call - * to dapl_os_wait_object_signal. I.e. Signals are not ignored, though - * they may be coallesced. - */ -typedef struct dapl_os_wait_object { - wait_queue_head_t wait_queue; -} DAPL_OS_WAIT_OBJECT; - -/* function prototypes */ -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj); - -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, - DAT_TIMEOUT timeout_val); - -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj); - -/* * Memory Functions */ @@ -154,15 +136,6 @@ static inline char *dapl_os_strdup(const } /* - * Timer Functions - */ - -typedef u64 DAPL_OS_TIMEVAL; - -/* timer function prototype */ -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *); - -/* * *printf format helper. We use the C string constant concatenation * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. Index: dat-provider/dapl_ep_dup_connect.c =================================================================== --- dat-provider/dapl_ep_dup_connect.c (revision 2525) +++ dat-provider/dapl_ep_dup_connect.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_ep_util.h" #include "dapl_openib_util.h" -#include "dapl_timer_util.h" /* * dapl_ep_dup_connect @@ -63,7 +62,7 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, - DAT_TIMEOUT timeout, int private_data_size, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { struct dapl_ep *ep_dup_ptr; @@ -114,6 +113,6 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep private_data_size, private_data, qos, DAT_CONNECT_DEFAULT_FLAG); - bail: +bail: return dat_status; } Index: dat-provider/dapl.h =================================================================== --- dat-provider/dapl.h (revision 2525) +++ dat-provider/dapl.h (working copy) @@ -236,14 +236,6 @@ struct dapl_evd { struct dapl_cno *cno_ptr; }; -/* DAPL timer entry, used to queue timeouts */ -struct dapl_timer_entry { - struct dapl_llist_entry list_entry; /* link entry on ia struct */ - DAPL_OS_TIMEVAL expires; - void (*function) (void *); - void *data; -}; - /* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { struct dapl_header header; @@ -264,9 +256,6 @@ struct dapl_ep { /* For passive connections we maintain a back pointer to the CR */ void *cr_ptr; - /* pointer to connection timer, if set */ - struct dapl_timer_entry *cxn_timer; - /* private data container */ char private_data[DAPL_MAX_PRIVATE_DATA_SIZE]; @@ -500,14 +489,15 @@ extern u32 dapl_ep_modify(DAT_EP_HANDLE extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, - DAT_TIMEOUT timeout, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, + DAT_EP_HANDLE ep_dup_handle, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); Index: patches/alt_dat_provider_makefile =================================================================== --- patches/alt_dat_provider_makefile (revision 2525) +++ patches/alt_dat_provider_makefile (working copy) @@ -71,7 +71,6 @@ PROVIDER_MODULES := \ dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ - dapl_timer_util \ dapl_util PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) @@ -97,4 +96,4 @@ default: endif clean: - rm -f *.o *.ko + rm -rf *.o *.ko .*.cmd *.mod.* .tmp_versions .*.d .*.o.tmp cscope.out Index: dat/dat.h =================================================================== --- dat/dat.h (revision 2525) +++ dat/dat.h (working copy) @@ -383,10 +383,7 @@ enum dat_completion_flags { DAT_COMPLETION_BARRIER_FENCE_FLAG = 0x08, }; -typedef u32 DAT_TIMEOUT; /* microseconds */ - -/* timeout = infinity */ -#define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) +#define DAT_TIMEOUT_MAX ULONG_MAX /* dat handles */ typedef void *DAT_HANDLE; @@ -1139,12 +1136,12 @@ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP const struct dat_ep_param *); typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, - DAT_CONN_QUAL, DAT_TIMEOUT, int, + DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, - DAT_TIMEOUT, int, const void *, + unsigned long timeout, int, const void *, enum dat_qos); typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); @@ -1493,7 +1490,7 @@ static inline u32 dat_evd_resize(DAT_EVD } static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, - DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, + DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags flags) @@ -1533,7 +1530,8 @@ static inline u32 dat_ep_disconnect(DAT_ } static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, - DAT_TIMEOUT timeout, int private_data_size, + unsigned long timeout, + int private_data_size, const void *private_data, enum dat_qos qos) { return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, From tduffy at sun.com Thu Jun 2 13:34:06 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 02 Jun 2005 13:34:06 -0700 Subject: [openib-general] Re: [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117655324.19908.3.camel@duffman> Message-ID: <1117744446.4119.3.camel@duffman> On Thu, 2005-06-02 at 16:18 -0400, James Lentini wrote: > I'd recommend these changes (see attached): > > - use the new microsecond conversion function you sent great > - keep a default DAT timeout constant (I propose DAT_TIMEOUT_MAX) fair enough. > - set the timeout a little differently. Instead of storing it in the > ep, store it directly in the CM param that is better > - add a kdat.h file to the patches directory. We can start using this > to place symbols that are in the official DAT spec (like > DAT_TIMEOUT_INFINITE), but not our linux variant. alright, but this could get ugly. > Did I call dapl_convert_us_to_kookyib() correctly? You were passing a > value divided by 1000 to dapl_convert_ms_to_kookyib(), so I assumed > that I could get rid of that. Yup. > What do you think? Looks good, James. I didn't test it, but it looks good. I'd say go for it. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Thu Jun 2 14:07:32 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 17:07:32 -0400 (EDT) Subject: [openib-general] Re: [PATCHv3][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117744446.4119.3.camel@duffman> References: <1117655324.19908.3.camel@duffman> <1117744446.4119.3.camel@duffman> Message-ID: On Thu, 2 Jun 2005, Tom Duffy wrote: > On Thu, 2005-06-02 at 16:18 -0400, James Lentini wrote: >> I'd recommend these changes (see attached): >> >> - use the new microsecond conversion function you sent > > great > >> - keep a default DAT timeout constant (I propose DAT_TIMEOUT_MAX) > > fair enough. > >> - set the timeout a little differently. Instead of storing it in the >> ep, store it directly in the CM param > > that is better > >> - add a kdat.h file to the patches directory. We can start using this >> to place symbols that are in the official DAT spec (like >> DAT_TIMEOUT_INFINITE), but not our linux variant. > > alright, but this could get ugly. I expect that it will. This kdat.h file will not be something we would submit for inclusion in the Linux kernel. It's the best way I've come up with to support applications written against the official DAT headers. >> Did I call dapl_convert_us_to_kookyib() correctly? You were passing a >> value divided by 1000 to dapl_convert_ms_to_kookyib(), so I assumed >> that I could get rid of that. > > Yup. > >> What do you think? > > Looks good, James. I didn't test it, but it looks good. I'd say go for > it. I tested it with dapltest and it worked. (i.e. it doesn't break things that used to work). I'll go ahead and commit it. Thanks for working on this Tom. > > -tduffy > From bjordan.ics at gmail.com Thu Jun 2 14:31:39 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Thu, 2 Jun 2005 17:31:39 -0400 Subject: [openib-general] [PATCH]libibcm: updated README for device node configuration In-Reply-To: <200506021957.j52Jv6Q19238@sbc2.infiniconsys.com> References: <200506021957.j52Jv6Q19238@sbc2.infiniconsys.com> Message-ID: <78d18e2050602143120e0d76c@mail.gmail.com> Libor, Ignore this patch. I'll submit a different one to change the ib_ucm device node from /dev/infiniband/cm to /dev/infiniband_cm. This makes the udev setup unnecessary. And, this probably wasn't the right README file for the udev rules anyway. They belong with the module, not with the library, as pointed out by Tom. On 6/2/05, Bill Jordan wrote: > Patch to add basic content to the CM library > README file. Add build instructions, install > directory, and device node configuration. > > Signed-off-by: Bill Jordan > > Index: README > =================================================================== > --- README (revision 2523) > +++ README (working copy) > @@ -0,0 +1,27 @@ > +This README is for userspace cm library. > + > +Building > + > +To make this directory, run: > +./autogen.sh && ./configure && make && make install > + > +Typically the autogen and configure steps only need be done the first > +time unless configure.in or Makefile.am changes. > + > +Libraries are installed by default at /usr/local/lib/infiniband. > + > +Device files > + > +The userspace CM uses a single device file regardless of the number > +of adapters or ports present. > + > +To create the appropriate character device file automatically with > +udev, a rule like > + > + KERNEL="ucm", NAME="infiniband/%k", MODE="0666" > + > +can be used. This will create the device node named > + > + /dev/infiniband/ucm > + > + > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > -- Bill Jordan SilverStorm Technologies From jlentini at netapp.com Thu Jun 2 14:50:18 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 2 Jun 2005 17:50:18 -0400 (EDT) Subject: [openib-general] Re: [patch][kdapl] enable kdapltest -T P In-Reply-To: <20050602132843.GA3760@mellanox.co.il> References: <20050602132843.GA3760@mellanox.co.il> Message-ID: A couple of questions below: On Thu, 2 Jun 2005, Itamar wrote: > in order to enable kdapltest -T P i needed to remark attr that are > not set by openib gen2 > 1) ia_attr.max_evd_qlen > 2) ia_attr.max_rdma_read_per_ep_in > 3) ia_attr.max_rdma_read_per_ep_out > > also there was bug in kdapltest memory registration (file dapl_bpool.c) > and there was bug where we free memory and then we try to print part of the memory > > i can run now kdapltest -T P for both RR and RW > > > enable kdapltest -T P > fix bug in memory registration > Signed-off-by: Itamar Rabenstein > > Index: test/dapl_performance_util.c > =================================================================== > --- test/dapl_performance_util.c (revision 2522) > +++ test/dapl_performance_util.c (working copy) > @@ -76,17 +76,20 @@ > } > > pipeline_len = DT_min ( > - DT_min (test_ptr->cmd->num_iterations, > - test_ptr->cmd->pipeline_len), > - DT_min (test_ptr->ia_attr.max_dto_per_ep, > - test_ptr->ia_attr.max_evd_qlen)); > + DT_min (test_ptr->cmd->num_iterations, > + test_ptr->cmd->pipeline_len), > + test_ptr->ia_attr.max_dto_per_ep); > +/* DT_min (test_ptr->ia_attr.max_dto_per_ep, */ > +/* test_ptr->ia_attr.max_evd_qlen) */ > +/* ); */ Could we initialize the ia_attr.max_evd_qlen value correctly in the provider? > > if ( RDMA_READ == test_ptr->cmd->op.transfer_type ) > - { > - pipeline_len = DT_min (pipeline_len, > - test_ptr->ia_attr.max_rdma_read_per_ep_in); > - pipeline_len = DT_min (pipeline_len, > - test_ptr->ia_attr.max_rdma_read_per_ep_out); > + { > +/* max_rdma_read_per_ep_in max_rdma_read_per_ep_out are not set by gen2 */ > +/* pipeline_len = DT_min (pipeline_len, */ > +/* test_ptr->ia_attr.max_rdma_read_per_ep_in); */ > +/* pipeline_len = DT_min (pipeline_len, */ > +/* test_ptr->ia_attr.max_rdma_read_per_ep_out); */ Again, why not initialize these correclty in the provider? > } > > test_ptr->reqt_evd_length = pipeline_len; > @@ -203,7 +206,7 @@ > DAT_HANDLE_NULL, /* rmr */ > DT_PERF_SYNC_BUFF_SIZE, > 2, /* 2 RMIs */ > - 255, /* FIXME should query for this */ > + 256, /* FIXME should query for this */ Good catch. 255 was not a good value to have been using. Could we use pt_ptr->provider_attr.optimal_buffer_alignment instead? > FALSE, > FALSE); > if ( !test_ptr->ep_context.bp ) > @@ -406,8 +409,8 @@ > } > } > > + DT_Tdep_PT_Debug (1, (phead,"Test[" F64x "]: cleanup is done\n", test_ptr->base_port)); > DT_MemListFree (test_ptr->pt_ptr, test_ptr); > - DT_Tdep_PT_Debug (1, (phead,"Test[" F64x "]: cleanup is done\n", test_ptr->base_port)); Good catch. > > return status; > } > Index: test/dapl_bpool.c > =================================================================== > --- test/dapl_bpool.c (revision 2522) > +++ test/dapl_bpool.c (working copy) > @@ -388,7 +388,7 @@ > u64 > DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index) > { > - return ( bpool_ptr->reg_addr + index * bpool_ptr->seg_size ); > + return (virt_to_phys(DT_Bpool_GetBuffer(bpool_ptr,index))); Was this also a problem in the transaction test? > } > > /*****************************************************************************/ > -- > Itamar > From jcarr at linuxmachines.com Thu Jun 2 15:09:03 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 02 Jun 2005 15:09:03 -0700 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: References: Message-ID: <429F837F.8020005@linuxmachines.com> On 05/31/05 16:30, Sean Hefty wrote: >>>>Has anyone seen ib_send_cm_req() return -22? >>> >>>I believe that this is a timeout error, possibly indicating that the >> >>server >> >>>side of the connection wasn't running. You may also want to verify the >> >>slid >> >>>and dlid are correct for your configuration. >> >>Don't you get a REJ now when there is no one listening on a service ID >>requested ? > > > You do if the CM is loaded on the destination. > > >>-22 is EINVAL. In terms of ib_send_cm_req, it is returned for a number >>of cases: >>1. peer to peer connection is requested >>2. No primary path is supplied >>3. QP is not RC or UC >>4. private data is supplied and length > 92 >>5. alternate path supplied and PKEY or MTU does not match primary path >>6. connection state is not IDLE >>7. Primary or alternate path SGID or PKey does not match those of port > > > You're right. I was thinking about the request failing asynchronously, not > synchronously when called. Mostly likely cause is a bad slid/dlid. OK, I'll check next time. I guess I made the bad assumption they didn't change. What is the proper way of retreiving the local lid? I see modify_query_port() has: props->lid = be16_to_cpup((u16 *) (out_mad->data + 16)); Is this correct/ok to get the lid for port 1: u16 mylid; ib_query_port(mydev, 1, &attr); mylid = attr.lid; Is there a simple way to discover the lid values of other systems? Jeff From hch at lst.de Thu Jun 2 15:10:08 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 3 Jun 2005 00:10:08 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> <1117649626.29566.2.camel@duffman> Message-ID: <20050602221008.GA18943@lst.de> On Thu, Jun 02, 2005 at 11:16:53AM -0400, James Lentini wrote: > This could be an improvement. We just need to be careful that we don't > expose anything transport specific. Off the top of my head, I can > think of one way to do this: > > - in dat.h, create a public structure for each object type: > > struct dat_ep { > struct dat_provider *provider; > }; > > - in the transport provider (dapl.h) have a private structure that > contains the public one: > > struct dapl_ep { > struct dat_ep dat_ep; > ... > }; > > Is that the sort of thing you were thinking of? That would follow the way the VFS does for inodes. I just think dapl_* is a really bad name for an IB-specific implementation. In the end when we converge thing more into a common RDMA stack that should probably become rdma_* for the transport-agnostic and ib_* for IB-specific structure. From hch at lst.de Thu Jun 2 15:13:28 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 3 Jun 2005 00:13:28 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1117726034.15225.1.camel@duffman> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> <1117649626.29566.2.camel@duffman> <1117726034.15225.1.camel@duffman> Message-ID: <20050602221328.GB18943@lst.de> On Thu, Jun 02, 2005 at 08:27:14AM -0700, Tom Duffy wrote: > Yes, exactly. Then you can create small inline functions that would > cast back and forth. just use container_of, then it doesn't need to be the first memeber even. From halr at voltaire.com Thu Jun 2 15:27:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 Jun 2005 18:27:51 -0400 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: <429F837F.8020005@linuxmachines.com> References: <429F837F.8020005@linuxmachines.com> Message-ID: <1117751271.4476.178.camel@hal.voltaire.com> On Thu, 2005-06-02 at 18:09, Jeff Carr wrote: > Is there a simple way to discover the lid values of other systems? Locally, you can run /usr/local/ib/bin/ibstatus or ibstat. Remotely, you would need to know the remote GID (subnet prefix + GUID) and ask the SA what the LID was for that (by getting a path record). -- Hal From robert.j.woodruff at intel.com Thu Jun 2 15:40:36 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Thu, 2 Jun 2005 15:40:36 -0700 Subject: [openib-general] RE: [ANNOUNCE][PATCH] New Linux 2.6.9 backportpatches and corresponding userspace tar ball available Message-ID: <1AC79F16F5C5284499BB9591B33D6F00049293E1@orsmsx408> Michael Wrote > >> >> Patches are located in the SVN tree under >> gen2/trunk/src/linux-kernel/patches/backport-to-2.6.9/ >> >> infiniband-backport-svn2425-to-2.6.9-kernel-fixups-01.diff >> infiniband-backport-svn2425-to-2.6.9-openib-drivers-02.diff >> infiniband-backport-svn2425-to-2.6.9-openib-fixups-03.diff >> infiniband-backport-svn2425-userspace.tar.gz >> >> woody >> >Woody, could you please move these patches to gen2/branches? >MST - Michael S. Tsirkin Yes. updated in svn2531. Backport patches now in gen2/branches/backport-to-2.6.9 I also added new backport patches for svn2515. woody From mshefty at ichips.intel.com Thu Jun 2 15:53:44 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 02 Jun 2005 15:53:44 -0700 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: <429F837F.8020005@linuxmachines.com> References: <429F837F.8020005@linuxmachines.com> Message-ID: <429F8DF8.3050806@ichips.intel.com> Jeff Carr wrote: > Is there a simple way to discover the lid values of other systems? Simple? Not really. You could query the SA to obtain a list of path records to all systems, and then extract the LIDs from those. I use: cat /sys/class/infiniband/mthca0/ports/1/lid to get the LIDs of both systems and then provide them manually into my test programs. Not necessarily what you want for a real app though. And, yes, it would be really nice if the SM reassigned the same LIDs to the same nodes where possible. - Sean From hozer at hozed.org Thu Jun 2 16:23:31 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Thu, 2 Jun 2005 18:23:31 -0500 Subject: [openib-general] opensm fails to bring up subnet.. Message-ID: <20050602232331.GD577@kalmia.hozed.org> I'm having intermittent problems with opensm.. It seems after a while IPoIB stops working and if I restart opensm, it starts spitting out errors. Do I have a misbehaving switch somewhere? ibnetdiscover seems to work fine. (this is from running 'opensm -v -o -r') -------------- next part -------------- ------------------------------------------------- OpenSM Rev:openib-1.0.0 Command Line Arguments: Verbose option -v (log flags = 0x7) Run Once Reassign LIDs ------------------------------------------------- Jun 02 18:14:35 [B7E7F600] -> OpenSM Rev:openib-1.0.0 Jun 02 18:14:35 [B7E7F600] -> osm_opensm_init: Forcing single threaded dispatcher. Jun 02 18:14:35 [B7E7F600] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0xfe80000000000000,0x0000000000000000 Jun 02 18:14:35 [B7E7F600] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0xfe80000000000000,0x0000000000000000 Jun 02 18:14:35 [B7E7F600] -> osm_vendor_get_all_port_attr: assign CA mthca0 port 1 guid (0x66a00a000044d) as the default port. using default guid 0x66a00a000044d Jun 02 18:14:35 [B7E7F600] -> osm_sm_mad_ctrl_bind: Binding to port 0x66a00a000044d. Jun 02 18:14:35 [B7E7F600] -> osm_vendor_bind: Binding to port 0x66a00a000044d. Jun 02 18:14:35 [B7E7F600] -> osm_sa_mad_ctrl_bind: Binding to port GUID = 0x66a00a000044d. Jun 02 18:14:35 [B7E7F600] -> osm_vendor_bind: Binding to port 0x66a00a000044d. Jun 02 18:14:35 [B7E7F600] -> __osm_sm_state_mgr_discovering_msg: ****************************************************************** ******************** ENTERING SM DISCOVERING STATE *************** ****************************************************************** Jun 02 18:14:35 [B7E7F600] -> __osm_state_mgr_sweep_heavy_msg: ****************************************************************** ******************** INITIATING HEAVY SWEEP ********************** ****************************************************************** Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 0 hop path: Initial path = [0] Return path = [0] Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Channel Adapter node, GUID = 0x66a009800044d, TID = 0x1234. Jun 02 18:14:35 [B567DBB0] -> __osm_nd_rcv_process_nd: Node 0x66a009800044d Description = MT23108 InfiniHost Mellanox Technologies. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 1 with GUID = 0x66a00a000044d for parent node GUID = 0x66a009800044d, TID = 0x1236. Jun 02 18:14:35 [B567DBB0] -> osm_vendor_set_sm: setting IS_SM capability mask failed; errno 2 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 1 with GUID = 0x66a00a000044d for parent node GUID = 0x66a009800044d, TID = 0x1237. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 1 with GUID = 0x66a00a000044d for parent node GUID = 0x66a009800044d, TID = 0x1238. Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 1 hop path: Initial path = [0][1] Return path = [0][4] Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Switch node, GUID = 0x84e000000014a, TID = 0x1239. Jun 02 18:14:35 [B567DBB0] -> __osm_nd_rcv_process_nd: Node 0x84e000000014a Description = DivergeNet Meshlink 4X Switch. Jun 02 18:14:35 [B567DBB0] -> __osm_si_rcv_process_new:Subnet max multicast lid is 0xC400 Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 0 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x123c. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x0. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 1 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x123d. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x1. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 2 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x123e. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x2. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 3 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x123f. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x3. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 4 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1240. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 5 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1241. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x5. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 6 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1242. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x6. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 7 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1243. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x7. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 8 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1244. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x8. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 0 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1246. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1247 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x10000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 1 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1247. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1248 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x10001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 1 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1248. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1249 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x20000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 2 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1249. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x124a attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x20001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 2 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x124a. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x124b attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x30000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 3 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x124b. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x124c attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x30001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 3 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x124c. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x124d attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x40000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 4 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x124d. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x124e attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x40001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 4 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x124e. Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 2 hop path: Initial path = [0][1][5] Return path = [0][4][6] Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Switch node, GUID = 0x84e0000000152, TID = 0x124f. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1250 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x50000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1251 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x50001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1252 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x60000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 5 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1250. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1253 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x60001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 5 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1251. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 6 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1252. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 6 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1253. Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 2 hop path: Initial path = [0][1][7] Return path = [0][4][7] Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Switch node, GUID = 0x2c900001cee10, TID = 0x1254. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1255 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x70000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 7 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1255. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1256 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x70001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 7 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1256. Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_existing: Rediscovered Switch node 0x2c900001cee10 TID = 0x1257, discovered 1 times already. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1258 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x80000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 8 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1258. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x1 trans_id................0x1259 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x80001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][4] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 8 with GUID = 0x84e000000014a for parent node GUID = 0x84e000000014a, TID = 0x1259. Jun 02 18:14:35 [B567DBB0] -> __osm_nd_rcv_process_nd: Node 0x84e0000000152 Description = DivergeNet Meshlink 4X Switch. Jun 02 18:14:35 [B567DBB0] -> osm_si_rcv_process: ERR 3610: Bad LinearFDBTop value = 0xC000 on switch 0x84e0000000152. Forcing correction to 0x0. Jun 02 18:14:35 [B567DBB0] -> __osm_nd_rcv_process_nd: Node 0x2c900001cee10 Description = MT43132 Mellanox Technologies. Jun 02 18:14:35 [B567DBB0] -> __osm_si_rcv_process_existing: discovery_count is:2 Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 0 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x125f. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x0. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 1 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1260. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x1. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 2 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1261. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x2. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 3 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1262. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x3. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 4 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1263. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x4. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 5 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1264. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x5. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 6 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1265. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 7 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1266. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x7. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 8 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1267. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x8. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 0 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x1269. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x0. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 1 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x126a. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x1. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 2 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x126b. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x2. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 3 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x126c. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x3. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 4 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x126d. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x4. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 5 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x126e. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x5. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 6 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x126f. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x6. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 7 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x1270. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Discovered port num 8 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x1271. Jun 02 18:14:35 [B567DBB0] -> osm_pi_rcv_process: Initializing port number 0x8. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 0 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1273. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1274 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x10000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1275 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x10001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 1 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1274. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1276 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x20000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 1 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1275. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 2 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1276. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1277 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x20001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 2 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1277. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1278 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x30000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 3 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1278. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1279 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x30001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 3 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1279. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x127a attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x40000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 4 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x127a. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x127b attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x40001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 4 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x127b. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x127c attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x50000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 5 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x127c. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x127d attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x50001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 5 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x127d. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x127e attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x60000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 6 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x127e. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x127f attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x60001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 6 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x127f. Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 3 hop path: Initial path = [0][1][5][7] Return path = [0][4][6][1] Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Channel Adapter node, GUID = 0x2c90108cd8ba0, TID = 0x1280. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1281 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x70000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 7 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1281. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1282 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x70001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 7 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1282. Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 3 hop path: Initial path = [0][1][5][8] Return path = [0][4][6][1] Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Channel Adapter node, GUID = 0x2c90108cd8a50, TID = 0x1283. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1284 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x80000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 8 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1284. Jun 02 18:14:35 [B3E7DBB0] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00. Jun 02 18:14:35 [B3E7DBB0] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x801C hop_ptr.................0x0 hop_count...............0x2 trans_id................0x1285 attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x80001 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1][5] Return path: [0][4][6] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 8 with GUID = 0x84e0000000152 for parent node GUID = 0x84e0000000152, TID = 0x1285. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 0 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x1286. Jun 02 18:14:35 [B567DBB0] -> Received a SMP on a 3 hop path: Initial path = [0][1][7][1] Return path = [0][4][7][1] Jun 02 18:14:35 [B567DBB0] -> __osm_ni_rcv_process_new: Discovered new Channel Adapter node, GUID = 0x66a009800043c, TID = 0x1287. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 1 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x1288. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 1 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x1289. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 2 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x128a. Jun 02 18:14:35 [B567DBB0] -> osm_pkey_rcv_process: Got GetResp(PKey) block:1 port_num 2 with GUID = 0x2c900001cee10 for parent node GUID = 0x2c900001cee10, TID = 0x128b. Jun 02 18:14:35 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:35 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:35 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:36 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:36 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:37 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:37 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:37 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:37 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:37 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:37 [B3E7DBB0] -> Initial path = [1][81][1] Jun 02 18:14:37 [B3E7DBB0] -> umad_receiver: send completed with error (method=1 attr=11) -- dropping. Jun 02 18:14:37 [B3E7DBB0] -> umad_receiver: DR SMP hop ptr 0 hop count 3 DR SLID 0xffff DR DLID 0xffff Jun 02 18:14:37 [B3E7DBB0] -> Initial path = [1][81][1] From mst at mellanox.co.il Thu Jun 2 16:45:19 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 3 Jun 2005 02:45:19 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602170734.GB29040@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> Message-ID: <20050602234519.GA29972@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [PATCH] rdma_lat-09 and results > > On Wed, Jun 01, 2005 at 05:52:28PM -0700, Grant Grundler wrote: > > Here's with the new rdma_lat.c: > > Here are some pfmon31 perf results for the new rdma_lat. > The first set is just L3 Cache misses. > The second set shows all cache misses. > > The sample rate on the second set was a bit high - note the > median cycle count was ~60 cycles higher than normal. > > Unfortunately pfmon doesn't look up symbols in shared libraries. > Shared libraries live in 0x2... address space on ia64. > I'll see if I can dig up the top hitters by hand. > 0x20000000002d7021 in particular is "of interest". > > And PLEASE, if you reply, please delete quoted text you are not responding > to from your reply. I'm getting tired of wading through 5 pages > of quotes to get to a 3 line comment. > > grant > > ====================== FIRST SET - L3 CACHE MISSES =============== > > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ taskset 1 ./rdma_lat -C -n 100000 > local address: LID 0x2d QPN 0x60406 PSN 0xd2cfc8 RKey 0x240434 VAddr 0x6000000000014001 > remote address: LID 0x2b QPN 0x60406 PSN 0x91ea3b RKey 0x240434 VAddr 0x6000000000014001 > Latency typical: 7140 cycles > Latency best : 6881 cycles > Latency worst : 293444 cycles > > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ pfmon31 --us-c --cpu-list=1 --resolve-addr --smpl-module=dear-hist-itanium2 -e data_ear_cache_lat64 --long-smpl-periods=20 -- ./rdma_lat -C 10.0.0.51 -n 100000 > added event set 0 > local address: LID 0x2b QPN 0x60406 PSN 0x91ea3b RKey 0x240434 VAddr 0x6000000000014001 > remote address: LID 0x2d QPN 0x60406 PSN 0xd2cfc8 RKey 0x240434 VAddr 0x6000000000014001 > Latency typical: 7136.5 cycles > Latency best : 6870 cycles > Latency worst : 293396 cycles > # total_samples 1937 > # instruction addr view > # sorted by count > # showing per per distinct value > # %L2 : percentage of L1 misses that hit L2 > # %L3 : percentage of L1 misses that hit L3 > # %RAM : percentage of L1 misses that hit memory > # L2 : 5 cycles load latency > # L3 : 14 cycles load latency > # sampling period: 20 > # #count %self %cum %L2 %L3 %RAM instruction addr > 630 32.52% 32.52% 0.00% 0.00% 100.00% 0x200000000032ba80 0x200000000032ba80 > 612 31.60% 64.12% 0.00% 0.00% 100.00% 0x4000000000003f10 main+0x1a60 > 568 29.32% 93.44% 0.00% 0.00% 100.00% 0x4000000000003f60 main+0x1ab0 > 60 3.10% 96.54% 0.00% 0.00% 100.00% 0x4000000000003fd1 main+0x1b21 > 36 1.86% 98.40% 0.00% 0.00% 100.00% 0x40000000000040c0 main+0x1c10 > 10 0.52% 98.92% 0.00% 0.00% 100.00% 0x2000000000141081 0x2000000000141081 > 8 0.41% 99.33% 0.00% 0.00% 100.00% 0x40000000000040b0 main+0x1c00 > 8 0.41% 99.74% 0.00% 0.00% 100.00% 0x2000000000141061 0x2000000000141061 > 3 0.15% 99.90% 0.00% 0.00% 100.00% 0x20000000002d7021 0x20000000002d7021 > 1 0.05% 99.95% 0.00% 0.00% 100.00% 0x20000000001316f0 0x20000000001316f0 > 1 0.05% 100.00% 0.00% 0.00% 100.00% 0x4000000000002470 cycles_compare > # level 0 : counts=0 avg_cycles=0.0ms 0.00% > # level 1 : counts=0 avg_cycles=0.0ms 0.00% > # level 2 : counts=1937 avg_cycles=0.0ms 100.00% > approx cost: 0.0s > > > ====================== SECOND SET - ALL CACHE LEVELS =============== > > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ taskset 1 ./rdma_lat -C -n 100000 > local address: LID 0x2d QPN 0x80406 PSN 0xbb769a RKey 0x300434 VAddr 0x6000000000014001 > remote address: LID 0x2b QPN 0x80406 PSN 0x60ce3b RKey 0x300434 VAddr 0x6000000000014001 > Latency typical: 7203.5 cycles > Latency best : 6881.5 cycles > Latency worst : 297164 cycles > > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ pfmon31 --us-c --cpu-list=1 --resolve-addr --smpl-module=dear-hist-itanium2 -e data_ear_cache_lat4 --long-smpl-periods=20 -- ./rdma_lat -C 10.0.0.51 -n 100000 > added event set 0 > local address: LID 0x2b QPN 0x80406 PSN 0x60ce3b RKey 0x300434 VAddr 0x6000000000014001 > remote address: LID 0x2d QPN 0x80406 PSN 0xbb769a RKey 0x300434 VAddr 0x6000000000014001 > Latency typical: 7196.5 cycles > Latency best : 6896.5 cycles > Latency worst : 297009 cycles > # total_samples 231428 > # instruction addr view > # sorted by count > # showing per per distinct value > # %L2 : percentage of L1 misses that hit L2 > # %L3 : percentage of L1 misses that hit L3 > # %RAM : percentage of L1 misses that hit memory > # L2 : 5 cycles load latency > # L3 : 14 cycles load latency > # sampling period: 20 > # #count %self %cum %L2 %L3 %RAM instruction addr > 55,598 24.02% 24.02% 99.94% 0.06% 0.00% 0x4000000000003f71 main+0x1ac1 > 55,461 23.96% 47.99% 0.00% 100.00% 0.00% 0x200000000032ba41 0x200000000032ba41 > 54,671 23.62% 71.61% 99.00% 0.09% 0.91% 0x4000000000003f60 main+0x1ab0 > 54,427 23.52% 95.13% 0.00% 0.00% 100.00% 0x20000000002d7021 0x20000000002d7021 > 1,327 0.57% 95.70% 74.91% 23.66% 1.43% 0x2000000000141061 0x2000000000141061 > 1,283 0.55% 96.26% 75.45% 21.75% 2.81% 0x2000000000141081 0x2000000000141081 > 808 0.35% 96.61% 93.56% 6.19% 0.25% 0x4000000000002470 cycles_compare > 744 0.32% 96.93% 34.81% 0.00% 65.19% 0x200000000032ba80 0x200000000032ba80 > 710 0.31% 97.23% 0.00% 89.01% 10.99% 0x4000000000002471 cycles_compare+0x1 > 545 0.24% 97.47% 4.04% 10.09% 85.87% 0x4000000000003f10 main+0x1a60 > 522 0.23% 97.70% 100.00% 0.00% 0.00% 0x200000000032da70 0x200000000032da70 > 507 0.22% 97.92% 99.80% 0.20% 0.00% 0x40000000000048f0 main+0x2440 > 500 0.22% 98.13% 0.00% 0.00% 100.00% 0x4000000000003fd1 main+0x1b21 > 490 0.21% 98.34% 98.57% 1.43% 0.00% 0x200000000032d4c0 0x200000000032d4c0 > 485 0.21% 98.55% 0.00% 0.00% 100.00% 0x200000000032bac1 0x200000000032bac1 > 452 0.20% 98.75% 99.78% 0.22% 0.00% 0x200000000032bdb0 0x200000000032bdb0 > 448 0.19% 98.94% 99.78% 0.22% 0.00% 0x200000000032dd20 0x200000000032dd20 > 431 0.19% 99.13% 0.00% 100.00% 0.00% 0x200000000032b5d1 0x200000000032b5d1 > 418 0.18% 99.31% 77.51% 22.49% 0.00% 0x200000000032b8d0 0x200000000032b8d0 > 334 0.14% 99.45% 99.10% 0.90% 0.00% 0x4000000000004931 main+0x2481 > 258 0.11% 99.56% 98.45% 1.55% 0.00% 0x20000000001410d1 0x20000000001410d1 > 220 0.10% 99.66% 98.18% 1.82% 0.00% 0x20000000001410b0 0x20000000001410b0 > 91 0.04% 99.70% 42.86% 19.78% 37.36% 0x40000000000040c0 main+0x1c10 > 83 0.04% 99.73% 100.00% 0.00% 0.00% 0x200000000032bcd1 0x200000000032bcd1 > 74 0.03% 99.77% 1.35% 98.65% 0.00% 0x4000000000003f90 main+0x1ae0 > 55 0.02% 99.79% 100.00% 0.00% 0.00% 0x2000000000141100 0x2000000000141100 > 44 0.02% 99.81% 20.45% 79.55% 0.00% 0x20000000000ae0b0 0x20000000000ae0b0 > 37 0.02% 99.82% 24.32% 70.27% 5.41% 0x20000000000ae0e0 0x20000000000ae0e0 > 25 0.01% 99.84% 4.00% 0.00% 96.00% 0x40000000000040b0 main+0x1c00 > 21 0.01% 99.84% 0.00% 100.00% 0.00% 0x200000000032b9d1 0x200000000032b9d1 > 21 0.01% 99.85% 14.29% 85.71% 0.00% 0x4000000000004950 main+0x24a0 > 20 0.01% 99.86% 100.00% 0.00% 0.00% 0x200000000032d571 0x200000000032d571 > 15 0.01% 99.87% 33.33% 53.33% 13.33% 0x20000000000231d0 0x20000000000231d0 > 14 0.01% 99.88% 100.00% 0.00% 0.00% 0x200000000032fb40 0x200000000032fb40 > 11 0.00% 99.88% 0.00% 100.00% 0.00% 0x2000000000141111 0x2000000000141111 > 10 0.00% 99.88% 70.00% 20.00% 10.00% 0x2000000000011020 0x2000000000011020 > 8 0.00% 99.89% 75.00% 25.00% 0.00% 0x2000000000010f90 0x2000000000010f90 > 8 0.00% 99.89% 62.50% 37.50% 0.00% 0x200000000032d500 0x200000000032d500 > 6 0.00% 99.89% 66.67% 16.67% 16.67% 0x2000000000011060 0x2000000000011060 > 5 0.00% 99.90% 0.00% 100.00% 0.00% 0x2000000000021570 0x2000000000021570 > 5 0.00% 99.90% 20.00% 60.00% 20.00% 0x2000000000014b21 0x2000000000014b21 > 5 0.00% 99.90% 100.00% 0.00% 0.00% 0x2000000000021880 0x2000000000021880 > 5 0.00% 99.90% 20.00% 80.00% 0.00% 0x2000000000010fd1 0x2000000000010fd1 > 5 0.00% 99.90% 100.00% 0.00% 0.00% 0x2000000000010ad0 0x2000000000010ad0 > 5 0.00% 99.91% 100.00% 0.00% 0.00% 0x200000000032b8f1 0x200000000032b8f1 > 4 0.00% 99.91% 0.00% 25.00% 75.00% 0x2000000000018c20 0x2000000000018c20 > 4 0.00% 99.91% 100.00% 0.00% 0.00% 0x2000000000021520 0x2000000000021520 > 4 0.00% 99.91% 25.00% 75.00% 0.00% 0x2000000000010a81 0x2000000000010a81 > 4 0.00% 99.91% 0.00% 100.00% 0.00% 0x4000000000004941 main+0x2491 > 3 0.00% 99.91% 66.67% 33.33% 0.00% 0x2000000000014b11 0x2000000000014b11 > 3 0.00% 99.92% 33.33% 33.33% 33.33% 0x2000000000017de0 0x2000000000017de0 > 3 0.00% 99.92% 66.67% 33.33% 0.00% 0x2000000000018421 0x2000000000018421 > 3 0.00% 99.92% 66.67% 33.33% 0.00% 0x200000000012f791 0x200000000012f791 > 3 0.00% 99.92% 0.00% 33.33% 66.67% 0x200000000012f850 0x200000000012f850 > 3 0.00% 99.92% 0.00% 100.00% 0.00% 0x2000000000010a41 0x2000000000010a41 > 3 0.00% 99.92% 66.67% 33.33% 0.00% 0x20000000000110c0 0x20000000000110c0 > 3 0.00% 99.92% 0.00% 0.00% 100.00% 0x20000000001dfa00 0x20000000001dfa00 > 3 0.00% 99.93% 100.00% 0.00% 0.00% 0x200000000032d740 0x200000000032d740 > 3 0.00% 99.93% 0.00% 100.00% 0.00% 0x200000000032fad0 0x200000000032fad0 > 2 0.00% 99.93% 0.00% 0.00% 100.00% 0x20000000000231d1 0x20000000000231d1 > 2 0.00% 99.93% 50.00% 50.00% 0.00% 0x2000000000010770 0x2000000000010770 > 2 0.00% 99.93% 0.00% 100.00% 0.00% 0x2000000000015020 0x2000000000015020 > 2 0.00% 99.93% 50.00% 50.00% 0.00% 0x2000000000014610 0x2000000000014610 > 2 0.00% 99.93% 0.00% 100.00% 0.00% 0x2000000000021850 0x2000000000021850 > 2 0.00% 99.93% 100.00% 0.00% 0.00% 0x2000000000014a20 0x2000000000014a20 > 2 0.00% 99.93% 100.00% 0.00% 0.00% 0x2000000000011090 0x2000000000011090 > 2 0.00% 99.93% 100.00% 0.00% 0.00% 0x2000000000014ba0 0x2000000000014ba0 > 2 0.00% 99.93% 0.00% 50.00% 50.00% 0x2000000000010f10 0x2000000000010f10 > 2 0.00% 99.94% 50.00% 50.00% 0.00% 0x2000000000018451 0x2000000000018451 > 2 0.00% 99.94% 0.00% 0.00% 100.00% 0x2000000000010f20 0x2000000000010f20 > 2 0.00% 99.94% 0.00% 100.00% 0.00% 0x20000000000126d0 0x20000000000126d0 > 2 0.00% 99.94% 100.00% 0.00% 0.00% 0x200000000012f781 0x200000000012f781 > 2 0.00% 99.94% 0.00% 100.00% 0.00% 0x20000000001906f1 0x20000000001906f1 > 2 0.00% 99.94% 100.00% 0.00% 0.00% 0x20000000000109e0 0x20000000000109e0 > 2 0.00% 99.94% 0.00% 0.00% 100.00% 0x20000000001deba0 0x20000000001deba0 > 2 0.00% 99.94% 50.00% 0.00% 50.00% 0x20000000002a4481 0x20000000002a4481 > 2 0.00% 99.94% 0.00% 100.00% 0.00% 0x2000000000190780 0x2000000000190780 > 2 0.00% 99.94% 0.00% 50.00% 50.00% 0x2000000000017d41 0x2000000000017d41 > 2 0.00% 99.94% 0.00% 100.00% 0.00% 0x20000000002b1ff1 0x20000000002b1ff1 > 2 0.00% 99.94% 100.00% 0.00% 0.00% 0x200000000013c7d1 0x200000000013c7d1 > 2 0.00% 99.95% 50.00% 50.00% 0.00% 0x200000000032d640 0x200000000032d640 > 2 0.00% 99.95% 0.00% 100.00% 0.00% 0x200000000032bc61 0x200000000032bc61 > 2 0.00% 99.95% 50.00% 50.00% 0.00% 0x200000000032a030 0x200000000032a030 > 2 0.00% 99.95% 100.00% 0.00% 0.00% 0x200000000032a450 0x200000000032a450 > 2 0.00% 99.95% 100.00% 0.00% 0.00% 0x200000000032fb70 0x200000000032fb70 > 2 0.00% 99.95% 100.00% 0.00% 0.00% 0x200000000032fac1 0x200000000032fac1 > 2 0.00% 99.95% 50.00% 50.00% 0.00% 0x2000000000083b30 0x2000000000083b30 > 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x2000000000005330 0x2000000000005330 > 1 0.00% 99.95% 0.00% 100.00% 0.00% 0x200000000000c3d0 0x200000000000c3d0 > 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x200000000000c530 0x200000000000c530 > 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x20000000000165b0 0x20000000000165b0 > 1 0.00% 99.95% 0.00% 100.00% 0.00% 0x20000000000246d1 0x20000000000246d1 > 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x20000000000059f0 0x20000000000059f0 > 1 0.00% 99.95% 0.00% 100.00% 0.00% 0x20000000000062e0 0x20000000000062e0 > 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x20000000000216f0 0x20000000000216f0 > 1 0.00% 99.95% 100.00% 0.00% 0.00% 0x2000000000010e30 0x2000000000010e30 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000014d81 0x2000000000014d81 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000021800 0x2000000000021800 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000006400 0x2000000000006400 > 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x2000000000011f40 0x2000000000011f40 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000014d71 0x2000000000014d71 > 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x2000000000018441 0x2000000000018441 > 1 0.00% 99.96% 0.00% 0.00% 100.00% 0x2000000000019070 0x2000000000019070 > 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x20000000001356e1 0x20000000001356e1 > 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x20000000000107c1 0x20000000000107c1 > 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x2000000000010a10 0x2000000000010a10 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000010e00 0x2000000000010e00 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000010a11 0x2000000000010a11 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000018ce1 0x2000000000018ce1 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000010f21 0x2000000000010f21 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000018c30 0x2000000000018c30 > 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x200000000000e6b0 0x200000000000e6b0 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x20000000000171d1 0x20000000000171d1 > 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x2000000000010f50 0x2000000000010f50 > 1 0.00% 99.96% 0.00% 0.00% 100.00% 0x2000000000044741 0x2000000000044741 > 1 0.00% 99.96% 0.00% 0.00% 100.00% 0x20000000002d83a1 0x20000000002d83a1 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x20000000002a4d10 0x20000000002a4d10 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x20000000000841e1 0x20000000000841e1 > 1 0.00% 99.96% 100.00% 0.00% 0.00% 0x20000000002a4e00 0x20000000002a4e00 > 1 0.00% 99.96% 0.00% 100.00% 0.00% 0x2000000000017e81 0x2000000000017e81 > 1 0.00% 99.97% 0.00% 0.00% 100.00% 0x20000000002a3f81 0x20000000002a3f81 > 1 0.00% 99.97% 0.00% 0.00% 100.00% 0x2000000000017d20 0x2000000000017d20 > 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000000109f0 0x20000000000109f0 > 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000002a6ca0 0x20000000002a6ca0 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x2000000000017f70 0x2000000000017f70 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a7b11 0x20000000002a7b11 > 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x2000000000017d21 0x2000000000017d21 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002acb90 0x20000000002acb90 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a6770 0x20000000002a6770 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x2000000000132290 0x2000000000132290 > 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000002a4cd0 0x20000000002a4cd0 > 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000002a43a1 0x20000000002a43a1 > 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000000109e1 0x20000000000109e1 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a4321 0x20000000002a4321 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x2000000000084190 0x2000000000084190 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x2000000000011b41 0x2000000000011b41 > 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x200000000010e4d1 0x200000000010e4d1 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a4461 0x20000000002a4461 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x200000000013a1f0 0x200000000013a1f0 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a4201 0x20000000002a4201 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002b27a0 0x20000000002b27a0 > 1 0.00% 99.97% 0.00% 100.00% 0.00% 0x20000000002a3da1 0x20000000002a3da1 > 1 0.00% 99.97% 100.00% 0.00% 0.00% 0x20000000002a4301 0x20000000002a4301 > 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x2000000000132860 0x2000000000132860 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4021 0x20000000002a4021 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002b1ff0 0x20000000002b1ff0 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4741 0x20000000002a4741 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a3f01 0x20000000002a3f01 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x200000000013cb70 0x200000000013cb70 > 1 0.00% 99.98% 0.00% 0.00% 100.00% 0x20000000001906b0 0x20000000001906b0 > 1 0.00% 99.98% 0.00% 0.00% 100.00% 0x200000000012f7b1 0x200000000012f7b1 > 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x20000000001321c1 0x20000000001321c1 > 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x20000000001314a0 0x20000000001314a0 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4210 0x20000000002a4210 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x2000000000132240 0x2000000000132240 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4430 0x20000000002a4430 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x200000000013ce11 0x200000000013ce11 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000001308d1 0x20000000001308d1 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000001d0ea1 0x20000000001d0ea1 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002b1b70 0x20000000002b1b70 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x2000000000131a20 0x2000000000131a20 > 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x20000000001318b0 0x20000000001318b0 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002b1e00 0x20000000002b1e00 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x20000000002a4810 0x20000000002a4810 > 1 0.00% 99.98% 100.00% 0.00% 0.00% 0x2000000000017e60 0x2000000000017e60 > 1 0.00% 99.98% 0.00% 100.00% 0.00% 0x4000000000002600 main+0x150 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x20000000001317b1 0x20000000001317b1 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x2000000000010df0 0x2000000000010df0 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x2000000000045c80 0x2000000000045c80 > 1 0.00% 99.99% 0.00% 0.00% 100.00% 0x200000000012f701 0x200000000012f701 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x4000000000001290 _init+0x50 > 1 0.00% 99.99% 0.00% 0.00% 100.00% 0x20000000000e0560 0x20000000000e0560 > 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x2000000000017e20 0x2000000000017e20 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x4000000000003650 main+0x11a0 > 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x20000000000e51f1 0x20000000000e51f1 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x2000000000017d00 0x2000000000017d00 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x20000000000b3660 0x20000000000b3660 > 1 0.00% 99.99% 0.00% 0.00% 100.00% 0x20000000000424e1 0x20000000000424e1 > 1 0.00% 99.99% 0.00% 0.00% 100.00% 0x200000000012d051 0x200000000012d051 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032ba40 0x200000000032ba40 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032be50 0x200000000032be50 > 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x200000000032d8b0 0x200000000032d8b0 > 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x200000000032a201 0x200000000032a201 > 1 0.00% 99.99% 100.00% 0.00% 0.00% 0x200000000032be10 0x200000000032be10 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x4000000000003f81 main+0x1ad1 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032ba50 0x200000000032ba50 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032bc01 0x200000000032bc01 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x4000000000003f91 main+0x1ae1 > 1 0.00% 99.99% 0.00% 100.00% 0.00% 0x200000000032bc60 0x200000000032bc60 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000032b570 0x200000000032b570 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000032a441 0x200000000032a441 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000032a210 0x200000000032a210 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000013f311 0x200000000013f311 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x200000000012faf0 0x200000000012faf0 > 1 0.00% 100.00% 0.00% 100.00% 0.00% 0x200000000012c181 0x200000000012c181 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x20000000000ae341 0x20000000000ae341 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x20000000000ae340 0x20000000000ae340 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x20000000000db490 0x20000000000db490 > 1 0.00% 100.00% 0.00% 100.00% 0.00% 0x20000000000ef5d0 0x20000000000ef5d0 > 1 0.00% 100.00% 0.00% 0.00% 100.00% 0x2000000000019620 0x2000000000019620 > 1 0.00% 100.00% 100.00% 0.00% 0.00% 0x20000000000198d1 0x20000000000198d1 > # level 0 : counts=116599 avg_cycles=0.0ms 50.38% > # level 1 : counts=57736 avg_cycles=0.0ms 24.95% > # level 2 : counts=57093 avg_cycles=0.0ms 24.67% > approx cost: 0.0s > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > Its possible to link libibverbs and libmthca statically. I did it once. -- MST From hozer at hozed.org Thu Jun 2 16:46:20 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Thu, 2 Jun 2005 18:46:20 -0500 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050602232331.GD577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> Message-ID: <20050602234620.GE577@kalmia.hozed.org> On Thu, Jun 02, 2005 at 06:23:31PM -0500, Troy Benjegerdes wrote: > I'm having intermittent problems with opensm.. It seems after a while > IPoIB stops working and if I restart opensm, it starts spitting out > errors. Do I have a misbehaving switch somewhere? > > ibnetdiscover seems to work fine. > > > (this is from running 'opensm -v -o -r') > Some more info.. I rebooted the switches, and tried to re-run it. I found that ibnetdiscover showed everything with a LID of 0 except 1 HCA card.. when I found that machine and did 'rmmod ib_mthca', opensm seemed to get unstuck and mapped all the other lids. And just now, as a sanity check, I was going to reload all the IB modules, but got the following panic: gozer.scl.ameslab.gov login: ib_mad: Invalid directed route ib_mthca 0000:03:00.0: dma_pool_destroy mthca_av, f71c2000 busy Unable to handle kernel NULL pointer dereference at virtual address 00000000 printing eip: c012d348 *pde = 00000000 Oops: 0002 [#1] SMP Modules linked in: ib_umad nfsd lockd sunrpc ipv6 evdev floppy pcspkr ib_mad ib_core shpchp pci_hotplug ohci_hcd usbcore serverworks i2c_piix4 i2c_core sworks_agp agpgart aic7xxx tg3 xfs exportfs capability commoncap ide_cd ide_core cdrom genrtc isofs ext2 ext3 jbd mbcache sd_mod aacraid scsi_mod unix fbcon font bitblit vesafb cfbcopyarea cfbimgblt cfbfillrect CPU: 0 EIP: 0060:[] Tainted: GF VLI EFLAGS: 00010812 (2.6.11-1-686-smp) EIP is at __queue_work+0x38/0x70 eax: c2157414 ebx: c2157400 ecx: 00000000 edx: f77ea2d4 esi: f77ea2d0 edi: 00000286 ebp: c012d3f0 esp: f7277f44 ds: 007b es: 007b ss: 0068 Process default.hotplug (pid: 3102, threadinfo=f7276000 task=f70d7a60) Stack: f7841580 f77ea200 f77ea2d0 c200c9a0 c012d432 c2157400 f77ea2d0 f77ea2d0 00000100 c01262e6 f77ea2d0 f7277fa0 c0118093 00000000 f7276000 f7277f80 f7277f80 000001f0 00000011 c035ff68 c0397aa0 00000000 c0121cca c035ff68 Call Trace: [] delayed_work_timer_fn+0x42/0x50 [] run_timer_softirq+0xd6/0x1c0 [] scheduler_tick+0x63/0x320 [] __do_softirq+0xba/0xd0 [] do_softirq+0x2d/0x30 [] apic_timer_interrupt+0x1c/0x24 Code: 24 08 8b 74 24 18 89 d8 89 7c 24 0c e8 22 fb 17 00 89 5e 14 89 c7 8d 56 04 8d 43 0c 8b 48 04 89 46 04 89 50 04 8d 43 14 89 4a 04 <89> 11 ba 03 00 00 00 b9 01 00 00 00 ff 43 08 c7 04 24 00 00 00 <0>Kernel panic - not syncing: Fatal exception in interrupt From hozer at hozed.org Thu Jun 2 17:25:51 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Thu, 2 Jun 2005 19:25:51 -0500 Subject: [openib-general] cable test/error count utilities? Message-ID: <20050603002551.GF577@kalmia.hozed.org> Some of my problems seem to be from intermittent cables.. Is there anything for OpenIB that can read error counters? What I'd really like to see is something that I can integrate with nagios ( http://www.nagios.org/about ) -- -------------------------------------------------------------------------- Troy Benjegerdes 'da hozer' hozer at hozed.org Somone asked my why I work on this free (http://www.fsf.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer: "Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life." -- Charles Shultz From jcarr at linuxmachines.com Thu Jun 2 18:37:21 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 02 Jun 2005 18:37:21 -0700 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: <78d18e20506011443175617b3@mail.gmail.com> References: <1117539047.4476.199.camel@hal.voltaire.com> <78d18e20506011443175617b3@mail.gmail.com> Message-ID: <429FB451.3060602@linuxmachines.com> On 06/01/05 14:43, William Jordan wrote: >>Has anyone seen ib_send_cm_req() return -22? > > > I'm not sure what you are testing with, Jeff, but I ran into the same > problem the first time I tried to use ucm_simple. The source and > destination lid and guid are embedded in the source, and need to be > modified to fit the test systems. Otherwise you will get an errno > EINVAL. > On the client system: # cat /sys/class/infiniband/mthca0/ports/1/lid 0x2 On the server system: # cat /sys/class/infiniband/mthca0/ports/1/lid 0x3 modprobe ib_cmpost "slid=0x3" "dlid=0x2" "message_count=0x10" "message_size=0x100" Yet on the client: root at delta:~# dmesg |tail -n 10 NET: Registered protocol family 10 Disabled Privacy Extensions on device c033a200(lo) IPv6 over IPv4 tunneling driver eth0: no IPv6 routers present ib1: no IPv6 routers present ib0: no IPv6 routers present cmpost: starting client cmpost: connecting cmpost: failure sending REQ: -22 cmpost: test complete So, it's not something simple like the LID's not matching. Of course, that doesn't rule out that I'm not doing something else stupid. :) Jeff From jcarr at linuxmachines.com Thu Jun 2 18:43:53 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 02 Jun 2005 18:43:53 -0700 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: <1117751271.4476.178.camel@hal.voltaire.com> References: <429F837F.8020005@linuxmachines.com> <1117751271.4476.178.camel@hal.voltaire.com> Message-ID: <429FB5D9.7040908@linuxmachines.com> On 06/02/05 15:27, Hal Rosenstock wrote: > On Thu, 2005-06-02 at 18:09, Jeff Carr wrote: > >>Is there a simple way to discover the lid values of other systems? > > > Locally, you can run /usr/local/ib/bin/ibstatus or ibstat. > > Remotely, you would need to know the remote GID (subnet prefix + GUID) > and ask the SA what the LID was for that (by getting a path record). Thanks guys. I'll keep at it then and see if I can figure it out. So far, I was looking at the IPoIB code where path_rec_start() calls ib_sa_path_rec_get(). At least that looks like where the discovery must occur. Jeff From iod00d at hp.com Thu Jun 2 18:47:53 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 2 Jun 2005 18:47:53 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602234519.GA29972@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> Message-ID: <20050603014753.GA31261@esmail.cup.hp.com> On Fri, Jun 03, 2005 at 02:45:19AM +0300, Michael S. Tsirkin wrote: > > And PLEASE, if you reply, please delete quoted text you are not responding > > to from your reply. I'm getting tired of wading through 5 pages > > of quotes to get to a 3 line comment. Michael, Could you please at least attempt to delete some of the text you are quoting? It's quite rude to quote several hundred lines of output just to make a two line comment. > Its possible to link libibverbs and libmthca statically. > I did it once. Ok. I can try that. thanks, grant From iod00d at hp.com Thu Jun 2 19:14:04 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 2 Jun 2005 19:14:04 -0700 Subject: [openib-general] cable test/error count utilities? In-Reply-To: <20050603002551.GF577@kalmia.hozed.org> References: <20050603002551.GF577@kalmia.hozed.org> Message-ID: <20050603021404.GC31261@esmail.cup.hp.com> On Thu, Jun 02, 2005 at 07:25:51PM -0500, Troy Benjegerdes wrote: > Some of my problems seem to be from intermittent cables.. > > Is there anything for OpenIB that can read error counters? cat /sys/class/infiniband/mthca0/ports/1/counters/*errors > What I'd really like to see is something that I can integrate with > nagios ( http://www.nagios.org/about ) Cool - nice looking tool. grant From tduffy at sun.com Thu Jun 2 19:20:23 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 02 Jun 2005 19:20:23 -0700 Subject: [openib-general] [PATCH] kdapltest: fix pointer to pointer bug Message-ID: <1117765223.4119.15.camel@duffman> In my work going through trying to get rid of the opaque dat handles, I came across what looks like a bug in kdapltest. I don't think DT_Performance_Test_Create() and DT_Performance_Test_Client() should take a DAT_IA_HANDLE * as an argument as this would be a pointer to a pointer. Of course, the compiler didn't catch it until I changed it to a real struct pointers thus showing one of the problems with using opaques. Signed-off-by: Tom Duffy Index: linux-kernel-clean/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel-clean/test/dapltest/test/dapl_performance_util.c (revision 2532) +++ linux-kernel-clean/test/dapltest/test/dapl_performance_util.c (working copy) @@ -33,7 +33,7 @@ boolean_t DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE *ia_handle, + DAT_IA_HANDLE ia_handle, struct sockaddr * remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, Index: linux-kernel-clean/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel-clean/test/dapltest/test/dapl_performance_client.c (revision 2532) +++ linux-kernel-clean/test/dapltest/test/dapl_performance_client.c (working copy) @@ -34,7 +34,7 @@ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE *ia_handle, + DAT_IA_HANDLE ia_handle, struct sockaddr * remote_ia_addr) { Performance_Test_t *test_ptr = NULL; Index: linux-kernel-clean/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel-clean/test/dapltest/include/dapl_proto.h (revision 2532) +++ linux-kernel-clean/test/dapltest/include/dapl_proto.h (working copy) @@ -236,7 +236,7 @@ void DT_Performance_Cmd_Endia /* dapl_performance_client.c */ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + DAT_IA_HANDLE ia_handle, struct sockaddr *remote); boolean_t DT_Performance_Test_Client_Connect ( @@ -261,7 +261,7 @@ boolean_t DT_Performance_Test /* dapl_performance_util.c */ boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + DAT_IA_HANDLE ia_handle, struct sockaddr *remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, From itamar at mellanox.co.il Thu Jun 2 22:54:10 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Fri, 3 Jun 2005 08:54:10 +0300 Subject: [openib-general] RE: [patch][kdapl] enable kdapltest -T P Message-ID: <91DB792C7985D411BEC300B40080D29CC35C24@mtvex01.mtv.mtl.com> */ > > > Could we initialize the ia_attr.max_evd_qlen value correctly in the > provider? > > > Current openib gen2 code is not reporting the max cq size and i dont think that we should put a fix number . if we want to get the number we need Roland to fill this number in mthca but as Roland said before "what real App will meed this number?" > > Again, why not initialize these correclty in the provider? > again the same as max_evd_qlen value > > Good catch. 255 was not a good value to have been using. > > Could we use pt_ptr->provider_attr.optimal_buffer_alignment instead? if it is 256 that it is O.k but i dont think we should fix it the current code is O.k for me. > > Index: test/dapl_bpool.c > > =================================================================== > > --- test/dapl_bpool.c (revision 2522) > > +++ test/dapl_bpool.c (working copy) > > @@ -388,7 +388,7 @@ > > u64 > > DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index) > > { > > - return ( bpool_ptr->reg_addr + index * bpool_ptr->seg_size ); > > + return (virt_to_phys(DT_Bpool_GetBuffer(bpool_ptr,index))); > > Was this also a problem in the transaction test? > Yes but in order to fall on the bug you need to alloc a very small buffer (like 12 byte) and this is only in -T P . Itamar From mst at mellanox.co.il Fri Jun 3 00:25:53 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 3 Jun 2005 10:25:53 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050603014753.GA31261@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050603014753.GA31261@esmail.cup.hp.com> Message-ID: <20050603072553.GA5184@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [PATCH] rdma_lat-09 and results > > On Fri, Jun 03, 2005 at 02:45:19AM +0300, Michael S. Tsirkin wrote: > > > And PLEASE, if you reply, please delete quoted text you are not responding > > > to from your reply. I'm getting tired of wading through 5 pages > > > of quotes to get to a 3 line comment. > > Michael, > Could you please at least attempt to delete some of the text you are quoting? > It's quite rude to quote several hundred lines of output just > to make a two line comment. Sorry, it was 2am or so here. I just wanted to help. -- MST From hch at lst.de Fri Jun 3 00:47:33 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 3 Jun 2005 09:47:33 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <469958e00506020949356275b9@mail.gmail.com> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050601073609.GA14042@lst.de> <20050601100410.GA16265@lst.de> <1117649626.29566.2.camel@duffman> <1117726034.15225.1.camel@duffman> <469958e00506020949356275b9@mail.gmail.com> Message-ID: <20050603074733.GA27214@lst.de> On Thu, Jun 02, 2005 at 09:49:51AM -0700, Caitlin Bestler wrote: > > The other "issue" right now is that dapl has a header struct that needs > > to come first in all the structs. So, that would need to be changed. > > > > That is what enables the use of the method table. It allows the > in-line function to know the actual method to call for 'dat_ep_post_send' > on this specific EP without knowing at compile time which Provider > code is being invoked. For that you don't need the header to come first, though. It can be anywhere in the public part of the object. From hch at lst.de Fri Jun 3 01:06:09 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 3 Jun 2005 10:06:09 +0200 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e0050602095659d486ab@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C15@mtvex01.mtv.mtl.com> <469958e0050602095659d486ab@mail.gmail.com> Message-ID: <20050603080609.GA27550@lst.de> On Thu, Jun 02, 2005 at 09:56:51AM -0700, Caitlin Bestler wrote: > Why is it that you believe that the DAT registry does not support > plug and play? The interface was most specifically designed > to allow that. DAT is based on a enumerate and request instead of a callback-based client interface. That means a client that wants to support hot plugged devices needs to poll the registry. It also means it needs to provide an awkward in-kernel enumeration inerface. > Keep in mind that loading/unloading DAT Provider is *not* > synonymous with loading/unloading drivers. In fact I believe > the intent is to have a single provider that supports multiple > devices. Such a provider would simply register itself when > it was loaded. The current code looks like that, yes. But long term there should be no separate DAT provider, but the functionality now covered by kDAP should be converged into a general RDMA provider. From halr at voltaire.com Fri Jun 3 03:14:34 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 06:14:34 -0400 Subject: [openib-general] cable test/error count utilities? In-Reply-To: <20050603002551.GF577@kalmia.hozed.org> References: <20050603002551.GF577@kalmia.hozed.org> Message-ID: <1117793673.4476.918.camel@hal.voltaire.com> On Thu, 2005-06-02 at 20:25, Troy Benjegerdes wrote: > Some of my problems seem to be from intermittent cables.. > > Is there anything for OpenIB that can read error counters? Aside from pulling these from the driver via /sys/class/infiniband/mthca0/ports/1/counters/, there is also perfquery which displays the portcounters (which contains the error counters): Usage: perfquery [-d(ebug) -G(uid_addr) -a(ll_ports) -r(reset_after_read) -C ca_name -P hca_port -R(eset_only) -t timeout_ms -V(ersion) -h(elp)] [ [[port] [reset_mask]]] Examples: perfquery # read local port's performance counters perfquery 32 1 # read performance counters from lid 32, port 1 perfquery -a 32 # read performance counters from lid 32, all ports perfquery -r 32 1 # read performance counters and reset perfquery -R 32 1 # reset performance counters of port 1 only perfquery -R -a 32 # reset performance counters of all ports perfquery -R 32 2 0xf000 # reset only non-error counters of port 2 perfquery 2 1 # Port counters: Lid 0x2 port 1 PortSelect:......................1 CounterSelect:...................0x0000 SymbolErrors:....................1506 LinkRecovers:....................255 LinkDowned:......................1 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................0 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............0 VL15Dropped:.....................0 XmtBytes:........................2612 RcvBytes:........................2160 XmtPkts:.........................36 RcvBytes:........................30 > What I'd really like to see is something that I can integrate with > nagios ( http://www.nagios.org/about ) Nagios says it runs external plugins so it would be possible to create one for this which based on polling counters at some rate could cause the contact notifications to be issued based on some algorithm for deciding that this is appropriate (e.g. error counters are increasing so a cable might be intermittent (e.g. certain link is suspect)). -- Hal From halr at voltaire.com Fri Jun 3 03:39:43 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 06:39:43 -0400 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: <429F8DF8.3050806@ichips.intel.com> References: <429F837F.8020005@linuxmachines.com> <429F8DF8.3050806@ichips.intel.com> Message-ID: <1117795183.4476.995.camel@hal.voltaire.com> On Thu, 2005-06-02 at 18:53, Sean Hefty wrote: > And, yes, it would be really nice if the SM reassigned the same LIDs to the > same nodes where possible. It does this for some but not all cases now. LIDs are not persistent in IB unless something above and beyond the IB spec is done. SM would need to maintain a persistent store of GUIDs and LIDs assigned and reuse these and somehow have a way to reclaim them. This is something on the futures list for OpenSM. -- Hal From caitlin.bestler at gmail.com Fri Jun 3 04:30:51 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Fri, 3 Jun 2005 04:30:51 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> Message-ID: <469958e00506030430687795a8@mail.gmail.com> The appropriate plafce to add that would be as an unafilliated asynchronous event reported via the async evd. The next question is to define what if any handshake is desired. My guess that the consumer would acknowledge this by closing the RNIC, and that there would be some sort of deadline for doing so (much like a shutdown, you have so long to clean up, after that -- well you were warned). That is a simple extension to the DAT API that nobody has found urgent enough yet to propose, but it fits within the general approach well. Why don't you propose such an extension? You could even just create it and not wait for a response, if the proposal is reasonable it will naturally be merged back into the official DAT API. On 6/2/05, Itamar Rabenstein wrote: > Hi , > > I am not an expert to pci hot plug but as far as know we currently dont have > away in kdapl > to inform the consumer of kdapl that a provider is going to be unloaded. > pci hot plug should enable any low level driver (openib) to inform high > level driver(kdapl) > that the devide is going down. > > In openib gen2 this is doen with Ccall back remove_one(). > > how it is doen in current kdapl ? > how the consumer who opened IA will know that this card is going down? > > Itamar > = From caitlin.bestler at gmail.com Fri Jun 3 04:38:15 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Fri, 3 Jun 2005 04:38:15 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <20050603080609.GA27550@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C15@mtvex01.mtv.mtl.com> <469958e0050602095659d486ab@mail.gmail.com> <20050603080609.GA27550@lst.de> Message-ID: <469958e0050603043810f40f46@mail.gmail.com> On 6/3/05, Christoph Hellwig wrote: > > > Keep in mind that loading/unloading DAT Provider is *not* > > synonymous with loading/unloading drivers. In fact I believe > > the intent is to have a single provider that supports multiple > > devices. Such a provider would simply register itself when > > it was loaded. > > The current code looks like that, yes. But long term there should > be no separate DAT provider, but the functionality now covered by > kDAP should be converged into a general RDMA provider. > This issue was discussed in OpenRDMA, and reached the same goal. Yes, in the long run there is no need for there to be both dynamic loading of DAT Providers and of Verb Providers. But that is a necessary interim transition. OpenRDMA discussed with the DAT Collaborative the idea of subsuming the responsibilities of the DAT Registry, so that the OpenRDMA directory could take the 'dat_xxx' calls directly. When the device dependent logic used a dynamically loaded verbs, those would be invoked directly. When the device dependent logic required a full DAT Provider (because it did not provide a recognized verbs interface) then the existing dynamic DAT dispatch would be used. That approach is certainly applicable for OpenIB as well. The key is recognizing the need for a transition plan. Customers have DAT Providers installed now, they cannot synchronize getting new DAT Providers from their suppliers with a new Linux release. This is especially true since OpenIB does not currently define a verbs interface that is suitable for iWARP vendors to use. So dat_ia_openv() needs to still support existing dat_registry logic and existing DAT_PROVIDERs, otherwise it is breaking existing code. From halr at voltaire.com Fri Jun 3 05:37:04 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 08:37:04 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050602232331.GD577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> Message-ID: <1117802224.4476.1025.camel@hal.voltaire.com> On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote: > I'm having intermittent problems with opensm.. It seems after a while > IPoIB stops working Wonder if there is some relation to the two: intermittent IPoIB and lack of response to SM query. > and if I restart opensm, How did you get around the ABI version mismatch issue ? > it starts spitting out > errors. Do I have a misbehaving switch somewhere? It appears that a node is not responding to a discovery packet (SM Get NodeInfo (attrID 0x11)). It's direct route initial path (an array of port numbers at the start of the next hop) is: Initial path = [1][81][1] which means that starting at the node running OpenSM, port 1 then port 129 then port 1. Is there a large switch in the middle ? Can you send the output of ibnetdiscover ? If that is valid, which HCA (port) is not responding (what is the GUID) ? Unfortunately on such an error osm does not appear to give up (it retries forever and is locked on such a node). This is obviously not good. > ibnetdiscover seems to work fine. Are you sure it displays all HCA and switches and their ports ? I wouldn't think it would respond to ibnetdiscover if it didn't respond to osm. -- Hal > (this is from running 'opensm -v -o -r') From halr at voltaire.com Fri Jun 3 05:53:32 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 08:53:32 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050602234620.GE577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> <20050602234620.GE577@kalmia.hozed.org> Message-ID: <1117803212.4476.1059.camel@hal.voltaire.com> On Thu, 2005-06-02 at 19:46, Troy Benjegerdes wrote: > Some more info.. I rebooted the switches, and tried to re-run it. > > I found that ibnetdiscover showed everything with a LID of 0 except 1 > HCA card.. when I found that machine and did 'rmmod ib_mthca', opensm > seemed to get unstuck and mapped all the other lids. Yes, that would do it ("unsticK' things) as the link to that HCA port would not become LinkUp so opensm would just ignore it and everything would be fine. (That's what OpenSM really needs to do when it encounters a non responsive port with LinkUp; this has been discussed on this list before). > And just now, as a sanity check, I was going to reload all the IB > modules, but got the following panic: > > gozer.scl.ameslab.gov login: ib_mad: Invalid directed route That message means that for some reason an SMP with an invalid directed route was attempted to be sent and was discarded. The SMP either had a "unreasonable" hop count, or hop count and pointer were for a switch but the node was not a switch (this is occuring on an HCA port). [It likely would have been one received from the SM.] > ib_mthca 0000:03:00.0: dma_pool_destroy mthca_av, f71c2000 busy > Unable to handle kernel NULL pointer dereference at virtual address 00000000 > printing eip: > c012d348 > *pde = 00000000 > Oops: 0002 [#1] > SMP > Modules linked in: ib_umad nfsd lockd sunrpc ipv6 evdev floppy pcspkr ib_mad ib_core shpchp pci_hotplug ohci_hcd usbcore serverworks i2c_piix4 i2c_core sworks_agp agpgart aic7xxx tg3 xfs exportfs capability commoncap ide_cd ide_core cdrom genrtc isofs ext2 ext3 jbd mbcache sd_mod aacraid scsi_mod unix fbcon font bitblit vesafb cfbcopyarea cfbimgblt cfbfillrect > CPU: 0 > EIP: 0060:[] Tainted: GF VLI > EFLAGS: 00010812 (2.6.11-1-686-smp) > EIP is at __queue_work+0x38/0x70 > eax: c2157414 ebx: c2157400 ecx: 00000000 edx: f77ea2d4 > esi: f77ea2d0 edi: 00000286 ebp: c012d3f0 esp: f7277f44 > ds: 007b es: 007b ss: 0068 > > Process default.hotplug (pid: 3102, threadinfo=f7276000 task=f70d7a60) > Stack: f7841580 f77ea200 f77ea2d0 c200c9a0 c012d432 c2157400 f77ea2d0 f77ea2d0 > 00000100 c01262e6 f77ea2d0 f7277fa0 c0118093 00000000 f7276000 f7277f80 > f7277f80 000001f0 00000011 c035ff68 c0397aa0 00000000 c0121cca c035ff68 > > Call Trace: > [] delayed_work_timer_fn+0x42/0x50 > [] run_timer_softirq+0xd6/0x1c0 > [] scheduler_tick+0x63/0x320 > [] __do_softirq+0xba/0xd0 > [] do_softirq+0x2d/0x30 > [] apic_timer_interrupt+0x1c/0x24 > > Code: 24 08 8b 74 24 18 89 d8 89 7c 24 0c e8 22 fb 17 00 89 5e 14 89 c7 8d 56 04 8d 43 0c 8b 48 04 89 46 04 89 50 04 8d 43 14 89 4a 04 <89> 11 ba 03 00 00 00 b9 01 00 00 00 ff 43 08 c7 04 24 00 00 00 > <0>Kernel panic - not syncing: Fatal exception in interrupt It looks like ib_mad somehow references through a NULL pointer after this error while processing delayed work (timeout_sends ?). but I don't see it. -- Hal From halr at voltaire.com Fri Jun 3 06:10:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 09:10:41 -0400 Subject: [openib-general] [PATCH] sa_query: In send_mad, initialize wr.next to NULL Message-ID: <1117804241.4476.1069.camel@hal.voltaire.com> sa_query: In send_mad, initialize wr.next to NULL Signed-off-by: Hal Rosenstock Index: sa_query.c =================================================================== -- sa_query.c (revision 2532) +++ sa_query.c (working copy) @@ -433,6 +433,7 @@ int ret; struct ib_sge gather_list; struct ib_send_wr *bad_wr, wr = { + .next = NULL, .opcode = IB_WR_SEND, .sg_list = &gather_list, .num_sge = 1, From halr at voltaire.com Fri Jun 3 06:15:52 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 09:15:52 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050602232331.GD577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> Message-ID: <1117804409.4476.1074.camel@hal.voltaire.com> On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote: > I'm having intermittent problems with opensm.. It seems after a while > IPoIB stops working and if I restart opensm, Another side point: I'm not sure that in all cases IPoIB currently registers (its multicast) when the SM restarts. In the single SM case, this may require support for client reregistration. -- Hal From robert.j.woodruff at intel.com Fri Jun 3 08:44:57 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Fri, 3 Jun 2005 08:44:57 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e0050603043810f40f46@mail.gmail.com> Message-ID: Catlin Wrote, >That approach is certainly applicable for OpenIB as well. >The key is recognizing the need for a transition plan. >Customers have DAT Providers installed now, they >cannot synchronize getting new DAT Providers from >their suppliers with a new Linux release. This is >especially true since OpenIB does not currently >define a verbs interface that is suitable for iWARP >vendors to use. So dat_ia_openv() needs to still >support existing dat_registry logic and existing >DAT_PROVIDERs, otherwise it is breaking >existing code. Are you talking about user-mode DAT/DAPL or kernel mode. For user-mode, the DAT registry mechanism and the dynamic loading of shared libraries seems to work OK and I see no need to change it. For kernel-mode the method for discovering RDMA (DAPL) providers, or having them register with an RDMA mid-layer should be something that fits in better with Linux. my 2 cents, woody From caitlin.bestler at gmail.com Fri Jun 3 09:13:25 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Fri, 3 Jun 2005 09:13:25 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: References: <469958e0050603043810f40f46@mail.gmail.com> Message-ID: <469958e0050603091319941cbf@mail.gmail.com> On 6/3/05, Bob Woodruff wrote: > Catlin Wrote, > >That approach is certainly applicable for OpenIB as well. > >The key is recognizing the need for a transition plan. > >Customers have DAT Providers installed now, they > >cannot synchronize getting new DAT Providers from > >their suppliers with a new Linux release. This is > >especially true since OpenIB does not currently > >define a verbs interface that is suitable for iWARP > >vendors to use. So dat_ia_openv() needs to still > >support existing dat_registry logic and existing > >DAT_PROVIDERs, otherwise it is breaking > >existing code. > > Are you talking about user-mode DAT/DAPL or kernel mode. > For user-mode, the DAT registry mechanism and the dynamic > loading of shared libraries seems to work OK and I > see no need to change it. > > For kernel-mode the method for discovering RDMA (DAPL) providers, > or having them register with an RDMA mid-layer should be something > that fits in better with Linux. > > my 2 cents, > Customers are using insmod to load DAPL Providers today, and then using the registry to find them. That applies to both IB and iWARP providers. The need for the registry reduces with each step, but it doesn't instantly vanish and there should be a transition plan. Customers have a right to use insmod to load DAPL Providers, especially if the DAPL Provider is under a GPL license. But the fact that it is under a GPL license does not guarantee that it has been forwarded t kernel.org for inclusion in the mainline. The transition plan might be as simple as *not* exporting the dat_ia_openv symbol outside of the kernel until the internal routine is ready to support legacy Providers or there is no longer a need for Legacy Providers. The consumer could simply insmod the existing DAT Registry and the existing DAT Provider and make no-use of the in-kernel code. But that is the customer base that the kdAPL code on sourceforge is supporting today. It would be strange to take that code and break the applications that it was created to support. From halr at voltaire.com Fri Jun 3 09:14:45 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 12:14:45 -0400 Subject: [openib-general] Re: opensm: new segv on shutdown In-Reply-To: <1117659061.25491.12.camel@duffman> References: <1117659061.25491.12.camel@duffman> Message-ID: <1117815200.4493.184.camel@hal.voltaire.com> On Wed, 2005-06-01 at 16:51, Tom Duffy wrote: > I am putting together a network with a dumb IB switch, a couple of Linux > OpenIB boxes, a Solaris 10 box, a Solaris Nevada box, etc. I fired up > opensm on one of the Linux nodes, tried to plumb Solaris, no luck. I > then hit control-c on opensm and it crashed. Here is the messages and > then crash. > > OpenSM: Got signal 2 - exiting... > > -- OpenSM Exiting (in 3 seconds) -- > There are still 44 mads out. Forcing the exit of the OpenSM application... > Segmentation fault (core dumped) > > #0 stack_dump () at src/stack.c:72 > 72 if (!__builtin_frame_address(2)) > #0 stack_dump () at src/stack.c:72 > #1 0x00002aaaaacbd1a6 in handler (x=11) at src/stack.c:151 > #2 > #3 __match_notice_to_inf_rec (p_list_item=0x585e50, context=0x10) > at osm_inform.c:413 ^^^^^^^^^^^^ Context looks weird here... __match_notice_to_inf_rec( IN cl_list_item_t* const p_list_item, IN void* context ) { osm_infr_match_ctxt_t* p_infr_match = (osm_infr_match_ctxt_t *)context; ib_mad_notice_attr_t* p_ntc = p_infr_match->p_ntc; cl_list_t* p_infr_to_remove_list = p_infr_match->p_remove_infr_list; osm_infr_t* p_infr_rec = (osm_infr_t*)p_list_item; ib_inform_info_t *p_ii = &(p_infr_rec->inform_record.inform_info); cl_status_t status = CL_NOT_FOUND; osm_log_t *p_log = p_infr_rec->p_infr_rcv->p_log; <==================== segv here > #4 0x00002aaaaadc49ff in cl_qlist_apply_func (p_list=0x550680, > pfn_func=0x4085e0 <__match_notice_to_inf_rec>, context=0x43004f80) > at cl_list.c:354 > #5 0x0000000000408c57 in osm_report_notice (p_log=0x552498, p_subn=0x5502e0, > p_ntc=0x430050d0) at osm_inform.c:680 > #6 0x000000000042b472 in __osm_trap_rcv_process_request (p_rcv=0x5511e8, > p_madw=0x0) at osm_trap_rcv.c:646 > #7 0x000000000042b7fb in osm_trap_rcv_process (p_rcv=0x5511e8, > p_madw=0x566e60) at osm_trap_rcv.c:724 > #8 0x00000000004033bb in __cl_disp_worker (context=0x585e50) > at cl_dispatcher.c:105 > #9 0x00002aaaaadc928f in __cl_thread_pool_routine (context=0x552558) > at cl_threadpool.c:78 > #10 0x00002aaaaadc911e in __cl_thread_wrapper (arg=0x51) at cl_thread.c:61 > #11 0x00000036d28060aa in start_thread () from /lib64/tls/libpthread.so.0 > #12 0x00000036d19c53d3 in clone () from /lib64/tls/libc.so.6 > #13 0x0000000000000000 in ?? () > > Also, here are the last 500 lines of osm.log. > > Jun 01 13:37:00 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:00 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:01 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:01 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:01 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x566178, size = 256. > Jun 01 13:37:01 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03210, size = 256. > Jun 01 13:37:01 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:01 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x566160, p_mad = 0x2aaaaaf03240, size = 256. > Jun 01 13:37:01 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 99 QP1 MADs received. > Jun 01 13:37:01 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000007 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:01 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:02 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:02 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:02 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5660a8, size = 256. > Jun 01 13:37:02 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03350, size = 256. > Jun 01 13:37:02 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:02 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x566090, p_mad = 0x2aaaaaf03380, size = 256. > Jun 01 13:37:02 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 100 QP1 MADs received. > Jun 01 13:37:02 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000007 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:02 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:03 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:03 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:03 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565fd8, size = 256. > Jun 01 13:37:03 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03490, size = 256. > Jun 01 13:37:03 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:03 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565fc0, p_mad = 0x2aaaaaf034c0, size = 256. > Jun 01 13:37:03 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 101 QP1 MADs received. > Jun 01 13:37:03 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000007 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:03 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:04 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:04 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:04 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565f08, size = 256. > Jun 01 13:37:04 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf035d0, size = 256. > Jun 01 13:37:04 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:04 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565ef0, p_mad = 0x2aaaaaf03600, size = 256. > Jun 01 13:37:04 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 102 QP1 MADs received. > Jun 01 13:37:04 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000008 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:04 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:04 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:37:04 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:37:04 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:37:05 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:05 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:05 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565e38, size = 256. > Jun 01 13:37:05 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03710, size = 256. > Jun 01 13:37:05 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:05 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565e20, p_mad = 0x2aaaaaf03740, size = 256. > Jun 01 13:37:05 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 103 QP1 MADs received. > Jun 01 13:37:05 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000008 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:05 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:06 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:06 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:06 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565d68, size = 256. > Jun 01 13:37:06 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03850, size = 256. > Jun 01 13:37:06 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:06 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565d50, p_mad = 0x2aaaaaf03880, size = 256. > Jun 01 13:37:06 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 104 QP1 MADs received. > Jun 01 13:37:06 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000008 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:06 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:07 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:07 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:07 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565c98, size = 256. > Jun 01 13:37:07 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03990, size = 256. > Jun 01 13:37:07 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:07 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565c80, p_mad = 0x2aaaaaf039c0, size = 256. > Jun 01 13:37:07 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 105 QP1 MADs received. > Jun 01 13:37:07 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000009 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:07 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:09 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:09 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:09 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565bc8, size = 256. > Jun 01 13:37:09 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03ad0, size = 256. > Jun 01 13:37:09 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:09 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565bb0, p_mad = 0x2aaaaaf03b00, size = 256. > Jun 01 13:37:09 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 106 QP1 MADs received. > Jun 01 13:37:09 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000009 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:09 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:10 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:10 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:10 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565af8, size = 256. > Jun 01 13:37:10 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03c10, size = 256. > Jun 01 13:37:10 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:10 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565ae0, p_mad = 0x2aaaaaf03c40, size = 256. > Jun 01 13:37:10 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 107 QP1 MADs received. > Jun 01 13:37:10 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x976571200000009 > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:10 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:11 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:11 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:11 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565a28, size = 256. > Jun 01 13:37:11 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03d50, size = 256. > Jun 01 13:37:11 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:11 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565a10, p_mad = 0x2aaaaaf03d80, size = 256. > Jun 01 13:37:11 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 108 QP1 MADs received. > Jun 01 13:37:11 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x97657120000000a > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:11 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:12 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:12 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:12 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565958, size = 256. > Jun 01 13:37:12 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03e90, size = 256. > Jun 01 13:37:12 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:12 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565940, p_mad = 0x2aaaaaf03ec0, size = 256. > Jun 01 13:37:12 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 109 QP1 MADs received. > Jun 01 13:37:12 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x97657120000000a > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:12 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:13 [44808960] -> osm_mad_pool_get: [ > Jun 01 13:37:13 [44808960] -> osm_vendor_get: [ > Jun 01 13:37:13 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x565888, size = 256. > Jun 01 13:37:13 [44808960] -> osm_vendor_get: Acquired UMAD 0x2aaaaaf03fd0, size = 256. > Jun 01 13:37:13 [44808960] -> osm_vendor_get: ] > Jun 01 13:37:13 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x565870, p_mad = 0x2aaaaaf04000, size = 256. > Jun 01 13:37:13 [44808960] -> osm_mad_pool_get: ] > Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: [ > Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: 110 QP1 MADs received. > Jun 01 13:37:13 [44808960] -> SA MAD dump: > base_ver................0x1 > mgmt_class..............0x3 > class_ver...............0x2 > method..................0x12 (SubnAdmGetTable) > status..................0x0 > resv....................0x0 > trans_id................0x97657120000000a > attr_id.................0x38 (MCMemberRecord) > resv1...................0x0 > attr_mod................0xFFFFFFFF > rmpp_version............0x0 > rmpp_type...............0x0 > rmpp_flags..............0x0 > rmpp_status.............0x0 > seg_num.................0x0 > payload_len/new_win.....0x0 > sm_key..................0x0000000000000000 > attr_offset.............0x0 > resv2...................0x0 > comp_mask...............0x0000000000008081 > > > Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_process: [ > Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_process: Posting Dispatcher message OSM_MSG_MAD_MCMEMBER_RECORD. > Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_process: ] > Jun 01 13:37:13 [44808960] -> __osm_sa_mad_ctrl_rcv_callback: ] > Jun 01 13:37:14 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:37:14 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:37:14 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:37:24 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:37:24 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:37:24 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:37:34 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:37:34 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:37:34 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:37:44 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:37:44 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:37:44 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:37:54 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:37:54 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:37:54 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:38:04 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:38:04 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:38:04 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:38:14 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:38:14 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:38:14 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:38:24 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:38:24 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:38:24 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:38:34 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:38:34 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:38:34 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:38:44 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:38:44 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:38:44 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:38:54 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:38:54 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:38:54 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:39:04 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:39:04 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:39:04 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:39:14 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:39:14 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:39:14 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:39:24 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:39:24 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:39:24 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:39:34 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:39:34 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:39:34 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:39:44 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:39:44 [44007960] -> osm_state_mgr_process: Received signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_SWEEP_HEAVY_SELF. > Jun 01 13:39:44 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:39:54 [44007960] -> osm_state_mgr_process: [ > Jun 01 13:39:54 [44007960] -> osm_state_mgr_process: ] > Jun 01 13:39:56 [AAEE0B60] -> osm_vl15_shutdown: [ > Jun 01 13:39:56 [AAEE0B60] -> osm_vl15_shutdown: ] > Jun 01 13:39:56 [AAEE0B60] -> osm_vendor_set_sm: [ > Jun 01 13:39:56 [AAEE0B60] -> osm_vendor_set_sm: ] > Jun 01 13:39:56 [AAEE0B60] -> clear_madw: [ > Jun 01 13:39:56 [AAEE0B60] -> clear_madw: ] > Jun 01 13:39:56 [43005960] -> osm_physp_share_pkey: ] > Jun 01 13:39:56 [43005960] -> osm_port_share_pkey: ] > Jun 01 13:39:56 [43005960] -> __match_notice_to_inf_rec: Mismatch by Pkey > Jun 01 13:39:56 [43005960] -> __match_notice_to_inf_rec: ] > Jun 01 13:39:56 [AAEE0B60] -> osm_sm_destroy: [ > Jun 01 13:39:56 [44007960] -> __osm_sm_sweeper: Off schedule sweep signalled. > Jun 01 13:39:56 [44007960] -> __osm_sm_sweeper: ] It's past any of the OpenIB vendor layer code involved in the shutdown but appears to be handling an incoming trap and trying to see whether it matches any event subscriptions (SA InformInfo) based on the stack trace. Since completing the call to osm_vendor_delete unregisters all the MAD agents, it should not be receiving any traps after that so this must be internally queued. Perhaps we have a race here with osm_opensm_destroy which is invoked late in the OpenSM shutdown: osm_opensm.c::osm_opensm_destroy does: osm_subn_destroy( &p_osm->subn ); osm_sm_destroy( &p_osm->sm ); osm_sa_destroy( &p_osm->sa ); It is clearly in osm_sm_destroy when this occurs and another thread is handling the incoming trap. -- Hal From mshefty at ichips.intel.com Fri Jun 3 09:28:23 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 03 Jun 2005 09:28:23 -0700 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: <429FB451.3060602@linuxmachines.com> References: <1117539047.4476.199.camel@hal.voltaire.com> <78d18e20506011443175617b3@mail.gmail.com> <429FB451.3060602@linuxmachines.com> Message-ID: <42A08527.7000502@ichips.intel.com> Jeff Carr wrote: > On the client system: > # cat /sys/class/infiniband/mthca0/ports/1/lid > 0x2 > > On the server system: > # cat /sys/class/infiniband/mthca0/ports/1/lid > 0x3 > > modprobe ib_cmpost "slid=0x3" "dlid=0x2" "message_count=0x10" > "message_size=0x100" slid is Source LID (not server). dlid is Destination LID. It looks like your assignment is backwards. - Sean From robert.j.woodruff at intel.com Fri Jun 3 09:29:38 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Fri, 3 Jun 2005 09:29:38 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e0050603091319941cbf@mail.gmail.com> Message-ID: Catlin wrote, >But that is the customer base that the kdAPL code >on sourceforge is supporting today. It would be strange >to take that code and break the applications that it was >created to support. Understood. Customers that are currently using the kDAPL code from sourceforge, using the DAT registry mechanism will eventually need to transition to the kDAPL derivative and interfaces it provides if/when it gets accepted into kernel.org. Is the question, once the code is in kernel.org, will they be able to still insmod the older DAPL code ? I think the answer to this is yes. They can always not configure the kernel with the kernel.org drivers and load their own until they are ready to transition. Same is true for the rest of the openib.org stack. People do not have to configure the kernel to build these until they are ready to abandon their proprietary stacks. woody From eitan at mellanox.co.il Fri Jun 3 09:47:38 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Fri, 3 Jun 2005 19:47:38 +0300 Subject: [openib-general] opensm fails to bring up subnet.. Message-ID: <506C3D7B14CDD411A52C00025558DED607C30292@mtlex01.yok.mtl.com> Hi, Sorry for catching up with this late in the thread. (Thanks Hal for waking me up...) > > It appears that a node is not responding to a discovery packet (SM Get > NodeInfo (attrID 0x11)). It's direct route initial path (an array of > port numbers at the start of the next hop) is: > Initial path = [1][81][1] which means that starting at the node running > OpenSM, port 1 then port 129 then port 1. Is there a large switch in the > middle ? Can you send the output of ibnetdiscover ? If that is valid, > which HCA (port) is not responding (what is the GUID) ? [EZ] Normally all directed route dumps should start with: Initial path = [0][.... The first hop is reserved to 0 - so I wonde if the above text is a direct quote from the osm.log ? The fact you got there a [81] means that the packet should leave from port 81 ?? I have never seen a switch with more then 24 ports... > Unfortunately on such an error osm does not appear to give up (it > retries forever and is locked on such a node). This is obviously not > good. Also Troy if you are able to capture the entire log it might put some light on the issue of "OpenSM never give up" on such cases - which we want to resolve. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tubertini at emailaccount.com Fri Jun 3 11:04:05 2005 From: tubertini at emailaccount.com (Chelsea Goodrich) Date: Fri, 03 Jun 2005 21:04:05 +0300 Subject: [openib-general] Re-finance at todays low rate Message-ID: Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $400,000 for as little as $400 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.cr3am.com/signs.asp Best Regards, Myrna Arthur to be remov(ed: http://www.cr3am.com/deletion.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From halr at voltaire.com Fri Jun 3 10:06:36 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 13:06:36 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30292@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C30292@mtlex01.yok.mtl.com> Message-ID: <1117818396.4493.210.camel@hal.voltaire.com> On Fri, 2005-06-03 at 12:47, Eitan Zahavi wrote: > Hi, > Sorry for catching up with this late in the thread. (Thanks Hal for > waking me up...) > > > > It appears that a node is not responding to a discovery packet (SM > Get > > NodeInfo (attrID 0x11)). It's direct route initial path (an array of > > port numbers at the start of the next hop) is: > > Initial path = [1][81][1] which means that starting at the node > running > > OpenSM, port 1 then port 129 then port 1. Is there a large switch in > the > > middle ? Can you send the output of ibnetdiscover ? If that is > valid, > > which HCA (port) is not responding (what is the GUID) ? > [EZ] Normally all directed route dumps should start with: > Initial path = [0][.... > The first hop is reserved to 0 - so I wonde if the above text is a > direct quote from the osm.log ? > The fact you got there a [81] means that the packet should leave from > port 81 ?? 81 being hex not decimal but it is still > 24. > I have never seen a switch with more then 24 ports... I thought that looked suspect. I didn't think there were any switch chassis that were hiding their multiple internal switch chips. > > Unfortunately on such an error osm does not appear to give up (it > > retries forever and is locked on such a node). This is obviously not > > good. > Also Troy if you are able to capture the entire log it might put some > light on the issue of "OpenSM never give up" on such cases - which we > want to resolve. OpenIB has retries built into the MAD layer as well as the OpenIB vendor layer doing some retries for a send which is supposed to be matched with a response and this times out. [There is a potential issue here relative to the VL15 counting on error which came up on the list a short while ago so I am looking at possibly a change to this area of the vendor layer but have not concluded my analysis of this yet.] -- Hal From hozer at hozed.org Fri Jun 3 10:14:31 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Fri, 3 Jun 2005 12:14:31 -0500 Subject: [openib-general] cable test/error count utilities? In-Reply-To: <1117793673.4476.918.camel@hal.voltaire.com> References: <20050603002551.GF577@kalmia.hozed.org> <1117793673.4476.918.camel@hal.voltaire.com> Message-ID: <20050603171431.GG577@kalmia.hozed.org> On Fri, Jun 03, 2005 at 06:14:34AM -0400, Hal Rosenstock wrote: > On Thu, 2005-06-02 at 20:25, Troy Benjegerdes wrote: > > Some of my problems seem to be from intermittent cables.. > > > > Is there anything for OpenIB that can read error counters? > > Aside from pulling these from the driver via > /sys/class/infiniband/mthca0/ports/1/counters/, there is also perfquery > which displays the portcounters (which contains the error counters): > > Usage: perfquery [-d(ebug) -G(uid_addr) -a(ll_ports) -r(reset_after_read) -C ca_name -P hca_port -R(eset_only) -t timeout_ms -V(ersion) -h(elp)] [ [[port] [reset_mask]]] How can I query a port between two switches? [1] {00066a009800044d} DR path [0][1] -> new remote switch {00084e000000014a} portnum 0 lid 0x3-0x3 "Di vergeNet Meshlink 4X Switch" [4] {00084e000000014a} DR path [0][1] -> processing switch {00084e000000014a} portnum 0 lid 0x2-0x2 "Di vergeNet Meshlink 4X Switch" [5] {00084e000000014a} DR path [0][1][5] -> new remote switch {00084e0000000152} portnum 0 lid 0x1-0x1 "DivergeNet Meshlink 4X Switch" [6] {00084e0000000152} DR path [0][1][5] -> processing switch {00084e0000000152} portnum 0 lid 0x0-0x0 "DivergeNet Meshlink 4X Switch" [2] {00084e0000000152} DR path [0][1][5][2] -> new remote hca {00066a009800043c} portnum 1 lid 0x7-0x7 "MT23108 InfiniHost Mellanox Technologies" [1] {00066a009800043c} warn: [17598] _do_madrpc: send failed; Invalid argument warn: [17598] handle_port: Nodeinfo on DR path [0][1][5][3] port 3 failed, skipp ing port [4] {00084e0000000152} DR path [0][1][5][4] -> new remote hca {00066a0098000458} portnum 1 lid 0x9-0x9 "MT23108 InfiniHost Mellanox Technologies" [1] {00066a0098000458} [8] {00084e0000000152} DR path [0][1][5][8] -> new remote hca {00066a0098000444} portnum 1 lid 0x8-0x8 "MT23108 InfiniHost Mellanox Technologies" [1] {00066a0098000444} # # Topology file: generated on Fri Jun 3 12:13:51 2005 # switchguids=0x84e000000014a Switch 8 "S-00084e000000014a" # DivergeNet Meshlink 4X Switch port 0 l id 3 [5] "S-00084e0000000152"[6] [4] "H-00066a009800044d"[1] switchguids=0x84e0000000152 Switch 8 "S-00084e0000000152" # DivergeNet Meshlink 4X Switch port 0 l id 1 [8] "H-00066a0098000444"[1] [4] "H-00066a0098000458"[1] [2] "H-00066a009800043c"[1] [6] "S-00084e000000014a"[5] hcaguids=0x66a0098000444 Hca 2 "H-00066a0098000444" # MT23108 InfiniHost Mellanox Technologi es [1] "S-00084e0000000152"[8] # lid 8 lmc 0 hcaguids=0x66a0098000458 Hca 2 "H-00066a0098000458" # MT23108 InfiniHost Mellanox Technologi es [1] "S-00084e0000000152"[4] # lid 9 lmc 0 hcaguids=0x66a009800043c Hca 2 "H-00066a009800043c" # MT23108 InfiniHost Mellanox Technologi es [1] "S-00084e0000000152"[2] # lid 7 lmc 0 hcaguids=0x66a009800044d Hca 2 "H-00066a009800044d" # MT23108 InfiniHost Mellanox Technologi es [1] "S-00084e000000014a"[4] # lid 2 lmc 0 From hozer at hozed.org Fri Jun 3 10:17:21 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Fri, 3 Jun 2005 12:17:21 -0500 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <1117802224.4476.1025.camel@hal.voltaire.com> References: <20050602232331.GD577@kalmia.hozed.org> <1117802224.4476.1025.camel@hal.voltaire.com> Message-ID: <20050603171721.GH577@kalmia.hozed.org> On Fri, Jun 03, 2005 at 08:37:04AM -0400, Hal Rosenstock wrote: > On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote: > > I'm having intermittent problems with opensm.. It seems after a while > > IPoIB stops working > > Wonder if there is some relation to the two: intermittent IPoIB and lack > of response to SM query. > > > and if I restart opensm, > > How did you get around the ABI version mismatch issue ? > > > it starts spitting out > > errors. Do I have a misbehaving switch somewhere? > > It appears that a node is not responding to a discovery packet (SM Get > NodeInfo (attrID 0x11)). It's direct route initial path (an array of > port numbers at the start of the next hop) is: > Initial path = [1][81][1] which means that starting at the node running > OpenSM, port 1 then port 129 then port 1. Is there a large switch in the > middle ? Can you send the output of ibnetdiscover ? If that is valid, > which HCA (port) is not responding (what is the GUID) ? > > Unfortunately on such an error osm does not appear to give up (it > retries forever and is locked on such a node). This is obviously not > good. > > > ibnetdiscover seems to work fine. > > Are you sure it displays all HCA and switches and their ports ? I > wouldn't think it would respond to ibnetdiscover if it didn't respond to > osm. I'm running a subversion checkout as of yesterday, so that's how I got around the ABI version stuff. the [81] port indicator is definitely bogus. All I have are 8 port switches. I've also seen [0][0][0] path indicators.. are those allowed as well? From eitan at mellanox.co.il Fri Jun 3 10:19:57 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Fri, 3 Jun 2005 20:19:57 +0300 Subject: [openib-general] opensm fails to bring up subnet.. Message-ID: <506C3D7B14CDD411A52C00025558DED607C30294@mtlex01.yok.mtl.com> So Troy - will you be able to capture an osm.log and send us a tar.gz ? Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Troy Benjegerdes [mailto:hozer at hozed.org] > Sent: Friday, June 03, 2005 8:17 PM > To: Hal Rosenstock > Cc: openib-general at openib.org > Subject: Re: [openib-general] opensm fails to bring up subnet.. > > On Fri, Jun 03, 2005 at 08:37:04AM -0400, Hal Rosenstock wrote: > > On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote: > > > I'm having intermittent problems with opensm.. It seems after a while > > > IPoIB stops working > > > > Wonder if there is some relation to the two: intermittent IPoIB and lack > > of response to SM query. > > > > > and if I restart opensm, > > > > How did you get around the ABI version mismatch issue ? > > > > > it starts spitting out > > > errors. Do I have a misbehaving switch somewhere? > > > > It appears that a node is not responding to a discovery packet (SM Get > > NodeInfo (attrID 0x11)). It's direct route initial path (an array of > > port numbers at the start of the next hop) is: > > Initial path = [1][81][1] which means that starting at the node running > > OpenSM, port 1 then port 129 then port 1. Is there a large switch in the > > middle ? Can you send the output of ibnetdiscover ? If that is valid, > > which HCA (port) is not responding (what is the GUID) ? > > > > Unfortunately on such an error osm does not appear to give up (it > > retries forever and is locked on such a node). This is obviously not > > good. > > > > > ibnetdiscover seems to work fine. > > > > Are you sure it displays all HCA and switches and their ports ? I > > wouldn't think it would respond to ibnetdiscover if it didn't respond to > > osm. > > I'm running a subversion checkout as of yesterday, so that's how I > got around the ABI version stuff. > > the [81] port indicator is definitely bogus. All I have are 8 port > switches. I've also seen [0][0][0] path indicators.. are those allowed > as well? > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Fri Jun 3 10:22:10 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 03 Jun 2005 10:22:10 -0700 Subject: [openib-general] Re: [PATCH] sa_query: In send_mad, initialize wr.next to NULL In-Reply-To: <1117804241.4476.1069.camel@hal.voltaire.com> (Hal Rosenstock's message of "03 Jun 2005 09:10:41 -0400") References: <1117804241.4476.1069.camel@hal.voltaire.com> Message-ID: <52y89rwe2l.fsf@topspin.com> Hal> sa_query: In send_mad, initialize wr.next to NULL Does this solve a real problem? As far as I know, the C standard requires any pointer fields not explicitly named in a designated initializer for a structure to be initialized to NULL. - R. From hch at lst.de Fri Jun 3 10:27:27 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 3 Jun 2005 19:27:27 +0200 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e00506030430687795a8@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> <469958e00506030430687795a8@mail.gmail.com> Message-ID: <20050603172727.GA3888@lst.de> On Fri, Jun 03, 2005 at 04:30:51AM -0700, Caitlin Bestler wrote: > The appropriate plafce to add that would be as an unafilliated asynchronous > event reported via the async evd. No. The async evds are a horrible API that should go away not beeing added to. > The next question is to define what if any handshake is desired. > My guess that the consumer would acknowledge this by closing > the RNIC, and that there would be some sort of deadline for doing > so (much like a shutdown, you have so long to clean up, after that > -- well you were warned). There should be no handshake involved. After ->remove is called the midlayer must fail all operations. This is total standard behaviour you get when using the driver model. There's no reason the rdma code should be any different from other subsystems. From roland at topspin.com Fri Jun 3 10:27:45 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 03 Jun 2005 10:27:45 -0700 Subject: [openib-general] Re: [PATCH] sa_query: In send_mad, initialize wr.next to NULL In-Reply-To: <52y89rwe2l.fsf@topspin.com> (Roland Dreier's message of "Fri, 03 Jun 2005 10:22:10 -0700") References: <1117804241.4476.1069.camel@hal.voltaire.com> <52y89rwe2l.fsf@topspin.com> Message-ID: <52r7fjwdta.fsf@topspin.com> Roland> Does this solve a real problem? As far as I know, the C Roland> standard requires any pointer fields not explicitly named Roland> in a designated initializer for a structure to be Roland> initialized to NULL. In other words -- since my previous post was kind of incomprehensible -- I think wr.next is already being initialized to NULL and this patch doesn't do anything. - R. From hch at lst.de Fri Jun 3 10:29:43 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 3 Jun 2005 19:29:43 +0200 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e0050603043810f40f46@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C15@mtvex01.mtv.mtl.com> <469958e0050602095659d486ab@mail.gmail.com> <20050603080609.GA27550@lst.de> <469958e0050603043810f40f46@mail.gmail.com> Message-ID: <20050603172943.GB3888@lst.de> On Fri, Jun 03, 2005 at 04:38:15AM -0700, Caitlin Bestler wrote: > OpenRDMA discussed with the DAT Collaborative the idea > of subsuming the responsibilities of the DAT Registry, so that > the OpenRDMA directory could take the 'dat_xxx' calls > directly. When the device dependent logic used a dynamically > loaded verbs, those would be invoked directly. When the > device dependent logic required a full DAT Provider (because > it did not provide a recognized verbs interface) then the > existing dynamic DAT dispatch would be used. There's no way we're gonna add support for legacy APIs in a new subsystems. We have enough problems with transitions of existing legacy APIs. > That approach is certainly applicable for OpenIB as well. > The key is recognizing the need for a transition plan. > Customers have DAT Providers installed now, they > cannot synchronize getting new DAT Providers from > their suppliers with a new Linux release. Then you customers have lost, period. Their fault for using badly designed APIs from industry consortia without any taste and a horrible corporate culture. Maybe they'll learn one day. From hch at lst.de Fri Jun 3 10:31:45 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 3 Jun 2005 19:31:45 +0200 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e0050603091319941cbf@mail.gmail.com> References: <469958e0050603043810f40f46@mail.gmail.com> <469958e0050603091319941cbf@mail.gmail.com> Message-ID: <20050603173145.GC3888@lst.de> On Fri, Jun 03, 2005 at 09:13:25AM -0700, Caitlin Bestler wrote: > Customers are using insmod to load DAPL Providers today, and > then using the registry to find them. That applies to both IB and > iWARP providers. The need for the registry reduces with each > step, but it doesn't instantly vanish and there should be a > transition plan. Just because you can do something technically it doesn't mean we care. Caitlin, you have schown continuing ignorance of how linux kernel development works, cpould you please shut up or try to get informed? > The transition plan might be as simple as *not* exporting > the dat_ia_openv symbol outside of the kernel until the The dat_ia_openv API as-is doesn't have the slightest chance of getting anywhere near the mainline tree. From halr at voltaire.com Fri Jun 3 10:28:36 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 13:28:36 -0400 Subject: [openib-general] cable test/error count utilities? In-Reply-To: <20050603171431.GG577@kalmia.hozed.org> References: <20050603002551.GF577@kalmia.hozed.org> <1117793673.4476.918.camel@hal.voltaire.com> <20050603171431.GG577@kalmia.hozed.org> Message-ID: <1117819715.4493.238.camel@hal.voltaire.com> On Fri, 2005-06-03 at 13:14, Troy Benjegerdes wrote: > On Fri, Jun 03, 2005 at 06:14:34AM -0400, Hal Rosenstock wrote: > > Usage: perfquery [-d(ebug) -G(uid_addr) -a(ll_ports) -r(reset_after_read) -C ca_name -P hca_port -R(eset_only) -t timeout_ms -V(ersion) -h(elp)] [ [[port] [reset_mask]]] > > How can I query a port between two switches? > > [1] {00066a009800044d} > DR path [0][1] -> new remote switch {00084e000000014a} portnum 0 lid 0x3-0x3 "Di > vergeNet Meshlink 4X Switch" > [4] {00084e000000014a} > DR path [0][1] -> processing switch {00084e000000014a} portnum 0 lid 0x2-0x2 "Di > vergeNet Meshlink 4X Switch" > [5] {00084e000000014a} > DR path [0][1][5] -> new remote switch {00084e0000000152} portnum 0 lid 0x1-0x1 > "DivergeNet Meshlink 4X Switch" > [6] {00084e0000000152} > DR path [0][1][5] -> processing switch {00084e0000000152} portnum 0 lid 0x0-0x0 > "DivergeNet Meshlink 4X Switch" > [2] {00084e0000000152} > DR path [0][1][5][2] -> new remote hca {00066a009800043c} portnum 1 lid 0x7-0x7 > "MT23108 InfiniHost Mellanox Technologies" > [1] {00066a009800043c} > warn: [17598] _do_madrpc: send failed; Invalid argument > warn: [17598] handle_port: Nodeinfo on DR path [0][1][5][3] port 3 failed, skipp > ing port > [4] {00084e0000000152} > DR path [0][1][5][4] -> new remote hca {00066a0098000458} portnum 1 lid 0x9-0x9 > "MT23108 InfiniHost Mellanox Technologies" > [1] {00066a0098000458} > [8] {00084e0000000152} > DR path [0][1][5][8] -> new remote hca {00066a0098000444} portnum 1 lid 0x8-0x8 > "MT23108 InfiniHost Mellanox Technologies" > [1] {00066a0098000444} > # > # Topology file: generated on Fri Jun 3 12:13:51 2005 > # > > switchguids=0x84e000000014a > Switch 8 "S-00084e000000014a" # DivergeNet Meshlink 4X Switch port 0 l > id 3 > [5] "S-00084e0000000152"[6] > [4] "H-00066a009800044d"[1] > > switchguids=0x84e0000000152 > Switch 8 "S-00084e0000000152" # DivergeNet Meshlink 4X Switch port 0 l > id 1 > [8] "H-00066a0098000444"[1] > [4] "H-00066a0098000458"[1] > [2] "H-00066a009800043c"[1] > [6] "S-00084e000000014a"[5] > > hcaguids=0x66a0098000444 > Hca 2 "H-00066a0098000444" # MT23108 InfiniHost Mellanox Technologi > es > [1] "S-00084e0000000152"[8] # lid 8 lmc 0 > > hcaguids=0x66a0098000458 > Hca 2 "H-00066a0098000458" # MT23108 InfiniHost Mellanox Technologi > es > [1] "S-00084e0000000152"[4] # lid 9 lmc 0 > > hcaguids=0x66a009800043c > Hca 2 "H-00066a009800043c" # MT23108 InfiniHost Mellanox Technologi > es > [1] "S-00084e0000000152"[2] # lid 7 lmc 0 > > hcaguids=0x66a009800044d > Hca 2 "H-00066a009800044d" # MT23108 InfiniHost Mellanox Technologi > es > [1] "S-00084e000000014a"[4] # lid 2 lmc 0 You need to give perfquery the port 0 LID of the switch and the physical port number. One of the switches is LID 3 and the other is LID 1. So its something like /usr/local/ib/bin/perfquery 3 Here's one of my switch physical ports port counters: # Port counters: Lid 0x2 port 2 PortSelect:......................2 CounterSelect:...................0x0000 SymbolErrors:....................102 LinkRecovers:....................255 LinkDowned:......................3 RcvErrors:.......................1 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................0 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............15 VL15Dropped:.....................0 XmtBytes:........................624996 RcvBytes:........................112635 XmtPkts:.........................8693 RcvBytes:........................1578 You can do it from both sides if you know the physical port for each switch for a given link. You can see this in ibnetdiscover: Switch 8 "S-00084e000000014a" # DivergeNet Meshlink 4X Switch port 0 lid 3 [5] "S-00084e0000000152"[6] port 5 on LID 3 is connected to port 6 on GUID ending in 153 (LID 1) and Switch 8 "S-00084e0000000152" # DivergeNet Meshlink 4X Switch port 0 lid 1 [6] "S-00084e000000014a"[5] port 6 on LID 1 is connected to port 5 on GUID ending in 14a (other switch) (LID 3) So you have 1 trunk between the 2 switches with 3 HCA ports off 1 and 1 HCA port off the other. -- Hal From halr at voltaire.com Fri Jun 3 10:34:40 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 13:34:40 -0400 Subject: [openib-general] Re: [PATCH] sa_query: In send_mad, initialize wr.next to NULL In-Reply-To: <52y89rwe2l.fsf@topspin.com> References: <1117804241.4476.1069.camel@hal.voltaire.com> <52y89rwe2l.fsf@topspin.com> Message-ID: <1117819777.4493.242.camel@hal.voltaire.com> On Fri, 2005-06-03 at 13:22, Roland Dreier wrote: > Hal> sa_query: In send_mad, initialize wr.next to NULL > > Does this solve a real problem? As far as I know, the C standard > requires any pointer fields not explicitly named in a designated > initializer for a structure to be initialized to NULL. If that is the case, this isn't needed. -- Hal From halr at voltaire.com Fri Jun 3 10:46:29 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 13:46:29 -0400 Subject: [openib-general] [PATCH] OpenSM: osm_sminfo_rcv.c: Eliminate some redundant checks Message-ID: <1117820628.4493.267.camel@hal.voltaire.com> osm_sminfo_rcv.c: Eliminate some redundant checks Signed-off-by: Hal Rosenstock Index: osm_sminfo_rcv.c =================================================================== --- osm_sminfo_rcv.c (revision 2538) +++ osm_sminfo_rcv.c (working copy) @@ -160,14 +160,7 @@ p_smp = osm_madw_get_smp_ptr( p_madw ); - if( p_smp->method != IB_MAD_METHOD_GET ) - { - osm_log( p_rcv->p_log, OSM_LOG_ERROR, - "__osm_sminfo_rcv_process_get_request: ERR 2F01: " - "Unsupported method 0x%X.\n", - p_smp->method ); - goto Exit; - } + CL_ASSERT( p_smp->method == IB_MAD_METHOD_GET ); p_smi->guid = p_rcv->p_subn->sm_port_guid; p_smi->act_count = cl_hton32( p_rcv->p_stats->qp0_mads_sent ); @@ -272,21 +265,13 @@ */ cl_memclr( payload, sizeof( payload ) ); - /* get the lock */ + /* get the lock */ CL_PLOCK_EXCL_ACQUIRE( p_rcv->p_lock ); p_smp = osm_madw_get_smp_ptr( p_madw ); p_rcv_smi = ib_smp_get_payload_ptr( p_smp ); - if( p_smp->method != IB_MAD_METHOD_SET ) - { - osm_log( p_rcv->p_log, OSM_LOG_ERROR, - "__osm_sminfo_rcv_process_set_request: ERR 2F03: " - "Unsupported method 0x%X.\n", - p_smp->method ); - CL_PLOCK_RELEASE( p_rcv->p_lock ); - goto Exit; - } + CL_ASSERT( p_smp->method == IB_MAD_METHOD_SET ); p_smi->guid = p_rcv->p_subn->sm_port_guid; p_smi->act_count = cl_hton32( p_rcv->p_stats->qp0_mads_sent ); From halr at voltaire.com Fri Jun 3 10:52:31 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 13:52:31 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050602232331.GD577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> Message-ID: <1117821141.4493.278.camel@hal.voltaire.com> Hi Troy, On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote: > I'm having intermittent problems with opensm.. It seems after a while > IPoIB stops working and if I restart opensm, it starts spitting out > errors. Please try the following workaround and let me know if this makes things better. -- Hal Index: libvendor/osm_vendor_ibumad.c =================================================================== --- libvendor/osm_vendor_ibumad.c (revision 2520) +++ libvendor/osm_vendor_ibumad.c (working copy) @@ -402,7 +402,7 @@ p_vend->p_log = p_log; p_vend->timeout = timeout; - p_vend->max_retries = OSM_DEFAULT_RETRY_COUNT; + p_vend->max_retries = 1; p_vend->umad_port_id = -1; p_vend->issmfd = -1; From halr at voltaire.com Fri Jun 3 10:59:42 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 13:59:42 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050603171721.GH577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> <1117802224.4476.1025.camel@hal.voltaire.com> <20050603171721.GH577@kalmia.hozed.org> Message-ID: <1117821579.4493.287.camel@hal.voltaire.com> On Fri, 2005-06-03 at 13:17, Troy Benjegerdes wrote: > I've also seen [0][0][0] path indicators.. are those allowed > as well? That is bogus too. Are all the bogus initial paths after a umad_send error (message like "umad_receiver: send completed with error" in the osm.log) ? Thanks. -- Hal From bjordan.ics at gmail.com Fri Jun 3 11:35:41 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Fri, 3 Jun 2005 14:35:41 -0400 Subject: [openib-general] [PATCH] ib_ucm: remove devfs usage In-Reply-To: <20050602134028.E15700@topspin.com> References: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> <20050601173638.A14794@topspin.com> <78d18e2050602124335d90d2d@mail.gmail.com> <20050602134028.E15700@topspin.com> Message-ID: <78d18e20506031135632cf2ba@mail.gmail.com> On 6/2/05, Libor Michalek wrote: > On Thu, Jun 02, 2005 at 03:43:47PM -0400, William Jordan wrote: > > > > The udev support/naming works fine. Regarding naming, the ib_uverbs > > and ib_umad drivers only put port specific devices in the /dev/infiniband > > directory. They also create non-port specific device nodes /dev/infiniband_mad > > and /dev/infiniband_verbs. Would it be more consistent to create a > > /dev/infiniband_cm instead of /dev/infiniband/cm? > > You are right, I hadn't noticed the distinction before, looks like > /dev/infiniband_cm is the way to go. > > -Libor > Actually, I think I'm wrong. On my system, /dev/infiniband_mad is the same device as /dev/infiniband/umad1, and /dev/infiniband_verbs is the same device as /dev/infiniband/uverbs0. Should the device nodes /dev/infiniband_mad and /dev/infiniband_verbs exist at all? Are they artifacts of the way the classes work, or a setup problem on my system? Do other people have these nodes? Unless I've gotten it wrong (again), I believe /dev/infiniband/ucm (as it exists) is an appropriate device name for the userlevel cm. Sorry for the confusion. -- Bill Jordan SilverStorm Technologies From hozer at hozed.org Fri Jun 3 11:38:32 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Fri, 3 Jun 2005 13:38:32 -0500 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <1117821579.4493.287.camel@hal.voltaire.com> References: <20050602232331.GD577@kalmia.hozed.org> <1117802224.4476.1025.camel@hal.voltaire.com> <20050603171721.GH577@kalmia.hozed.org> <1117821579.4493.287.camel@hal.voltaire.com> Message-ID: <20050603183832.GJ577@kalmia.hozed.org> On Fri, Jun 03, 2005 at 01:59:42PM -0400, Hal Rosenstock wrote: > On Fri, 2005-06-03 at 13:17, Troy Benjegerdes wrote: > > I've also seen [0][0][0] path indicators.. are those allowed > > as well? > > That is bogus too. Are all the bogus initial paths after a umad_send > error (message like "umad_receiver: send completed with error" in the > osm.log) ? Thanks. I can't say for sure, but I think so. One of the problems turned out to be a crashed Xserve G5, which seemed to have the link up, but the driver was obviously not responding to MAD packets. From roland at topspin.com Fri Jun 3 11:40:35 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 03 Jun 2005 11:40:35 -0700 Subject: [openib-general] [PATCH] ib_ucm: remove devfs usage In-Reply-To: <78d18e20506031135632cf2ba@mail.gmail.com> (William Jordan's message of "Fri, 3 Jun 2005 14:35:41 -0400") References: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> <20050601173638.A14794@topspin.com> <78d18e2050602124335d90d2d@mail.gmail.com> <20050602134028.E15700@topspin.com> <78d18e20506031135632cf2ba@mail.gmail.com> Message-ID: <527jhbwafw.fsf@topspin.com> William> Should the device nodes /dev/infiniband_mad and William> /dev/infiniband_verbs exist at all? Are they artifacts of William> the way the classes work, or a setup problem on my William> system? Do other people have these nodes? I don't get those nodes -- I seem to recall a udev bug (fixed a few versions back) that created spurious device nodes based on sysfs class names. - R. From caitlin.bestler at gmail.com Fri Jun 3 11:58:50 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Fri, 3 Jun 2005 11:58:50 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <20050603172727.GA3888@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> <469958e00506030430687795a8@mail.gmail.com> <20050603172727.GA3888@lst.de> Message-ID: <469958e0050603115887db8fa@mail.gmail.com> On 6/3/05, Christoph Hellwig wrote: > > > The next question is to define what if any handshake is desired. > > My guess that the consumer would acknowledge this by closing > > the RNIC, and that there would be some sort of deadline for doing > > so (much like a shutdown, you have so long to clean up, after that > > -- well you were warned). > > There should be no handshake involved. After ->remove is called the > midlayer must fail all operations. This is total standard behaviour > you get when using the driver model. There's no reason the rdma > code should be any different from other subsystems. > > Since you haven't bothered to learn much about RDMA, let me explain that unlike typical devices an RDMA device can be pefrforming IO even when the OS is not aware of it. That is why the Quadrics extensions for zero placement are not applicable to an RDMA device. That extension, for example, only remaps memory when the IO device is not actively doing a transfer. Something that the kernel does not know with an RDMA device. In any event, "after ->remove is called" *is* a handshake. The question is whether the tolerances on how long the call can take are acceptable to this type of device. From hozer at hozed.org Fri Jun 3 12:03:50 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Fri, 3 Jun 2005 14:03:50 -0500 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <1117821141.4493.278.camel@hal.voltaire.com> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> Message-ID: <20050603190350.GK577@kalmia.hozed.org> On Fri, Jun 03, 2005 at 01:52:31PM -0400, Hal Rosenstock wrote: > Hi Troy, > > On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote: > > I'm having intermittent problems with opensm.. It seems after a while > > IPoIB stops working and if I restart opensm, it starts spitting out > > errors. > > Please try the following workaround and let me know if this makes things > better. > > -- Hal > > Index: libvendor/osm_vendor_ibumad.c > =================================================================== > --- libvendor/osm_vendor_ibumad.c (revision 2520) > +++ libvendor/osm_vendor_ibumad.c (working copy) > @@ -402,7 +402,7 @@ > > p_vend->p_log = p_log; > p_vend->timeout = timeout; > - p_vend->max_retries = OSM_DEFAULT_RETRY_COUNT; > + p_vend->max_retries = 1; > > p_vend->umad_port_id = -1; > p_vend->issmfd = -1; No, it doesn't seem to help. To get anything to work at all, I seem to need to reload all the IB modules on every maching I want to use ipoib on. There have been two times now I've been able to see about 4 ping packets, and then one of the arp entries seems to go away. (On the sm machine, also the machine I am trying to ping) 10.40.5.213 (incomplete) ib0 (on another machine, trying to ping from..) 10.40.137.12 ether 00:00:04:04:FE:80:00C ib0 From halr at voltaire.com Fri Jun 3 11:57:47 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 14:57:47 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050603183832.GJ577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> <1117802224.4476.1025.camel@hal.voltaire.com> <20050603171721.GH577@kalmia.hozed.org> <1117821579.4493.287.camel@hal.voltaire.com> <20050603183832.GJ577@kalmia.hozed.org> Message-ID: <1117825066.4493.350.camel@hal.voltaire.com> On Fri, 2005-06-03 at 14:38, Troy Benjegerdes wrote: > One of the problems turned out to be a crashed Xserve G5, which seemed > to have the link up, but the driver was obviously not responding to MAD > packets. Yes, that is exactly the scenario than the OpenIB OpenSM vendor layer is not handling properly. This may also be the same thing Ron saw some time back... -- Hal From halr at voltaire.com Fri Jun 3 12:05:14 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 15:05:14 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050603190350.GK577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> <20050603190350.GK577@kalmia.hozed.org> Message-ID: <1117825513.4493.359.camel@hal.voltaire.com> On Fri, 2005-06-03 at 15:03, Troy Benjegerdes wrote: > On Fri, Jun 03, 2005 at 01:52:31PM -0400, Hal Rosenstock wrote: > > Hi Troy, > > > > On Thu, 2005-06-02 at 19:23, Troy Benjegerdes wrote: > > > I'm having intermittent problems with opensm.. It seems after a while > > > IPoIB stops working and if I restart opensm, it starts spitting out > > > errors. > > > > Please try the following workaround and let me know if this makes things > > better. > > > > -- Hal > > > > Index: libvendor/osm_vendor_ibumad.c > > =================================================================== > > --- libvendor/osm_vendor_ibumad.c (revision 2520) > > +++ libvendor/osm_vendor_ibumad.c (working copy) > > @@ -402,7 +402,7 @@ > > > > p_vend->p_log = p_log; > > p_vend->timeout = timeout; > > - p_vend->max_retries = OSM_DEFAULT_RETRY_COUNT; > > + p_vend->max_retries = 1; > > > > p_vend->umad_port_id = -1; > > p_vend->issmfd = -1; > > No, it doesn't seem to help. To get anything to work at all, I seem to > need to reload all the IB modules on every maching I want to use ipoib > on. > > There have been two times now I've been able to see about 4 ping > packets, and then one of the arp entries seems to go away. > > (On the sm machine, also the machine I am trying to ping) > 10.40.5.213 (incomplete) ib0 > > (on another machine, trying to ping from..) > 10.40.137.12 ether 00:00:04:04:FE:80:00C ib0 That may be another issue. Are all your links active and the OpenSM appears to be behaving better now ? -- Hal From mshefty at ichips.intel.com Fri Jun 3 12:13:07 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 03 Jun 2005 12:13:07 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <001401c56083$b27dfa70$9b5aa8c0@infiniconsys.com> References: <001401c56083$b27dfa70$9b5aa8c0@infiniconsys.com> Message-ID: <42A0ABC3.30203@ichips.intel.com> Fab Tillier wrote: > Ok, so this question is from a noob, but here goes anyway. Why can't IPoIB > advertise a larger MTU than the UD MTU, and then just fragment large IP > packets up if they need to go over the IB UD transport? Is there any reason > this couldn't work? If it does, it allows IPoIB to expose a single MTU to > the OS, and take care of the rest under the covers. > > Just a thought. I don't remember seeing a response to this. Something like this could work. I guess one disadvantage is that it can be less efficient if you lose a lot of packets. TCP would resend an entire MTU (as seen by TCP) of data if only a single IB packet were lost. - Sean From tduffy at sun.com Fri Jun 3 12:21:30 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 03 Jun 2005 12:21:30 -0700 Subject: [openib-general] ipoib: bringing ib0 down kills ib0.8001? Message-ID: <1117826490.28889.14.camel@duffman> Should it be the case that bringing down ib0 should kill off the other pkey devices: [root at flopteron2 ~]# netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.6.98.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0.8001 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 10.6.98.1 0.0.0.0 UG 0 0 0 eth0 [root at flopteron2 ~]# ifconfig ib0 down ib_mthca 0000:06:00.0: modify QP 7 returned status 0a. ib0.8001: Failed to modify QP to ERROR state ib_mthca 0000:06:00.0: modify QP 11 returned status 0a. ib0.8001: Failed to modify QP to RESET state In any event, seems like these errors shouldn't happen... -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Fri Jun 3 12:22:59 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 03 Jun 2005 12:22:59 -0700 Subject: [openib-general] ipoib: bringing ib0 down kills ib0.8001? In-Reply-To: <1117826490.28889.14.camel@duffman> (Tom Duffy's message of "Fri, 03 Jun 2005 12:21:30 -0700") References: <1117826490.28889.14.camel@duffman> Message-ID: <52k6lbutws.fsf@topspin.com> Tom> Should it be the case that bringing down ib0 should kill off Tom> the other pkey devices: Looks like a bug -- I'll take a look. - R. From hozer at hozed.org Fri Jun 3 12:24:38 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Fri, 3 Jun 2005 14:24:38 -0500 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <1117825513.4493.359.camel@hal.voltaire.com> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> <20050603190350.GK577@kalmia.hozed.org> <1117825513.4493.359.camel@hal.voltaire.com> Message-ID: <20050603192438.GL577@kalmia.hozed.org> > > No, it doesn't seem to help. To get anything to work at all, I seem to > > need to reload all the IB modules on every maching I want to use ipoib > > on. > > > > There have been two times now I've been able to see about 4 ping > > packets, and then one of the arp entries seems to go away. > > > > (On the sm machine, also the machine I am trying to ping) > > 10.40.5.213 (incomplete) ib0 > > > > (on another machine, trying to ping from..) > > 10.40.137.12 ether 00:00:04:04:FE:80:00C ib0 > > That may be another issue. Are all your links active and the OpenSM > appears to be behaving better now ? > The links all seem better.. opensm definitely started working better when the hung machine was rebooted. I can't tell anything conclusive from the patch though. How do I go about debugging the multicast arp/ipoib stuff? From halr at voltaire.com Fri Jun 3 12:33:48 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 15:33:48 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050603192438.GL577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> <20050603190350.GK577@kalmia.hozed.org> <1117825513.4493.359.camel@hal.voltaire.com> <20050603192438.GL577@kalmia.hozed.org> Message-ID: <1117827228.4493.379.camel@hal.voltaire.com> On Fri, 2005-06-03 at 15:24, Troy Benjegerdes wrote: > I can't tell anything conclusive from the patch though. OK. So you can't recreate the original problem ? > How do I go about debugging the multicast arp/ipoib stuff? Do you have any additional known good cables to try ? -- Hal From tduffy at sun.com Fri Jun 3 12:39:06 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 03 Jun 2005 12:39:06 -0700 Subject: [openib-general] [OOPS]: unloaded ib_mthca out from under opensm causes oops Message-ID: <1117827546.16881.2.camel@duffman> I had opensm running on this node, before stopping opensm, unloaded ib_mthca, caused oops in kernel. [root at flopteron2 ~]# rmmod ib_mthca [root at flopteron2 ~]# general protection fault: 0000 [1] SMP CPU 1 Modules linked in: ib_ipoib ib_sa ib_umad ib_mad ib_core nfs lockd md5 ipv6 parport_pc lp parport autofs4 sunrpc pcmcia yenta_socket rsrc_nonstatic pcmcia_core ext3 jbd video container button battery ac uhci_hcd ehci_hcd hw_random i2c_i801 i2c_core e1000 floppy dm_snapshot dm_zero dm_mirror xfs exportfs dm_mod mptscsih mptbase sd_mod scsi_mod Pid: 8544, comm: opensm Not tainted 2.6.12-rc5openib RIP: 0010:[] {:ib_core:ib_create_ah+0} RSP: 0018:ffff8100319e5e70 EFLAGS: 00010246 RAX: ffff81002fe97080 RBX: ffffffffffffffea RCX: ffff81001deb6088 RDX: ffff8100021e6d20 RSI: ffff8100319e5ea8 RDI: 6b6b6b6b6b6b6b6b RBP: 0000000000000130 R08: 0000000000000000 R09: 0100000001018101 R10: 7f12000000000000 R11: 0000000000000000 R12: ffff81001deb6040 R13: 0000000000591780 R14: ffff81002e36d4b0 R15: 0000000000000100 FS: 0000000043806960(0063) GS:ffffffff804e8100(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fffff82b106 CR3: 000000002d408000 CR4: 00000000000006e0 Process opensm (pid: 8544, threadinfo ffff8100319e4000, task ffff81003b81e1b0) Stack: ffffffff88131f3f 0000000000000000 000000000028195d 0000000000000360 ffff8100ffffffff ffff8100021e6d20 ffff81001deb6078 0000000000000000 0000000000000000 0000000000000000 Call Trace:{:ib_umad:ib_umad_write+367} {vfs_write+218} {sys_write+83} {system_call+126} Code: 48 8b 07 53 48 89 fb ff 90 20 01 00 00 48 3d 18 fc ff ff 48 RIP {:ib_core:ib_create_ah+0} RSP -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mshefty at ichips.intel.com Fri Jun 3 12:50:44 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 03 Jun 2005 12:50:44 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <4294B375.8070808@ichips.intel.com> References: <1116530995.980.9.camel@duffman> <4294B375.8070808@ichips.intel.com> Message-ID: <42A0B494.9060700@ichips.intel.com> I made several modifications to the MAD layer to assist with debugging this, and after a multitude of test runs I was able to see the output shown below. Basically, a send work request is completing, but the MAD associated with the request has been freed/corrupted. It's likely that the errors for MADs "not at the head of send queue" are a result of the corrupted MAD. - Sean cmpost: starting client cmpost: connecting cmpost: connect time: 4072000 us cmpost: waiting to disconnect cmpost: test complete send_comp_handler - completion for MAD not on send queue QP: 1 send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not on send queue QP: 1 send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not on send queue QP: 1 send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue send_comp_handler - completion for MAD not at head of send queue QP: 1 found later in queue Slab corruption: start=dfc9ddd8, len=256 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [](worker_thread+0x1a8/0x230) 000: 60 d3 c9 df 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Prev obj: start=dfc9dccc, len=256 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [](ib_mad_send_done_handler+0x2b/0x120 [ib_mad]) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=dfc9dee4, len=256 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [](ib_mad_send_done_handler+0x2b/0x120 [ib_mad]) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b From hozer at hozed.org Fri Jun 3 12:58:16 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Fri, 3 Jun 2005 14:58:16 -0500 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <1117827228.4493.379.camel@hal.voltaire.com> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> <20050603190350.GK577@kalmia.hozed.org> <1117825513.4493.359.camel@hal.voltaire.com> <20050603192438.GL577@kalmia.hozed.org> <1117827228.4493.379.camel@hal.voltaire.com> Message-ID: <20050603195816.GM577@kalmia.hozed.org> On Fri, Jun 03, 2005 at 03:33:48PM -0400, Hal Rosenstock wrote: > On Fri, 2005-06-03 at 15:24, Troy Benjegerdes wrote: > > I can't tell anything conclusive from the patch though. > > OK. So you can't recreate the original problem ? > > > How do I go about debugging the multicast arp/ipoib stuff? > > Do you have any additional known good cables to try ? I have several cables I *could* try, but I have no idea which ones are good, or what ports are getting errors. Some lids seem to respond to 'perfquery', but I haven't been able to get any of my 8 port switches to respond. Is this something only in the 24 port switch chips? Also, I have two machines in a state right now where they are printing out: kernel: unregister_netdevice: waiting for ib0 to become free. Usage count = 1 Finally.. perfquery has two 'RcvBytes' fields.. should the second one be 'RcvPkts' ? From halr at voltaire.com Fri Jun 3 13:08:05 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 16:08:05 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050603195816.GM577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> <20050603190350.GK577@kalmia.hozed.org> <1117825513.4493.359.camel@hal.voltaire.com> <20050603192438.GL577@kalmia.hozed.org> <1117827228.4493.379.camel@hal.voltaire.com> <20050603195816.GM577@kalmia.hozed.org> Message-ID: <1117829284.4493.403.camel@hal.voltaire.com> On Fri, 2005-06-03 at 15:58, Troy Benjegerdes wrote: > > Do you have any additional known good cables to try ? > > I have several cables I *could* try, but I have no idea which ones are > good, or what ports are getting errors. > > Some lids seem to respond to 'perfquery', but I haven't been able to get > any of my 8 port switches to respond. Is this something only in the 24 > port switch chips? No. It may be the following: Although PMA is a required IB subagent per the IB spec, it might not be available in all vendor products. That's one possibility. Do all HCAs respond but not your switches ? > Also, I have two machines in a state right now where they are printing > out: > > kernel: unregister_netdevice: waiting for ib0 to become free. Usage > count = 1 That's an IPoIB issue... What was the sequence of events which caused this ? > Finally.. perfquery has two 'RcvBytes' fields.. should the second one be > 'RcvPkts' ? Yes, good catch. Thanks. -- Hal From halr at voltaire.com Fri Jun 3 13:26:48 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 16:26:48 -0400 Subject: [openib-general] [OOPS]: unloaded ib_mthca out from under opensm causes oops In-Reply-To: <1117827546.16881.2.camel@duffman> References: <1117827546.16881.2.camel@duffman> Message-ID: <1117830373.4493.417.camel@hal.voltaire.com> On Fri, 2005-06-03 at 15:39, Tom Duffy wrote: > I had opensm running on this node, before stopping opensm, unloaded > ib_mthca, caused oops in kernel. > > [root at flopteron2 ~]# rmmod ib_mthca > [root at flopteron2 ~]# general protection fault: 0000 [1] SMP > CPU 1 > Modules linked in: ib_ipoib ib_sa ib_umad ib_mad ib_core nfs lockd md5 ipv6 parport_pc lp parport autofs4 sunrpc pcmcia yenta_socket rsrc_nonstatic pcmcia_core ext3 jbd video container button battery ac uhci_hcd ehci_hcd hw_random i2c_i801 i2c_core e1000 floppy dm_snapshot dm_zero dm_mirror xfs exportfs dm_mod mptscsih mptbase sd_mod scsi_mod > Pid: 8544, comm: opensm Not tainted 2.6.12-rc5openib > RIP: 0010:[] {:ib_core:ib_create_ah+0} > RSP: 0018:ffff8100319e5e70 EFLAGS: 00010246 > RAX: ffff81002fe97080 RBX: ffffffffffffffea RCX: ffff81001deb6088 > RDX: ffff8100021e6d20 RSI: ffff8100319e5ea8 RDI: 6b6b6b6b6b6b6b6b > RBP: 0000000000000130 R08: 0000000000000000 R09: 0100000001018101 > R10: 7f12000000000000 R11: 0000000000000000 R12: ffff81001deb6040 > R13: 0000000000591780 R14: ffff81002e36d4b0 R15: 0000000000000100 > FS: 0000000043806960(0063) GS:ffffffff804e8100(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00007fffff82b106 CR3: 000000002d408000 CR4: 00000000000006e0 > Process opensm (pid: 8544, threadinfo ffff8100319e4000, task ffff81003b81e1b0) > Stack: ffffffff88131f3f 0000000000000000 000000000028195d 0000000000000360 > ffff8100ffffffff ffff8100021e6d20 ffff81001deb6078 0000000000000000 > 0000000000000000 0000000000000000 > Call Trace:{:ib_umad:ib_umad_write+367} {vfs_write+218} > {sys_write+83} {system_call+126} > > > Code: 48 8b 07 53 48 89 fb ff 90 20 01 00 00 48 3d 18 fc ff ff 48 > RIP {:ib_core:ib_create_ah+0} RSP user_mad (as well as ucm) needs to indicate error on any read/write file operations related to a driver which has been removed. Maybe more than this needs to be done on a driver removal with open fd(s). -- Hal From hozer at hozed.org Fri Jun 3 13:33:14 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Fri, 3 Jun 2005 15:33:14 -0500 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <1117829284.4493.403.camel@hal.voltaire.com> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> <20050603190350.GK577@kalmia.hozed.org> <1117825513.4493.359.camel@hal.voltaire.com> <20050603192438.GL577@kalmia.hozed.org> <1117827228.4493.379.camel@hal.voltaire.com> <20050603195816.GM577@kalmia.hozed.org> <1117829284.4493.403.camel@hal.voltaire.com> Message-ID: <20050603203314.GN577@kalmia.hozed.org> On Fri, Jun 03, 2005 at 04:08:05PM -0400, Hal Rosenstock wrote: > On Fri, 2005-06-03 at 15:58, Troy Benjegerdes wrote: > > > Do you have any additional known good cables to try ? > > > > I have several cables I *could* try, but I have no idea which ones are > > good, or what ports are getting errors. > > > > Some lids seem to respond to 'perfquery', but I haven't been able to get > > any of my 8 port switches to respond. Is this something only in the 24 > > port switch chips? > > No. It may be the following: Although PMA is a required IB subagent per > the IB spec, it might not be available in all vendor products. That's > one possibility. Do all HCAs respond but not your switches ? That's my suspicion too. > > Also, I have two machines in a state right now where they are printing > > out: > > > > kernel: unregister_netdevice: waiting for ib0 to become free. Usage > > count = 1 > > That's an IPoIB issue... What was the sequence of events which caused > this ? IPoIB quit working, so I wrote a script to ifconfig down ib0, unload all the modules, then reload them. The 'waiting for ib0 to become free' happened when the script tried to unload the modules. Also, the following fixes ibchecknet for me.. (I seem to have a bad cable or two) gozer:~/gen2/trunk/src/userspace/management/diags/scripts# svn diff Index: ibcheckerrors =================================================================== --- ibcheckerrors (revision 2532) +++ ibcheckerrors (working copy) @@ -122,7 +122,7 @@ function blue(s) { if (mono) - normal(s) + printf s else if (!quiet) { printf "\033[1;034m" s printf "\033[0;39m" Index: ibcheckport =================================================================== --- ibcheckport (revision 2532) +++ ibcheckport (working copy) @@ -65,7 +65,7 @@ function blue(s) { if (mono) - normal(s) + printf s else if (!quiet) { printf "\033[1;034m" s printf "\033[0;39m" From halr at voltaire.com Fri Jun 3 13:40:24 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 16:40:24 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050603203314.GN577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> <20050603190350.GK577@kalmia.hozed.org> <1117825513.4493.359.camel@hal.voltaire.com> <20050603192438.GL577@kalmia.hozed.org> <1117827228.4493.379.camel@hal.voltaire.com> <20050603195816.GM577@kalmia.hozed.org> <1117829284.4493.403.camel@hal.voltaire.com> <20050603203314.GN577@kalmia.hozed.org> Message-ID: <1117830985.4493.423.camel@hal.voltaire.com> On Fri, 2005-06-03 at 16:33, Troy Benjegerdes wrote: > > > Also, I have two machines in a state right now where they are printing > > > out: > > > > > > kernel: unregister_netdevice: waiting for ib0 to become free. Usage > > > count = 1 > > > > That's an IPoIB issue... What was the sequence of events which caused > > this ? > > IPoIB quit working, so I wrote a script to ifconfig down ib0, unload all the > modules, then reload them. The 'waiting for ib0 to become free' happened > when the script tried to unload the modules. So perhaps in the midst of some error, IPoIB cannot unregister_netdev. Wonder what it is... This seems like a good test case. -- Hal From halr at voltaire.com Fri Jun 3 13:47:50 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 Jun 2005 16:47:50 -0400 Subject: [openib-general] opensm fails to bring up subnet.. In-Reply-To: <20050603203314.GN577@kalmia.hozed.org> References: <20050602232331.GD577@kalmia.hozed.org> <1117821141.4493.278.camel@hal.voltaire.com> <20050603190350.GK577@kalmia.hozed.org> <1117825513.4493.359.camel@hal.voltaire.com> <20050603192438.GL577@kalmia.hozed.org> <1117827228.4493.379.camel@hal.voltaire.com> <20050603195816.GM577@kalmia.hozed.org> <1117829284.4493.403.camel@hal.voltaire.com> <20050603203314.GN577@kalmia.hozed.org> Message-ID: <1117831670.4493.428.camel@hal.voltaire.com> On Fri, 2005-06-03 at 16:33, Troy Benjegerdes wrote: > Also, the following fixes ibchecknet for me.. Thanks. Applied. > (I seem to have a bad cable or two) Hopefully you can figure it out. You can at least check all the HCA counters. If they look OK, swap the trunk cable (the one between the 2 switches). -- Hal From bjordan at silverstorm.com Fri Jun 3 14:06:19 2005 From: bjordan at silverstorm.com (Jordan, Bill) Date: Fri, 3 Jun 2005 17:06:19 -0400 Subject: [openib-general] FW: [PATCH] ib_ucm: Change sys class name to match ib_uverbs and ib_mad Message-ID: <5D78D28F88822E4D8702BB9EEF1A4367140264@mercury.infiniconsys.com> Patch to change sys class name of userspace cm to match format of userpace verbs and userspace mad class names. Signed-off-by: Bill Jordan Index: gen2/trunk/src/linux-kernel/infiniband/core/ucm.c =================================================================== --- gen2/trunk/src/linux-kernel/infiniband/core/ucm.c (revision 2538) +++ gen2/trunk/src/linux-kernel/infiniband/core/ucm.c (working copy) @@ -1363,7 +1363,7 @@ goto err_cdev; } - ib_ucm_class = class_simple_create(THIS_MODULE, "ucm"); + ib_ucm_class = class_simple_create(THIS_MODULE, "infiniband_cm"); if (IS_ERR(ib_ucm_class)) { result = PTR_ERR(ib_ucm_class); printk(KERN_ERR "UCM: Error <%d> creating class\n", result); From libor at topspin.com Fri Jun 3 14:07:20 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 3 Jun 2005 14:07:20 -0700 Subject: [openib-general] [PATCH] ib_ucm: remove devfs usage In-Reply-To: <78d18e205060313507737f557@mail.gmail.com>; from bjordan.ics@gmail.com on Fri, Jun 03, 2005 at 04:50:02PM -0400 References: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> <20050601173638.A14794@topspin.com> <78d18e2050602124335d90d2d@mail.gmail.com> <20050602134028.E15700@topspin.com> <78d18e20506031135632cf2ba@mail.gmail.com> <527jhbwafw.fsf@topspin.com> <78d18e205060313507737f557@mail.gmail.com> Message-ID: <20050603140720.A17948@topspin.com> On Fri, Jun 03, 2005 at 04:50:02PM -0400, William Jordan wrote: > On 6/3/05, Roland Dreier wrote: > > William> Should the device nodes /dev/infiniband_mad and > > William> /dev/infiniband_verbs exist at all? Are they artifacts of > > William> the way the classes work, or a setup problem on my > > William> system? Do other people have these nodes? > > > > I don't get those nodes -- I seem to recall a udev bug (fixed a few > > versions back) that created spurious device nodes based on sysfs class > > names. > > Upgraded my udev, and they went away. Thanks. Good. So I think this covers everything except the README patch you sent out earlier. -Libor From bjordan.ics at gmail.com Fri Jun 3 13:50:02 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Fri, 3 Jun 2005 16:50:02 -0400 Subject: [openib-general] [PATCH] ib_ucm: remove devfs usage In-Reply-To: <527jhbwafw.fsf@topspin.com> References: <200506012152.j51LqlB22501@sbc2.infiniconsys.com> <20050601173638.A14794@topspin.com> <78d18e2050602124335d90d2d@mail.gmail.com> <20050602134028.E15700@topspin.com> <78d18e20506031135632cf2ba@mail.gmail.com> <527jhbwafw.fsf@topspin.com> Message-ID: <78d18e205060313507737f557@mail.gmail.com> On 6/3/05, Roland Dreier wrote: > William> Should the device nodes /dev/infiniband_mad and > William> /dev/infiniband_verbs exist at all? Are they artifacts of > William> the way the classes work, or a setup problem on my > William> system? Do other people have these nodes? > > I don't get those nodes -- I seem to recall a udev bug (fixed a few > versions back) that created spurious device nodes based on sysfs class > names. > > - R. > Upgraded my udev, and they went away. Thanks. -- Bill Jordan SilverStorm Technologies From surs at cse.ohio-state.edu Fri Jun 3 14:44:02 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Fri, 3 Jun 2005 17:44:02 -0400 Subject: [openib-general] low number of CQ entries allowed Message-ID: <20050603214400.GA28357@cse.ohio-state.edu> Hi, Today I upgraded Gen2 drivers to revision 2539. I am seeing that there are CQ & QP creation problems with larger number of CQ entries. For example, with rdma_lat (part of perftest), I can go upto number of CQ entries = 256 when I hit an error: [surs at x5:perftest] ./rdma_lat --tx-depth=256 Couldn't create QP Upon increasing the CQ entries to slightly larger values, the CQ creation fails (at exactly 512) [surs at x5:perftest] ./rdma_lat --tx-depth=512 Couldn't create CQ I am running these tests on a 3.4 GHz EM64T machine. Has anyone else also seen these problems? For applications like MPI, generally the number of CQ entries required are quite large (since we have only one CQ per MPI process and a lot of communicating peers). Typical number of CQ entries used (in MVAPICH) is around 40,000. Please let me know how to work around this problem. Thanks, Sayantan. -- http://www.cse.ohio-state.edu/~surs From xma at us.ibm.com Fri Jun 3 15:01:39 2005 From: xma at us.ibm.com (Shirley Ma) Date: Fri, 3 Jun 2005 15:01:39 -0700 Subject: [openib-general] low number of CQ entries allowed In-Reply-To: <20050603214400.GA28357@cse.ohio-state.edu> Message-ID: I have no problem to create 65000 QPs on r2519. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From surs at cse.ohio-state.edu Fri Jun 3 15:09:42 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Fri, 3 Jun 2005 18:09:42 -0400 Subject: [openib-general] low number of CQ entries allowed In-Reply-To: References: <20050603214400.GA28357@cse.ohio-state.edu> Message-ID: <20050603220941.GA28413@cse.ohio-state.edu> Hi, * On Jun,2 Shirley Ma wrote : > > I have no problem to create 65000 QPs on r2519. Thanks for your input. However, I am talking about r2539 and about the number of *completion* queue entries. I haven't yet tested creating large number of QPs. Thanks, Sayantan. > > Shirley Ma > IBM Linux Technology Center > 15300 SW Koll Parkway > Beaverton, OR 97006-6063 > Phone(Fax): (503) 578-7638 > -- http://www.cse.ohio-state.edu/~surs From tduffy at sun.com Fri Jun 3 15:32:37 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 03 Jun 2005 15:32:37 -0700 Subject: [openib-general] [PATCH] kDAPL: remove use of HANDLEs Message-ID: <1117837957.16881.21.camel@duffman> This patch is HUGE. Sorry. Not much I could do to split it. It makes a superclass for all the dat objects, and subclasses dat-provider (dapl) off of it. No more opaque handles. No more magic checking. I actually found a few bugs from the compiler because of it (more than reported to openib). The dapl_header struct is renamed dapl_common. Most of the _handle variables are renamed removing _handle. It removes the get_handle_type() function since this is no longer used. Also, it unimplements [get|set]_consumer_context (for now) since nobody was using it. DAT_BAD_HANDLE() goes as it becomes a simple NULL check. I have tested kdapltest in quit, performance, and transaction modes all with success. Please apply ASAP cause it touches a lot of code and is a pain to update (as experienced earlier today when you collapsed ep to one file). Signed-off-by: Tom Duffy dat-provider/Makefile | 3 dat-provider/dapl.h | 266 ++++------- dat-provider/dapl_cno_util.c | 26 - dat-provider/dapl_cr_accept.c | 106 +--- dat-provider/dapl_cr_callback.c | 147 ++---- dat-provider/dapl_cr_handoff.c | 2 dat-provider/dapl_cr_query.c | 10 dat-provider/dapl_cr_reject.c | 28 - dat-provider/dapl_cr_util.c | 21 dat-provider/dapl_ep.c | 609 +++++++++++--------------- dat-provider/dapl_evd_connection_callb.c | 53 +- dat-provider/dapl_evd_cq_async_error_callb.c | 4 dat-provider/dapl_evd_dequeue.c | 18 dat-provider/dapl_evd_dto_callb.c | 5 dat-provider/dapl_evd_free.c | 12 dat-provider/dapl_evd_kcreate.c | 19 dat-provider/dapl_evd_kquery.c | 14 dat-provider/dapl_evd_modify_upcall.c | 8 dat-provider/dapl_evd_post_se.c | 8 dat-provider/dapl_evd_qp_async_error_callb.c | 4 dat-provider/dapl_evd_resize.c | 16 dat-provider/dapl_evd_un_async_error_callb.c | 2 dat-provider/dapl_evd_util.c | 73 +-- dat-provider/dapl_evd_util.h | 8 dat-provider/dapl_get_consumer_context.c | 80 --- dat-provider/dapl_get_handle_type.c | 74 --- dat-provider/dapl_ia_close.c | 19 dat-provider/dapl_ia_memtype_hint.c | 10 dat-provider/dapl_ia_open.c | 62 +- dat-provider/dapl_ia_query.c | 20 dat-provider/dapl_ia_util.c | 180 +++---- dat-provider/dapl_lmr.c | 221 ++++----- dat-provider/dapl_mr_util.c | 2 dat-provider/dapl_openib_cm.c | 64 +- dat-provider/dapl_openib_qp.c | 10 dat-provider/dapl_openib_util.c | 14 dat-provider/dapl_openib_util.h | 14 dat-provider/dapl_provider.c | 5 dat-provider/dapl_psp.c | 105 ++-- dat-provider/dapl_pz.c | 79 +-- dat-provider/dapl_rmr.c | 78 +-- dat-provider/dapl_rsp.c | 87 +-- dat-provider/dapl_set_consumer_context.c | 74 --- dat-provider/dapl_sp_util.c | 46 - dat-provider/dapl_srq.c | 117 ++-- dat/api.c | 16 dat/dat.h | 469 ++++++++++---------- patches/alt_dat_provider_makefile | 3 test/dapltest/include/dapl_bpool.h | 10 test/dapltest/include/dapl_fft_util.h | 12 test/dapltest/include/dapl_performance_test.h | 14 test/dapltest/include/dapl_proto.h | 84 +-- test/dapltest/include/dapl_tdep.h | 12 test/dapltest/include/dapl_test_data.h | 20 test/dapltest/include/dapl_transaction_test.h | 18 test/dapltest/kdapl/kdapl_tdep_evd.c | 34 - test/dapltest/test/dapl_bpool.c | 38 - test/dapltest/test/dapl_client.c | 94 ++-- test/dapltest/test/dapl_cnxn.c | 4 test/dapltest/test/dapl_fft_connmgt.c | 4 test/dapltest/test/dapl_fft_dataxfer.c | 4 test/dapltest/test/dapl_fft_dataxfer_client.c | 12 test/dapltest/test/dapl_fft_endpoint.c | 118 ++--- test/dapltest/test/dapl_fft_hwconn.c | 22 test/dapltest/test/dapl_fft_mem.c | 42 - test/dapltest/test/dapl_fft_pz.c | 90 +-- test/dapltest/test/dapl_fft_queryinfo.c | 190 ++++---- test/dapltest/test/dapl_fft_util.c | 102 ++-- test/dapltest/test/dapl_limit.c | 204 ++++---- test/dapltest/test/dapl_performance_client.c | 24 - test/dapltest/test/dapl_performance_server.c | 68 +- test/dapltest/test/dapl_performance_util.c | 88 +-- test/dapltest/test/dapl_server.c | 102 ++-- test/dapltest/test/dapl_test_util.c | 108 ++-- test/dapltest/test/dapl_transaction_test.c | 164 +++---- test/dapltest/test/dapl_transaction_util.c | 50 +- 76 files changed, 2271 insertions(+), 2772 deletions(-) Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -33,7 +33,7 @@ boolean_t DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -61,10 +61,10 @@ DT_Performance_Test_Create ( test_ptr->remote_ia_addr = remote_ia_addr; test_ptr->is_remote_little_endian = is_remote_little_endian; test_ptr->base_port = (DAT_CONN_QUAL) pt_ptr->Server_Info.first_port_number; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; - ret = dat_ia_query (test_ptr->ia_handle, + ret = dat_ia_query (test_ptr->ia, NULL, &test_ptr->ia_attr, NULL); @@ -95,17 +95,17 @@ DT_Performance_Test_Create ( test_ptr->creq_evd_length = DT_PERF_DFLT_EVD_LENGTH; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; return FALSE; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->recv_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG, @@ -114,11 +114,11 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; return FALSE; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->reqt_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -127,7 +127,7 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: 0x%x dat_evd_create (request) error: %s\n", test_ptr->reqt_evd_length,test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; return FALSE; } @@ -135,30 +135,30 @@ DT_Performance_Test_Create ( if ( is_server ) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->creq_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CR_FLAG, &test_ptr->creq_evd_hdl); /* cr */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; return FALSE; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->conn_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CONNECTION_FLAG, &test_ptr->conn_evd_hdl); /* conn */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; return FALSE; } @@ -176,18 +176,18 @@ DT_Performance_Test_Create ( test_ptr->ep_context.ep_attr.max_request_dtos = pipeline_len; /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &test_ptr->ep_context.ep_attr, /* EP attrs */ - &test_ptr->ep_context.ep_handle); + &test_ptr->ep_context.ep); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->ep_context.ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context.ep = NULL; return FALSE; } @@ -197,10 +197,10 @@ DT_Performance_Test_Create ( */ test_ptr->ep_context.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, + NULL, /* rmr */ DT_PERF_SYNC_BUFF_SIZE, 2, /* 2 RMIs */ 255, /* FIXME should query for this */ @@ -226,7 +226,7 @@ DT_Performance_Test_Create ( * Post recv and sync buffers */ if ( !DT_post_recv_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_RECV_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE) ) @@ -269,13 +269,13 @@ DT_Performance_Test_Destroy ( boolean_t is_server) { u32 ret; - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; DT_Tdep_Print_Head *phead; int status = 0; phead = pt_ptr->Params.phead; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ if (test_ptr->ep_context.op.bp) @@ -310,9 +310,9 @@ DT_Performance_Test_Destroy ( * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context.ep_handle) + if (test_ptr->ep_context.ep) { - ret = dat_ep_disconnect (test_ptr->ep_context.ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context.ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -323,7 +323,7 @@ DT_Performance_Test_Destroy ( status = 1; } else if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -331,10 +331,10 @@ DT_Performance_Test_Destroy ( } } - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free error: %s\n", @@ -394,9 +394,9 @@ DT_Performance_Test_Destroy ( } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -417,7 +417,7 @@ DT_Performance_Test_Destroy ( boolean_t DT_performance_post_rdma_op ( Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats) { unsigned int j; @@ -456,7 +456,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -473,7 +473,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -502,17 +502,17 @@ DT_performance_post_rdma_op ( unsigned int DT_performance_reap ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Mode_Type mode, Performance_Stats_t *stats) { if ( BLOCKING_MODE == mode ) { - return DT_performance_wait (phead, evd_handle, stats); + return DT_performance_wait (phead, evd, stats); } else { - return DT_performance_poll (phead, evd_handle, stats); + return DT_performance_poll (phead, evd, stats); } } @@ -520,7 +520,7 @@ DT_performance_reap ( unsigned int DT_performance_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { int i; @@ -537,7 +537,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_wait ( evd_handle, + ret = DT_Tdep_evd_wait ( evd, DAT_TIMEOUT_MAX, &event); @@ -571,7 +571,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); @@ -611,7 +611,7 @@ DT_performance_wait ( unsigned int DT_performance_poll ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { u32 ret; @@ -626,7 +626,7 @@ DT_performance_poll ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: linux-kernel/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_client.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_performance_client.c (working copy) @@ -34,7 +34,7 @@ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr) { Performance_Test_t *test_ptr = NULL; @@ -47,7 +47,7 @@ DT_Performance_Test_Client ( DT_Tdep_PT_Debug (1,(phead,"Client: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - ia_handle, + ia, remote_ia_addr, FALSE, pt_ptr->Server_Info.is_little_endian, @@ -102,7 +102,7 @@ DT_Performance_Test_Client_Connect ( test_ptr->base_port, test_ptr->ep_context.port)); retry: - ret = dat_ep_connect (test_ptr->ep_context.ep_handle, + ret = dat_ep_connect (test_ptr->ep_context.ep, test_ptr->remote_ia_addr, test_ptr->ep_context.port, DAT_TIMEOUT_MAX, @@ -119,7 +119,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -297,7 +297,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -308,7 +308,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -375,9 +375,9 @@ DT_Performance_Test_Client_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -406,7 +406,7 @@ DT_Performance_Test_Client_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) @@ -461,7 +461,7 @@ DT_Performance_Test_Client_Exchange ( DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Sending Sync Msg\n", test_ptr->base_port)); if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -478,7 +478,7 @@ DT_Performance_Test_Client_Exchange ( if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Client_Sync_Send")) Index: linux-kernel/test/dapltest/test/dapl_cnxn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_cnxn.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_cnxn.c (working copy) @@ -29,7 +29,7 @@ /****************************************************************************/ int -get_ep_connection_state (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle) +get_ep_connection_state (DT_Tdep_Print_Head *phead, struct dat_ep *ep) { enum dat_ep_state ep_state; boolean_t in_dto_idle; @@ -39,7 +39,7 @@ get_ep_connection_state (DT_Tdep_Print_H char *req_status = "Idle"; - ret = dat_ep_get_status (ep_handle, &ep_state, &in_dto_idle, + ret = dat_ep_get_status (ep, &ep_state, &in_dto_idle, &out_dto_idle); if (ret != 0) { Index: linux-kernel/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_server.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_server.c (working copy) @@ -103,7 +103,7 @@ DT_cs_Server (Params_t * params_ptr) ret = dat_ia_open (Server_Cmd->dapl_name, DFLT_QLEN, &ps_ptr->async_evd_hdl, - &ps_ptr->ia_handle); + &ps_ptr->ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -111,28 +111,28 @@ DT_cs_Server (Params_t * params_ptr) module, Server_Cmd->dapl_name, DT_RetToString (ret)); - ps_ptr->ia_handle = DAT_HANDLE_NULL; + ps_ptr->ia = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: IA %s opened\n", module, Server_Cmd->dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ps_ptr->ia_handle, &ps_ptr->pz_handle); + ret = dat_pz_create (ps_ptr->ia, &ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->pz_handle = DAT_HANDLE_NULL; + ps_ptr->pz = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: PZ created\n", module)); /* Create 4 events - recv, request, connection-request, connect */ - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -142,11 +142,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->recv_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -156,11 +156,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->reqt_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CR_FLAG, @@ -170,11 +170,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (cr) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->creq_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -184,44 +184,44 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->conn_evd_hdl = NULL; status = 1; goto server_exit; } /* Create the EP */ - ret = dat_ep_create (ps_ptr->ia_handle, /* IA */ - ps_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (ps_ptr->ia, /* IA */ + ps_ptr->pz, /* PZ */ ps_ptr->recv_evd_hdl, /* recv */ ps_ptr->reqt_evd_hdl, /* request */ ps_ptr->conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ps_ptr->ep_handle); + &ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->ep_handle = DAT_HANDLE_NULL; + ps_ptr->ep = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: EP created\n", module)); /* Create PSP */ - ret = dat_psp_create (ps_ptr->ia_handle, + ret = dat_psp_create (ps_ptr->ia, SERVER_PORT_NUMBER, ps_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &ps_ptr->psp_handle); + &ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->psp_handle = DAT_HANDLE_NULL; + ps_ptr->psp = NULL; status = 1; goto server_exit; } @@ -233,10 +233,10 @@ DT_cs_Server (Params_t * params_ptr) */ ps_ptr->bpool = DT_BpoolAlloc (NULL, phead, - ps_ptr->ia_handle, - ps_ptr->pz_handle, - ps_ptr->ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ps_ptr->ia, + ps_ptr->pz, + ps_ptr->ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ 256, /* FIXME query for this value */ @@ -275,7 +275,7 @@ DT_cs_Server (Params_t * params_ptr) */ for (;/* EVER */;) { - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; @@ -299,7 +299,7 @@ DT_cs_Server (Params_t * params_ptr) /* Server_Info, Client_Info, Params set up below */ /* Gather whatever info we want about defaults */ - if (!DT_query (pt_ptr, ps_ptr->ia_handle, ps_ptr->ep_handle)) + if (!DT_query (pt_ptr, ps_ptr->ia, ps_ptr->ep)) { status = 1; goto server_exit; @@ -308,7 +308,7 @@ DT_cs_Server (Params_t * params_ptr) /* Post recv buffers for ClientInfo and Transaction_Cmd_t */ DT_Tdep_PT_Debug (1,(phead,"%s: Posting 2 recvs\n", module)); if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 0, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0))) @@ -320,7 +320,7 @@ DT_cs_Server (Params_t * params_ptr) goto server_exit; } if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 1, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1))) @@ -343,9 +343,9 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_cr_event_wait (phead, ps_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check (phead, &cr_stat, - ps_ptr->psp_handle, + ps_ptr->psp, SERVER_PORT_NUMBER, - &cr_handle, + &cr, module)) { @@ -357,7 +357,7 @@ DT_cs_Server (Params_t * params_ptr) } DT_Tdep_PT_Debug (1,(phead,"%s: Accepting Connection Request\n", module)); - ret = dat_cr_accept (cr_handle, ps_ptr->ep_handle, 0, (void *)0); + ret = dat_cr_accept (cr, ps_ptr->ep, 0, (void *)0); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -370,7 +370,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Debug (1,(phead,"%s: Awaiting connection ...\n", module)); if (!DT_conn_event_wait (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->conn_evd_hdl, &event_num)) { @@ -384,7 +384,7 @@ DT_cs_Server (Params_t * params_ptr) if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ps_ptr->ep_handle); + get_ep_connection_state (phead, ps_ptr->ep); } /* Wait for Client_Info */ @@ -395,7 +395,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0), dto_cookie, "Client_Info_Recv")) @@ -420,7 +420,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1), dto_cookie, "Client_Cmd_Recv")) @@ -585,7 +585,7 @@ DT_cs_Server (Params_t * params_ptr) /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -601,7 +601,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) @@ -619,7 +619,7 @@ DT_cs_Server (Params_t * params_ptr) /* we passed the pt_ptr to the thread and must now 'forget' it */ pt_ptr = NULL; - ret = dat_ep_disconnect (ps_ptr->ep_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect fails: %s\n", @@ -635,7 +635,7 @@ DT_cs_Server (Params_t * params_ptr) } /* reset the EP to get back into the game */ - dat_ep_reset (ps_ptr->ep_handle); + dat_ep_reset (ps_ptr->ep); DT_Tdep_PT_Debug (1,(phead,"%s: Waiting for another client...\n", module)); } /* end loop accepting connections */ @@ -669,9 +669,9 @@ server_exit: * we also get here on error, hence abrupt closure to * flush any lingering buffers posted. */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_disconnect (ps_ptr->ep_handle, + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -704,9 +704,9 @@ server_exit: } /* Free the PSP */ - if (ps_ptr->psp_handle) + if (ps_ptr->psp) { - ret = dat_psp_free (ps_ptr->psp_handle); + ret = dat_psp_free (ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_free error: %s\n", @@ -718,9 +718,9 @@ server_exit: } /* Free the EP */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_free (ps_ptr->ep_handle); + ret = dat_ep_free (ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free error: %s\n", @@ -781,9 +781,9 @@ server_exit: } /* Free the PZ */ - if (ps_ptr->pz_handle) + if (ps_ptr->pz) { - ret = dat_pz_free (ps_ptr->pz_handle); + ret = dat_pz_free (ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free error: %s\n", @@ -794,17 +794,17 @@ server_exit: } /* Close the IA */ - if (ps_ptr->ia_handle) + if (ps_ptr->ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) error: %s\n", module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -871,7 +871,7 @@ send_control_data ( sizeof (Server_Info_t)); DT_Server_Info_Endian ((Server_Info_t *) buffp); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -886,7 +886,7 @@ send_control_data ( if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) Index: linux-kernel/test/dapltest/test/dapl_fft_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) @@ -55,22 +55,22 @@ void DT_assert_fail (DT_Tdep_Print_Head } /* helper function to open an IA */ -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle) +int DT_ia_open (char *dev_name, struct dat_ia **ia) { - DAT_EVD_HANDLE evd_handle; - evd_handle = DAT_HANDLE_NULL; - return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, ia_handle); + struct dat_evd *evd; + evd = NULL; + return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, ia); } /* helper function to create an endpoint and its associated EVDs */ int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle) + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep) { u32 status; DT_Tdep_Print_Head *phead; @@ -80,7 +80,7 @@ int DT_ep_create (Params_t *params_ptr, *cr_evd = NULL; phead = params_ptr->phead; - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CR_FLAG, cr_evd); if (status != DAT_SUCCESS) { @@ -88,7 +88,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CONNECTION_FLAG, conn_evd); if (status != DAT_SUCCESS) { @@ -96,7 +96,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, send_evd); if (status != DAT_SUCCESS) @@ -105,7 +105,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, recv_evd); if (status != DAT_SUCCESS) { @@ -113,8 +113,8 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = dat_ep_create (ia_handle, pz_handle, *recv_evd, - *send_evd, *conn_evd, NULL, ep_handle); + status = dat_ep_create (ia, pz, *recv_evd, + *send_evd, *conn_evd, NULL, ep); if (status != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "dat_ep_create failed %s\n", DT_RetToString (status)); @@ -125,15 +125,15 @@ int DT_ep_create (Params_t *params_ptr, /* function that initializes the connection struct */ void DT_fft_init_conn_struct (FFT_Connection_t *conn) { - conn->ia_handle = NULL; - conn->pz_handle = NULL; - conn->psp_handle = NULL; - conn->ep_handle = NULL; + conn->ia = NULL; + conn->pz = NULL; + conn->psp = NULL; + conn->ep = NULL; conn->cr_evd = NULL; conn->send_evd = NULL; conn->conn_evd = NULL; conn->recv_evd = NULL; - conn->cr_handle = NULL; + conn->cr = NULL; conn->remote_netaddr = NULL; conn->bpool = NULL; conn->pt_ptr = NULL; @@ -154,27 +154,27 @@ void DT_fft_init_client (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); if (rc != DAT_SUCCESS) { /* make sure the handle has an invalid value */ - conn->ia_handle = NULL; + conn->ia = NULL; } DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* if a server name is given, allocate memory for a net address and set it @@ -198,11 +198,11 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_Print_Head *phead; phead = params_ptr->phead; - if (conn->ep_handle) + if (conn->ep) { if (conn->connected) { - rc = dat_ep_disconnect (conn->ep_handle, DAT_CLOSE_DEFAULT); + rc = dat_ep_disconnect (conn->ep, DAT_CLOSE_DEFAULT); DT_assert_clean (phead, rc == DAT_SUCCESS); if (!DT_disco_event_wait ( phead, conn->cr_evd, NULL )) @@ -211,16 +211,16 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_PT_Printf (phead, "DT_fft_destroy_conn_struct: bad disconnect event\n"); } } - rc = dat_ep_free (conn->ep_handle); + rc = dat_ep_free (conn->ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->bpool) { DT_Bpool_Destroy (NULL, phead, conn->bpool); } - if (conn->psp_handle) + if (conn->psp) { - rc = dat_psp_free (conn->psp_handle); + rc = dat_psp_free (conn->psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->cr_evd) @@ -247,14 +247,14 @@ int DT_fft_destroy_conn_struct (Params_t { DT_Free_Per_Test_Data (conn->pt_ptr); } - if (conn->pz_handle) + if (conn->pz) { - rc = dat_pz_free (conn->pz_handle); + rc = dat_pz_free (conn->pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn->ia_handle) + if (conn->ia) { - rc = dat_ia_close (conn->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (conn->ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return rc; @@ -272,31 +272,31 @@ void DT_fft_init_server (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PSP */ - rc = dat_psp_create (conn->ia_handle, SERVER_PORT_NUMBER, conn->cr_evd, - DAT_PSP_CONSUMER_FLAG, &conn->psp_handle); + rc = dat_psp_create (conn->ia, SERVER_PORT_NUMBER, conn->cr_evd, + DAT_PSP_CONSUMER_FLAG, &conn->psp); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate memory for buffers */ - conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia_handle, conn->pz_handle, + conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia, conn->pz, NULL, NULL, 8192, 2, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, conn->bpool); @@ -316,19 +316,19 @@ void DT_fft_listen (Params_t *params_ptr DT_assert_dat (phead, DT_cr_event_wait (phead, conn->cr_evd, &conn->cr_stat) && DT_cr_check (phead, &conn->cr_stat, - conn->psp_handle, + conn->psp, SERVER_PORT_NUMBER, - &conn->cr_handle, + &conn->cr, "DT_fft_listen")); /* accept the connection */ - rc =dat_cr_accept (conn->cr_handle, conn->ep_handle, 0, (void *)0); + rc =dat_cr_accept (conn->cr, conn->ep, 0, (void *)0); DT_assert_dat (phead, rc == DAT_SUCCESS); /* wait on a conn event via the conn EVD */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); conn->connected = TRUE; @@ -351,7 +351,7 @@ int DT_fft_connect (Params_t *params_ptr DT_Tdep_PT_Printf (phead, "Connection to server, attempt #%d\n", wait_count+1); /* attempt to connect, timeout = 10 secs */ - rc = dat_ep_connect (conn->ep_handle, conn->remote_netaddr, + rc = dat_ep_connect (conn->ep, conn->remote_netaddr, SERVER_PORT_NUMBER, 10*1000000, 0, (void *)0, DAT_QOS_BEST_EFFORT, DAT_CONNECT_DEFAULT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); @@ -359,7 +359,7 @@ int DT_fft_connect (Params_t *params_ptr /* wait on conn event */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); Index: linux-kernel/test/dapltest/test/dapl_fft_mem.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_mem.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_mem.c (working copy) @@ -37,28 +37,28 @@ int DT_mem_generic (Params_t *params_ptr FFT_Connection_t conn; DAT_REGION_DESCRIPTION region; u64 reg_size; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; DAT_LMR_CONTEXT lmr_context; u64 reg_addr; unsigned char *alloc_ptr; int res; u64 buffer_size; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; rc = 0; expect = 0; res = 1; - lmr_handle = NULL; + lmr = NULL; lmr_context = 0; reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; DT_fft_init_client (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); if (flag == 2) { @@ -76,24 +76,24 @@ int DT_mem_generic (Params_t *params_ptr memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - ia_handle = conn.ia_handle; + ia = conn.ia; if (flag != 3) { - pz_handle = conn.pz_handle; + pz = conn.pz; } if (flag != 4) { DT_Tdep_PT_Printf (phead, "Registering memory\n"); - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -110,21 +110,21 @@ int DT_mem_generic (Params_t *params_ptr } if (flag == 1) { - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_dat (phead, rc == DAT_SUCCESS); } - lmr_handle = NULL; + lmr = NULL; - rc = dat_lmr_kcreate (conn.ia_handle, + rc = dat_lmr_kcreate (conn.ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -133,9 +133,9 @@ int DT_mem_generic (Params_t *params_ptr } cleanup: - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (alloc_ptr) @@ -197,7 +197,7 @@ int DT_mem_case4 ( Params_t *params_ptr, DT_Tdep_Print_Head *phead; phead = params_ptr->phead; DT_Tdep_PT_Printf (phead, "\ - Description: Try to deregister memory with null lmr_handle\n"); + Description: Try to deregister memory with null lmr\n"); return DT_mem_generic (params_ptr, cmd, 4); } Index: linux-kernel/test/dapltest/test/dapl_limit.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) @@ -33,13 +33,13 @@ static boolean_t more_handles (DT_Tdep_Print_Head *phead, - DAT_HANDLE **old_ptrptr, /* pointer to current pointer */ + void ***old_ptrptr, /* pointer to current pointer */ unsigned int *old_count, /* number pointed to */ unsigned int size) /* size of one datum */ { unsigned int count = *old_count; - DAT_HANDLE *old_handles = *old_ptrptr; - DAT_HANDLE *handle_tmp = DT_Mdep_Malloc (count * 2 * size); + void * *old_handles = *old_ptrptr; + void * *handle_tmp = DT_Mdep_Malloc (count * 2 * size); if (!handle_tmp) { @@ -74,19 +74,19 @@ limit_test ( DT_Tdep_Print_Head *phead, Limit_Cmd_t *cmd, Limit_Index depth) { - DAT_EVD_HANDLE conn_handle; + struct dat_evd * conn_handle; typedef struct obj_set { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; - DAT_LMR_HANDLE lmr_handle; + struct dat_evd * evd; + struct dat_ep * ep; + struct dat_lmr * lmr; char * lmr_buffer; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; } Obj_Set; @@ -135,19 +135,19 @@ limit_test ( DT_Tdep_Print_Head *phead, for (w = 0; w < cmd->width; w++) { /* Specify that we want to get back an async EVD. */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdl_sets[w].ia_async_handle, - &hdl_sets[w].ia_handle); + &hdl_sets[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", module, cmd->device_name, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; - hdl_sets[w].ia_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; + hdl_sets[w].ia = NULL; goto clean_up_now; } } @@ -159,8 +159,8 @@ limit_test ( DT_Tdep_Print_Head *phead, */ typedef struct _ia { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; } OneOpen; unsigned int count = START_COUNT; @@ -178,7 +178,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -187,11 +187,11 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } /* Specify that we want to get back an async EVD. */ - hdlptr[w].ia_async_handle = DAT_HANDLE_NULL; + hdlptr[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdlptr[w].ia_async_handle, - &hdlptr[w].ia_handle); + &hdlptr[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", @@ -209,14 +209,14 @@ limit_test ( DT_Tdep_Print_Head *phead, for (tmp = 0; tmp < w; tmp++) { DT_Mdep_Schedule(); - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) fails: %s\n", module, DT_RetToString (ret)); retval = FALSE; - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -245,14 +245,14 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_pz_create (hdl_sets[w].ia_handle, - &hdl_sets[w].pz_handle); + ret = dat_pz_create (hdl_sets[w].ia, + &hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].pz_handle = DAT_HANDLE_NULL; + hdl_sets[w].pz = NULL; goto clean_up_now; } } @@ -263,7 +263,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PZs we can create */ unsigned int count = START_COUNT; - DAT_PZ_HANDLE *hdlptr = (DAT_PZ_HANDLE *) + struct dat_pz **hdlptr = (struct dat_pz **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PZ Exhaustion test loop */ @@ -279,7 +279,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -287,7 +287,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_pz_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_pz_create (hdl_sets[w % cmd->width].ia, &hdlptr[w]); if (ret != DAT_SUCCESS) { @@ -335,13 +335,13 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_cno_create (hdl_sets[w].ia_handle, + ret = dat_cno_create (hdl_sets[w].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdl_sets[w].cno_handle); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { DT_Tdep_PT_Printf (phead, "%s: dat_cno_create unimplemented\n", module); - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; /* ignore this error */ break; } @@ -350,7 +350,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_cno_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; goto clean_up_now; } } @@ -377,7 +377,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -385,7 +385,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_cno_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_cno_create (hdl_sets[w % cmd->width].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) @@ -446,7 +446,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -456,22 +456,22 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; goto clean_up_now; } for (w = 0; w < cmd->width; w++) { - ret = DT_Tdep_evd_create (hdl_sets[w].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w].ia, DFLT_QLEN, hdl_sets[w].cno_handle, flags, - &hdl_sets[w].evd_handle); + &hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].evd_handle = DAT_HANDLE_NULL; + hdl_sets[w].evd = NULL; goto clean_up_now; } } @@ -482,7 +482,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EVDs we can create */ unsigned int count = START_COUNT; - DAT_EVD_HANDLE *hdlptr = (DAT_EVD_HANDLE *) + struct dat_evd **hdlptr = (struct dat_evd **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); enum dat_evd_flags flags = ( DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG @@ -500,7 +500,7 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -510,13 +510,13 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->maximum; w++) { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -524,7 +524,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia, DFLT_QLEN, hdl_sets[w % cmd->width].cno_handle, flags, @@ -542,10 +542,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; /* EVD Cleanup loop */ - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (tmp = 0; tmp < w; tmp++) { @@ -578,19 +578,19 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_ep_create (hdl_sets[w].ia_handle, - hdl_sets[w].pz_handle, - hdl_sets[w].evd_handle, /* recv */ - hdl_sets[w].evd_handle, /* request */ + ret = dat_ep_create (hdl_sets[w].ia, + hdl_sets[w].pz, + hdl_sets[w].evd, /* recv */ + hdl_sets[w].evd, /* request */ conn_handle, /* connect */ (struct dat_ep_attr *) NULL, - &hdl_sets[w].ep_handle); + &hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ep_handle = DAT_HANDLE_NULL; + hdl_sets[w].ep = NULL; goto clean_up_now; } } @@ -601,7 +601,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EPs we can create */ unsigned int count = START_COUNT; - DAT_EP_HANDLE *hdlptr = (DAT_EP_HANDLE *) + struct dat_ep **hdlptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* EP Exhaustion test loop */ @@ -615,7 +615,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -623,10 +623,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, /* connect */ (struct dat_ep_attr *) NULL, &hdlptr[w]); @@ -672,9 +672,9 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many RSPs we can create */ unsigned int count = START_COUNT; - DAT_RSP_HANDLE *hdlptr = (DAT_RSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); - DAT_EP_HANDLE *epptr = (DAT_EP_HANDLE *) + struct dat_ep **epptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*epptr)); /* RSP Exhaustion test loop */ @@ -692,7 +692,7 @@ limit_test ( DT_Tdep_Print_Head *phead, unsigned int count1 = count; unsigned int count2 = count; - if (!more_handles (phead, (DAT_HANDLE **) &hdlptr, + if (!more_handles (phead, (void ***) &hdlptr, &count1, sizeof (*hdlptr))) { @@ -700,7 +700,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - if (!more_handles (phead, (DAT_HANDLE **) &epptr, + if (!more_handles (phead, (void ***) &epptr, &count2, sizeof (*epptr))) { @@ -723,10 +723,10 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * Each RSP needs a unique EP, so create one first */ - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, (struct dat_ep_attr *) NULL, &epptr[w]); @@ -738,10 +738,10 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } - ret = dat_rsp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_rsp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, epptr[w], - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { @@ -808,7 +808,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PSPs we can create */ unsigned int count = START_COUNT; - DAT_PSP_HANDLE *hdlptr = (DAT_PSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PSP Exhaustion test loop */ @@ -822,7 +822,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -830,9 +830,9 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_psp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_psp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, DAT_PSP_CONSUMER_FLAG, &hdlptr[w]); if (ret != DAT_SUCCESS) @@ -840,7 +840,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_psp_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); retval = TRUE; - hdlptr[w] = DAT_HANDLE_NULL; + hdlptr[w] = NULL; break; } } @@ -897,14 +897,14 @@ limit_test ( DT_Tdep_Print_Head *phead, memset (®ion, 0, sizeof (region)); region.for_va = hdl_sets[w].lmr_buffer; - ret = dat_lmr_kcreate (hdl_sets[w].ia_handle, + ret = dat_lmr_kcreate (hdl_sets[w].ia, DAT_MEM_TYPE_VIRTUAL, region, DFLT_BUFFSZ, - hdl_sets[w].pz_handle, + hdl_sets[w].pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &hdl_sets[w].lmr_handle, + &hdl_sets[w].lmr, &hdl_sets[w].lmr_context, NULL, /* FIXME */ ®_size, ®_addr); @@ -913,7 +913,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_lmr_kcreate #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].lmr_handle = DAT_HANDLE_NULL; + hdl_sets[w].lmr = NULL; goto clean_up_now; } if ((uintptr_t)reg_addr > (uintptr_t)hdl_sets[w].lmr_buffer @@ -949,7 +949,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -966,10 +966,10 @@ limit_test ( DT_Tdep_Print_Head *phead, */ hdlptr[w] = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].ep_handle, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].ep, + hdl_sets[w % cmd->width].evd, DFLT_BUFFSZ, 1, 256, /* FIXME should query for this */ @@ -1026,7 +1026,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, cmd->width * sizeof (*hdlptr))) { @@ -1054,7 +1054,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_ep_post_recv #%d\n", module, w * cmd->width + i + 1); - ret = dat_ep_post_recv (hdl_sets[i].ep_handle, + ret = dat_ep_post_recv (hdl_sets[i].ep, 1, iovp, cookie, @@ -1091,7 +1091,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * outstanding recv DTOs in error, and otherwise * be a no-op. */ - ret = dat_ep_reset (hdl_sets[i].ep_handle); + ret = dat_ep_reset (hdl_sets[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect (abrupt) fails: %s\n", @@ -1107,7 +1107,7 @@ limit_test ( DT_Tdep_Print_Head *phead, */ do { - ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd_handle, + ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd, &event); } while (ret == DAT_SUCCESS); } @@ -1130,10 +1130,10 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Mdep_Schedule(); test_bpool = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[0].ia_handle, - hdl_sets[0].pz_handle, - hdl_sets[0].ep_handle, - hdl_sets[0].evd_handle, + hdl_sets[0].ia, + hdl_sets[0].pz, + hdl_sets[0].ep, + hdl_sets[0].evd, test_size, 1, 256, /* FIXME should query for this */ @@ -1185,9 +1185,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].lmr_handle) + if (hdl_sets[w].lmr) { - ret = dat_lmr_free (hdl_sets[w].lmr_handle); + ret = dat_lmr_free (hdl_sets[w].lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_lmr_free fails: %s\n", @@ -1220,9 +1220,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ep_handle) + if (hdl_sets[w].ep) { - ret = dat_ep_free (hdl_sets[w].ep_handle); + ret = dat_ep_free (hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free fails: %s\n", @@ -1237,16 +1237,16 @@ clean_up_now: { unsigned int w; - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].evd_handle) + if (hdl_sets[w].evd) { - ret = DT_Tdep_evd_free (hdl_sets[w].evd_handle); + ret = DT_Tdep_evd_free (hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_free fails: %s\n", @@ -1284,9 +1284,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].pz_handle) + if (hdl_sets[w].pz) { - ret = dat_pz_free (hdl_sets[w].pz_handle); + ret = dat_pz_free (hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free fails: %s\n", @@ -1303,10 +1303,10 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ia_handle) + if (hdl_sets[w].ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { @@ -1318,7 +1318,7 @@ clean_up_now: * we may as well try the largest hammer we have. */ retval = FALSE; - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { Index: linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -37,31 +37,31 @@ int DT_queryinfo_basic (Params_t *params u32 result_wanted) { char *dev_name; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; struct dat_ia_attr ia_attributes; struct dat_provider_attr provider_attributes; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE cr_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *cr_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; + struct dat_ep *ep; struct dat_ep_param ep_param; DAT_CNO_HANDLE cno_handle; #ifndef __KDAPLTEST__ DAT_CNO_PARAM cno_param; #endif struct dat_evd_param evd_param; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * psp; struct dat_psp_param psp_param; - DAT_RSP_HANDLE rsp_handle; + struct dat_sp * rsp; struct dat_rsp_param rsp_param; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; struct dat_pz_param pz_param; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; struct dat_lmr_param lmr_param; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; struct dat_rmr_param rmr_param; DAT_REGION_DESCRIPTION region; u64 reg_size; @@ -77,30 +77,30 @@ int DT_queryinfo_basic (Params_t *params reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - lmr_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + lmr = NULL; rmr_handle = NULL; - pz_handle = NULL; - psp_handle = NULL; - rsp_handle = NULL; + pz = NULL; + psp = NULL; + rsp = NULL; cno_handle = NULL; - evd_handle = DAT_HANDLE_NULL; - conn_evd_handle = DAT_HANDLE_NULL; - cr_evd_handle = DAT_HANDLE_NULL; - recv_evd_handle = DAT_HANDLE_NULL; - send_evd_handle = DAT_HANDLE_NULL; + evd = NULL; + conn_evd = NULL; + cr_evd = NULL; + recv_evd = NULL; + send_evd = NULL; dev_name = cmd->device_name; - /* All functions require an ia_handle to be created */ + /* All functions require an ia to be created */ rc = dat_ia_open ((const char *)dev_name, DEFAULT_QUEUE_LEN, - &evd_handle, - &ia_handle); + &evd, + &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - /* These functions require a pz_handle to be created */ + /* These functions require a pz to be created */ if ( (object_to_query == QUERY_EVD) || (object_to_query == QUERY_RMR) || (object_to_query == QUERY_LMR) || @@ -108,43 +108,43 @@ int DT_queryinfo_basic (Params_t *params (object_to_query == QUERY_RSP) || (object_to_query == QUERY_PZ) ) { - rc = dat_pz_create (ia_handle, - &pz_handle); + rc = dat_pz_create (ia, + &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - /* These functions require a ep_handle to be created */ + /* These functions require a ep to be created */ if ( (object_to_query == QUERY_EP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CONNECTION_FLAG, - &conn_evd_handle); + &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, - pz_handle, - recv_evd_handle, - send_evd_handle, - conn_evd_handle, + rc = dat_ep_create (ia, + pz, + recv_evd, + send_evd, + conn_evd, NULL, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -152,11 +152,11 @@ int DT_queryinfo_basic (Params_t *params if ( (object_to_query == QUERY_PSP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CR_FLAG, - &cr_evd_handle); + &cr_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -165,8 +165,8 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -177,15 +177,15 @@ int DT_queryinfo_basic (Params_t *params * NULL out ia_attr and for the DAT_IA_ATTR_MASK to * have values */ - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, NULL, &provider_attributes); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_ia_query (evd_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -197,11 +197,11 @@ int DT_queryinfo_basic (Params_t *params #ifndef __KDAPLTEST__ #if defined(WIN32) - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, NULLPROXY, &cno_handle); #else - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &cno_handle); #endif @@ -220,7 +220,7 @@ int DT_queryinfo_basic (Params_t *params } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_cno_query (ia_handle, + rc = dat_cno_query (ia, &cno_param); } #endif @@ -230,17 +230,17 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, &evd_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_evd_query (ia_handle, + rc = dat_evd_query (evd, &evd_param); } } @@ -248,25 +248,25 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_psp_query function */ else if (object_to_query == QUERY_PSP) { - rc = dat_psp_create (ia_handle, + rc = dat_psp_create (ia, SERVER_PORT_NUMBER, - cr_evd_handle, + cr_evd, DAT_PSP_PROVIDER_FLAG, - &psp_handle); + &psp); DT_assert_dat (phead, rc == DAT_SUCCESS); if (result_wanted == DAT_SUCCESS) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, &psp_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_psp_query (evd_handle, + rc = dat_psp_query (psp, &psp_param); } } @@ -274,13 +274,13 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_rsp_query function */ else if (object_to_query == QUERY_RSP) { - rc = dat_rsp_create (ia_handle, + rc = dat_rsp_create (ia, SERVER_PORT_NUMBER, - ep_handle, - cr_evd_handle, - &rsp_handle); + ep, + cr_evd, + &rsp); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_rsp_query (rsp_handle, + rc = dat_rsp_query (rsp, &rsp_param); } @@ -294,14 +294,14 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_ep_query function */ else if (object_to_query == QUERY_EP) { - rc = dat_ep_query (ep_handle, + rc = dat_ep_query (ep, &ep_param); } /* Test dat_pz_query function */ else if (object_to_query == QUERY_PZ) { - rc = dat_pz_query (pz_handle, + rc = dat_pz_query (pz, &pz_param); } @@ -312,27 +312,27 @@ int DT_queryinfo_basic (Params_t *params DT_assert (phead, alloc_ptr); memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, ®_addr); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_lmr_query (lmr_handle, + rc = dat_lmr_query (lmr, &lmr_param); } /* Test dat_rmr_query function */ else if (object_to_query == QUERY_RMR) { - rc = dat_rmr_create (pz_handle, + rc = dat_rmr_create (pz, &rmr_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_rmr_query (rmr_handle, @@ -342,39 +342,39 @@ int DT_queryinfo_basic (Params_t *params DT_assert_dat (phead, DAT_GET_TYPE (rc) == result_wanted); cleanup: - if (rsp_handle) + if (rsp) { - rc = dat_rsp_free (rsp_handle); + rc = dat_rsp_free (rsp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -391,27 +391,27 @@ cleanup: DT_assert_clean (phead, rc == DAT_SUCCESS); } #endif - if (psp_handle) + if (psp) { - rc = dat_psp_free (psp_handle); + rc = dat_psp_free (psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (cr_evd_handle) + if (cr_evd) { - rc = DT_Tdep_evd_free (cr_evd_handle); + rc = DT_Tdep_evd_free (cr_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -47,7 +47,7 @@ /****************************************************************************/ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr) { Transaction_Cmd_t *cmd = &pt_ptr->Params.u.Transaction_Cmd; @@ -68,7 +68,7 @@ DT_Transaction_Test_Client (Per_Test_Dat DT_Tdep_PT_Debug (1,(phead,"Client: Starting Client side of test\n")); if (!DT_Transaction_Create_Test (pt_ptr, - ia_handle, + ia, FALSE, port_num, pt_ptr->Server_Info.is_little_endian, @@ -110,7 +110,7 @@ DT_Transaction_Test_Server (void *params + i * cmd->eps_per_thread; if (!DT_Transaction_Create_Test (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, TRUE, port_num, pt_ptr->Client_Info.is_little_endian, @@ -156,7 +156,7 @@ DT_Transaction_Test_Server (void *params boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -181,7 +181,7 @@ DT_Transaction_Create_Test (Per_Test_Dat test_ptr->remote_is_little_endian = remote_is_little_endian; test_ptr->is_server = is_server; test_ptr->pt_ptr = pt_ptr; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->base_port = (DAT_CONN_QUAL) port_num; test_ptr->cmd = &pt_ptr->Params.u.Transaction_Cmd; test_ptr->time_out = DFLT_TMO * 1000; /* DFLT_TMO seconds */ @@ -244,18 +244,18 @@ DT_Transaction_Main (void *param) private_data_str = "DAPL and RDMA rule! Test 4321."; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; status = 1; goto test_failure; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG, @@ -264,12 +264,12 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; status = 1; goto test_failure; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -278,7 +278,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (request) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; status = 1; goto test_failure; } @@ -286,7 +286,7 @@ DT_Transaction_Main (void *param) if (pt_ptr->local_is_server) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CR_FLAG, @@ -295,13 +295,13 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; status = 1; goto test_failure; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CONNECTION_FLAG, @@ -310,7 +310,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; status = 1; goto test_failure; } @@ -363,18 +363,18 @@ DT_Transaction_Main (void *param) } /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &ep_attr, /* EP attrs */ - &test_ptr->ep_context[i].ep_handle); + &test_ptr->ep_context[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - test_ptr->ep_context[i].ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context[i].ep = NULL; status = 1; goto test_failure; } @@ -385,10 +385,10 @@ DT_Transaction_Main (void *param) */ test_ptr->ep_context[i].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ buff_size, 4, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -423,7 +423,7 @@ DT_Transaction_Main (void *param) * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -433,7 +433,7 @@ DT_Transaction_Main (void *param) goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -457,11 +457,11 @@ DT_Transaction_Main (void *param) * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->creq_evd_hdl, - &test_ptr->ep_context[i].rsp_handle); + &test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create #%d error: %s\n", @@ -472,11 +472,11 @@ DT_Transaction_Main (void *param) } else { - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &test_ptr->ep_context[i].psp_handle); + &test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create #%d error: %s\n", @@ -519,7 +519,7 @@ DT_Transaction_Main (void *param) RemoteMemoryInfo *RemoteMemInfo; struct dat_dto_completion_event_data dto_stat; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; /* * Establish the connection @@ -538,16 +538,16 @@ DT_Transaction_Main (void *param) &cr_stat) || !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].rsp_handle, + test_ptr->ep_context[i].rsp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error:(%x) %s\n", @@ -571,21 +571,21 @@ DT_Transaction_Main (void *param) } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, + ret = dat_cr_accept (cr, NULL, /* NULL for RSP */ 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -594,7 +594,7 @@ DT_Transaction_Main (void *param) goto test_failure; } /* throw away single-use PSP */ - ret = dat_rsp_free (test_ptr->ep_context[i].rsp_handle); + ret = dat_rsp_free (test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free #%d error: %s\n", @@ -622,16 +622,16 @@ DT_Transaction_Main (void *param) if ( !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].psp_handle, + test_ptr->ep_context[i].psp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error: %s\n", @@ -656,33 +656,33 @@ DT_Transaction_Main (void *param) /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context[i].ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context[i].ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ - (void) dat_psp_free (test_ptr->ep_context[i].psp_handle); + /* cr consumed on failure */ + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { /* error message printed by DT_cr_event_wait */ - (void) dat_psp_free (&test_ptr->ep_context[i].psp_handle); + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* throw away single-use PSP */ - ret = dat_psp_free (test_ptr->ep_context[i].psp_handle); + ret = dat_psp_free (test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free #%d error: %s\n", @@ -709,7 +709,7 @@ DT_Transaction_Main (void *param) test_ptr->ep_context[i].ia_port)); retry: - ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_connect (test_ptr->ep_context[i].ep, test_ptr->remote_ia_addr, test_ptr->ep_context[i].ia_port, DAT_TIMEOUT_MAX, @@ -726,7 +726,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -742,7 +742,7 @@ retry: struct dat_event event; int drained = 0; - dat_ep_reset (test_ptr->ep_context[i].ep_handle); + dat_ep_reset (test_ptr->ep_context[i].ep); do { ret = DT_Tdep_evd_dequeue ( test_ptr->recv_evd_hdl, @@ -756,7 +756,7 @@ retry: * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -766,7 +766,7 @@ retry: goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -835,9 +835,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -871,9 +871,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -906,10 +906,10 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -976,7 +976,7 @@ retry: /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_SEND_BUFFER_ID, buff_size)) @@ -993,7 +993,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Send" @@ -1017,7 +1017,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Recv" @@ -1082,9 +1082,9 @@ test_failure: /* Foreach EP */ for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ for (j = 0; j < test_ptr->cmd->num_ops; j++) @@ -1122,9 +1122,9 @@ test_failure: * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context[i].ep_handle) + if (test_ptr->ep_context[i].ep) { - ret = dat_ep_disconnect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context[i].ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -1148,7 +1148,7 @@ test_failure: { if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -1164,16 +1164,16 @@ test_failure: */ for (j = 0; j < test_ptr->cmd->eps_per_thread; j++) { - if ( test_ptr->ep_context[j].ep_handle == ep_handle ) + if ( test_ptr->ep_context[j].ep == ep ) { - test_ptr->ep_context[j].ep_handle = NULL; + test_ptr->ep_context[j].ep = NULL; } } } } else /* !success - QP may be in error state */ { - ep_handle = test_ptr->ep_context[i].ep_handle; + ep = test_ptr->ep_context[i].ep; } /* @@ -1182,7 +1182,7 @@ test_failure: * disconnected as we are racing with the remote side * disconnects. */ - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { struct dat_event event; /* @@ -1195,7 +1195,7 @@ test_failure: &event); } while (ret == DAT_SUCCESS); /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free #%d error: %s\n", @@ -1259,9 +1259,9 @@ test_failure: } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -1357,7 +1357,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1432,7 +1432,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1923,8 +1923,8 @@ DT_Print_Transaction_Test (DT_Tdep_Print test_ptr->remote_is_little_endian); DT_Tdep_PT_Printf (phead, "TransTest.base_port : " F64x "\n", test_ptr->base_port); - DT_Tdep_PT_Printf (phead, "TransTest.pz_handle : %p\n", - test_ptr->pz_handle); + DT_Tdep_PT_Printf (phead, "TransTest.pz : %p\n", + test_ptr->pz); /* statistics */ DT_Tdep_PT_Printf (phead, "TransTest.bytes_send : %d\n", test_ptr->stats.stat_bytes_send); Index: linux-kernel/test/dapltest/test/dapl_performance_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_server.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_performance_server.c (working copy) @@ -41,7 +41,7 @@ DT_Performance_Test_Server ( DT_Tdep_PT_Debug (1,(phead,"Server: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, (struct sockaddr *) 0, TRUE, pt_ptr->Client_Info.is_little_endian, @@ -97,15 +97,15 @@ DT_Performance_Test_Server_Connect ( { u32 ret; boolean_t status; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; enum dat_event_number event_num; - rsp_handle = DAT_HANDLE_NULL; - psp_handle = DAT_HANDLE_NULL; + rsp = NULL; + psp = NULL; #if 0 /* FIXME */ if (test_ptr->cmd->use_rsp) { @@ -113,11 +113,11 @@ DT_Performance_Test_Server_Connect ( * Server - create a single-use RSP and * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context.port, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->creq_evd_hdl, - &rsp_handle); + &rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create error: %s\n", @@ -132,9 +132,9 @@ DT_Performance_Test_Server_Connect ( /* wait for the connection request */ if (!DT_cr_event_wait (test_ptr->conn_evd_hdl, &cr_stat) || !DT_cr_check ( &cr_stat, - DAT_HANDLE_NULL, + NULL, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -142,20 +142,20 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ - if (!DT_conn_event_wait ( test_ptr->ep_context.ep_handle, + if (!DT_conn_event_wait ( test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -173,17 +173,17 @@ DT_Performance_Test_Server_Connect ( */ status = TRUE; - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context.port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &psp_handle); + &psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); status = FALSE; - psp_handle = DAT_HANDLE_NULL; + psp = NULL; return (status); } @@ -204,9 +204,9 @@ DT_Performance_Test_Server_Connect ( if (!DT_cr_event_wait (phead, test_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check ( phead, &cr_stat, - psp_handle, + psp, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -214,22 +214,22 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num ) ) { @@ -241,10 +241,10 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_PT_Debug (1, (phead,"Server[" F64x "]: Accept on port 0x" F64x "\n", test_ptr->base_port, test_ptr->ep_context.port)); psp_free: - if ( DAT_HANDLE_NULL != psp_handle ) + if ( NULL != psp ) { /* throw away single-use PSP */ - ret = dat_psp_free (psp_handle); + ret = dat_psp_free (psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free error: %s\n", @@ -252,10 +252,10 @@ psp_free: status = FALSE; } } - if ( DAT_HANDLE_NULL != rsp_handle ) + if ( NULL != rsp ) { /* throw away single-use PSP */ - ret = dat_rsp_free (rsp_handle); + ret = dat_rsp_free (rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free error: %s\n", @@ -282,9 +282,9 @@ DT_Performance_Test_Server_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -340,7 +340,7 @@ DT_Performance_Test_Server_Exchange ( /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -358,7 +358,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Send Sync_Msg") ) @@ -380,7 +380,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (working copy) @@ -40,7 +40,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ u32 rc=0; DT_fft_init_client (phead, cmd, &conn); - DT_assert_dat (phead, conn.ia_handle != NULL) + DT_assert_dat (phead, conn.ia != NULL) DT_assert (phead, DT_fft_connect (phead, &conn)); @@ -58,8 +58,8 @@ int DT_dataxfer_client_generic (DT_Tdep_ { conn.bpool = DT_BpoolAlloc (0, phead, - conn.ia_handle, - conn.pz_handle, + conn.ia, + conn.pz, NULL, NULL, 4096, @@ -68,7 +68,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ FALSE, FALSE); DT_assert (phead, conn.bpool != 0); - rc = DT_post_send_buffer (phead, conn.ep_handle, conn.bpool, 0, + rc = DT_post_send_buffer (phead, conn.ep, conn.bpool, 0, DT_Bpool_GetBuffSize (conn.bpool, 0)); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_evd_wait (conn.send_evd, 10*1000000, 1, &conn.event, @@ -80,11 +80,11 @@ int DT_dataxfer_client_generic (DT_Tdep_ /* cleanup */ cleanup: - if (conn.ep_handle) + if (conn.ep) { /* disconnect */ DT_Tdep_PT_Printf (phead, "Disconnect\n"); - rc = dat_ep_disconnect (conn.ep_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ep_disconnect (conn.ep, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } rc = DT_fft_destroy_conn_struct (phead, &conn); Index: linux-kernel/test/dapltest/test/dapl_fft_connmgt.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (working copy) @@ -39,7 +39,7 @@ int DT_connmgt_case0 (Params_t *params_p Description: Ensure time in dat_evd_wait works correctly\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 10000, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); @@ -62,7 +62,7 @@ int DT_connmgt_case1 (Params_t *params_p Description: Attempt to use timeout of 0 in dat_evd_wait\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 0, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -86,10 +86,10 @@ Bpool * DT_BpoolAlloc ( Per_Test_Data_t *pt_ptr, DT_Tdep_Print_Head *phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia * ia, + struct dat_pz * pz, + struct dat_ep * ep, + struct dat_evd * rmr_evd, int seg_size, int num_segs, int alignment, @@ -131,16 +131,16 @@ DT_BpoolAlloc ( bpool_ptr->alloc_ptr = alloc_ptr; bpool_ptr->alloc_size = alloc_size; - bpool_ptr->pz_handle = pz_handle; + bpool_ptr->pz = pz; bpool_ptr->num_segs = num_segs; - bpool_ptr->ep_handle = ep_handle; + bpool_ptr->ep = ep; bpool_ptr->buffer_size = seg_size * num_segs; bpool_ptr->buffer_start = DT_AlignPtr (alloc_ptr, alignment) + GG_ALLOC_OFFSET; bpool_ptr->tripl_start = (struct dat_lmr_triplet *) (bpool_ptr + 1); bpool_ptr->seg_size = seg_size; bpool_ptr->enable_rdma_write = enable_rdma_write; bpool_ptr->enable_rdma_read = enable_rdma_read; - bpool_ptr->rmr_evd_handle = rmr_evd_handle; + bpool_ptr->rmr_evd = rmr_evd; DT_Tdep_PT_Debug (3, (phead, "lmr_create [%p, " F64x "]\n", @@ -160,14 +160,14 @@ DT_BpoolAlloc ( region.for_pa = virt_to_phys(region.for_va); } - ret = dat_lmr_kcreate (ia_handle, + ret = dat_lmr_kcreate (ia, DT_mem_type, region, bp_len, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &bpool_ptr->lmr_handle, + &bpool_ptr->lmr, &bpool_ptr->lmr_context, &bpool_ptr->rmr_context, &bpool_ptr->reg_size, @@ -213,7 +213,7 @@ DT_BpoolAlloc ( struct dat_rmr_bind_completion_event_data rmr_stat; /* create the RMR */ - ret = dat_rmr_create (pz_handle, &bpool_ptr->rmr_handle); + ret = dat_rmr_create (pz, &bpool_ptr->rmr_handle); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_rmr_create failed %s\n", @@ -238,7 +238,7 @@ DT_BpoolAlloc ( ret = dat_rmr_bind ( bpool_ptr->rmr_handle, &iov, mflags, - bpool_ptr->ep_handle, + bpool_ptr->ep, cookie, DAT_COMPLETION_DEFAULT_FLAG, &bpool_ptr->rmr_context); @@ -253,7 +253,7 @@ DT_BpoolAlloc ( /* await the bind result */ if (!DT_rmr_event_wait (phead, - bpool_ptr->rmr_evd_handle, + bpool_ptr->rmr_evd, &rmr_stat) || !DT_rmr_check (phead, &rmr_stat, @@ -290,9 +290,9 @@ err: DT_RetToString (ret)); } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - ret = dat_lmr_free (bpool_ptr->lmr_handle); + ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -357,9 +357,9 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - u32 ret = dat_lmr_free (bpool_ptr->lmr_handle); + u32 ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -433,8 +433,8 @@ DT_Bpool_print (DT_Tdep_Print_Head *phea "BPOOL alloc_size %x\n", (int) bpool_ptr->alloc_size); DT_Tdep_PT_Printf (phead, - "BPOOL pz_handle %p\n", - bpool_ptr->pz_handle); + "BPOOL pz %p\n", + bpool_ptr->pz); DT_Tdep_PT_Printf (phead, "BPOOL num_segs %x\n", (int) bpool_ptr->num_segs); Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -32,11 +32,11 @@ */ boolean_t DT_query ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle) + struct dat_ia * ia, + struct dat_ep * ep) { unsigned char *module = "DT_query"; - DAT_EVD_HANDLE async_evd_hdl; /* not used */ + struct dat_evd * async_evd_hdl; /* not used */ struct dat_ep_param ep_params; u32 ret; DT_Tdep_Print_Head *phead; @@ -44,7 +44,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, phead = pt_ptr->Params.phead; /* Query the IA */ - ret = dat_ia_query (ia_handle, + ret = dat_ia_query (ia, &async_evd_hdl, &pt_ptr->ia_attr, &pt_ptr->provider_attr); @@ -57,7 +57,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, } /* Query the EP */ - ret = dat_ep_query (ep_handle, &ep_params); + ret = dat_ep_query (ep, &ep_params); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_query error: %s\n", @@ -160,7 +160,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, */ boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -184,7 +184,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Recv #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_recv (ep_handle, + ret = dat_ep_post_recv (ep, 1, iov, cookie, @@ -205,7 +205,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head */ boolean_t DT_post_send_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -228,7 +228,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Send #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_send (ep_handle, + ret = dat_ep_post_send (ep, 1, iov, cookie, @@ -249,7 +249,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head */ boolean_t DT_cr_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p) { int err_cnt; @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -306,8 +306,8 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p */ boolean_t DT_conn_event_wait (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep * ep, + struct dat_evd *evd, enum dat_event_number *event_number) { @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -341,7 +341,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * { /* * Could return struct dat_connection_event_data and verify: - * event.event_data.connect_event_data.ep_handle + * event.event_data.connect_event_data.ep * event.event_data.connect_event_data.private_data_size * event.event_data.connect_event_data.private_data */ @@ -361,15 +361,15 @@ DT_conn_event_wait (DT_Tdep_Print_Head * */ boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ) + struct dat_evd *evd, + struct dat_ep * *ep ) { for (;;) { u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -392,9 +392,9 @@ DT_disco_event_wait ( DT_Tdep_Print_Head if (event.event_number == DAT_CONNECTION_EVENT_DISCONNECTED) { - if ( ep_handle != NULL ) + if ( ep != NULL ) { - *ep_handle = event.event_data.connect_event_data.ep_handle; + *ep = event.event_data.connect_event_data.ep; } return (TRUE); } @@ -412,17 +412,17 @@ DT_disco_event_wait ( DT_Tdep_Print_Head */ boolean_t DT_dto_event_reap (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dto_statusp) { if (poll) { - return DT_dto_event_poll (phead, evd_handle, dto_statusp); + return DT_dto_event_poll (phead, evd, dto_statusp); } else { - return DT_dto_event_wait (phead, evd_handle, dto_statusp); + return DT_dto_event_wait (phead, evd, dto_statusp); } } @@ -432,7 +432,7 @@ DT_dto_event_reap (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_poll (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -440,7 +440,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); if (DAT_GET_TYPE (ret) == DAT_QUEUE_EMPTY) @@ -460,7 +460,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -484,7 +484,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -505,7 +505,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -528,7 +528,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p */ boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_statusp) { for (;;) @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", @@ -572,12 +572,12 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p boolean_t DT_dto_check ( DT_Tdep_Print_Head *phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message ) { - if ( ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) || dto_p->transfered_length != len_expected || dto_p->user_cookie.as_64 != cookie_expected.as_64 || dto_p->status != DAT_DTO_SUCCESS ) @@ -588,10 +588,10 @@ DT_dto_check ( DT_Tdep_Print_Head *phead : (dto_p->status == DAT_DTO_ERR_FLUSHED ? "FAILURE" : "LengthError"))); DT_Test_Error (); - if ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) { DT_Tdep_PT_Printf (phead, "\tEndPoint mismatch (got %p wanted %p)\n", - dto_p->ep_handle, + dto_p->ep, ep_expected); } if (dto_p->transfered_length != len_expected) @@ -620,11 +620,11 @@ DT_dto_check ( DT_Tdep_Print_Head *phead boolean_t DT_rmr_check ( DT_Tdep_Print_Head *phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message) { - if ( rmr_p->rmr_handle != rmr_expected + if ( rmr_p->rmr != rmr_expected || rmr_p->user_cookie.as_ptr != cookie_expected || rmr_p->status != DAT_RMR_BIND_SUCCESS ) { @@ -634,10 +634,10 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead (rmr_p->status == DAT_RMR_BIND_SUCCESS ? "OK" : "FAILURE")); DT_Test_Error (); - if (rmr_p->rmr_handle != rmr_expected) + if (rmr_p->rmr != rmr_expected) { DT_Tdep_PT_Printf (phead, "\tRMR handle mismatch (got 0x%p wanted 0x%p)\n", - rmr_p->rmr_handle, + rmr_p->rmr, rmr_expected); } if (rmr_p->user_cookie.as_ptr != cookie_expected) @@ -659,21 +659,21 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead boolean_t DT_cr_check ( DT_Tdep_Print_Head *phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp * psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr * *crp, char *message) { u32 ret; - if (cr_handlep) + if (crp) { - *cr_handlep = (DAT_CR_HANDLE) 0; + *crp = (struct dat_cr *) 0; } if (cr_stat_p->conn_qual != port_expected || - (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected)) + (psp_expected && + cr_stat_p->sp != psp_expected)) { DT_Tdep_PT_Printf (phead, "Test Error: %s CR data problem\n", message); @@ -684,20 +684,20 @@ DT_cr_check ( DT_Tdep_Print_Head *phead " (got 0x" F64x " wanted 0x" F64x ")\n", cr_stat_p->conn_qual, port_expected); } - if (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected) + if (psp_expected && + cr_stat_p->sp != psp_expected) { DT_Tdep_PT_Printf (phead, "\tPSP mismatch (got 0x%p wanted 0x%p)\n", - cr_stat_p->sp_handle.psp_handle, - psp_handle_expected); + cr_stat_p->sp, + psp_expected); } - if (!cr_stat_p->cr_handle) + if (!cr_stat_p->cr) { - DT_Tdep_PT_Printf (phead, "\tGot NULL cr_handle\n"); + DT_Tdep_PT_Printf (phead, "\tGot NULL cr\n"); } else { - ret = dat_cr_reject (cr_stat_p->cr_handle); + ret = dat_cr_reject (cr_stat_p->cr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "\tdat_cr_reject error: %s\n", @@ -707,9 +707,9 @@ DT_cr_check ( DT_Tdep_Print_Head *phead return ( FALSE ); } - if (cr_handlep) + if (crp) { - *cr_handlep = cr_stat_p->cr_handle; + *crp = cr_stat_p->cr; } return ( TRUE ); } Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -41,13 +41,13 @@ DT_cs_Client (Params_t * params_ptr, u32 total_threads) { Per_Test_Data_t *pt_ptr = NULL; - DAT_IA_HANDLE ia_handle = DAT_HANDLE_NULL; - DAT_PZ_HANDLE pz_handle = DAT_HANDLE_NULL; - DAT_EVD_HANDLE recv_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE reqt_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE conn_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE async_evd_hdl = DAT_HANDLE_NULL; - DAT_EP_HANDLE ep_handle = DAT_HANDLE_NULL; + struct dat_ia *ia = NULL; + struct dat_pz *pz = NULL; + struct dat_evd *recv_evd_hdl = NULL; + struct dat_evd *reqt_evd_hdl = NULL; + struct dat_evd *conn_evd_hdl = NULL; + struct dat_evd *async_evd_hdl = NULL; + struct dat_ep *ep = NULL; Server_Info_t *sinfo = NULL; Transaction_Cmd_t *Transaction_Cmd = NULL; Quit_Cmd_t *Quit_Cmd = NULL; @@ -98,7 +98,7 @@ DT_cs_Client (Params_t * params_ptr, ret = dat_ia_open (dapl_name, DFLT_QLEN, &async_evd_hdl, - &ia_handle); + &ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -106,28 +106,28 @@ DT_cs_Client (Params_t * params_ptr, module, dapl_name, DT_RetToString (ret)); - ia_handle = DAT_HANDLE_NULL; + ia = NULL; status = 1; goto client_exit; } DT_Tdep_PT_Debug (1,(phead, "%s: IA %s opened\n", module, dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ia_handle, &pz_handle); + ret = dat_pz_create (ia, &pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - pz_handle = DAT_HANDLE_NULL; + pz = NULL; status = 1; goto client_exit; } /* Create 3 events - recv, request, connect */ - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -138,11 +138,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - recv_evd_hdl = DAT_HANDLE_NULL; + recv_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -153,11 +153,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - reqt_evd_hdl = DAT_HANDLE_NULL; + reqt_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -168,26 +168,26 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - conn_evd_hdl = DAT_HANDLE_NULL; + conn_evd_hdl = NULL; status = 1; goto client_exit; } /* Create an EP */ - ret = dat_ep_create (ia_handle, /* IA */ - pz_handle, /* PZ */ + ret = dat_ep_create (ia, /* IA */ + pz, /* PZ */ recv_evd_hdl, /* recv */ reqt_evd_hdl, /* request */ conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ep_handle); + &ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ep_handle = DAT_HANDLE_NULL; + ep = NULL; status = 1; goto client_exit; } @@ -197,7 +197,7 @@ DT_cs_Client (Params_t * params_ptr, * Gather whatever info we want about defaults, * and check that we can handle the requested parameters. */ - if (!DT_query (pt_ptr, ia_handle, ep_handle) || + if (!DT_query (pt_ptr, ia, ep) || !DT_check_params (pt_ptr, module)) { status = 1; @@ -206,10 +206,10 @@ DT_cs_Client (Params_t * params_ptr, bpool = DT_BpoolAlloc (pt_ptr, phead, - ia_handle, - pz_handle, - ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ia, + pz, + ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ pt_ptr->provider_attr.optimal_buffer_alignment, @@ -238,7 +238,7 @@ DT_cs_Client (Params_t * params_ptr, DT_Tdep_PT_Debug (1,(phead, "%s: Posting 1 recv buffer\n", module)); retry_repost: if (!DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, DT_Bpool_GetBuffSize (bpool, 0))) @@ -253,7 +253,7 @@ retry_repost: DT_Tdep_PT_Debug (1,(phead, "%s: Connect Endpoint\n", module)); try_connect =1; retry: - ret = dat_ep_connect (ep_handle, + ret = dat_ep_connect (ep, server_netaddr, SERVER_PORT_NUMBER, DAT_TIMEOUT_MAX, @@ -271,7 +271,7 @@ retry: } DT_Tdep_PT_Debug (1,(phead, "%s: Await connection ...\n", module)); - if (!DT_conn_event_wait (phead, ep_handle, conn_evd_hdl, &event_num)) + if (!DT_conn_event_wait (phead, ep, conn_evd_hdl, &event_num)) { if ( event_num == DAT_CONNECTION_EVENT_PEER_REJECTED ) { @@ -287,7 +287,7 @@ retry: * See if any buffers were flushed as a result of * the REJECT; clean them up and repost if so */ - dat_ep_reset (ep_handle); + dat_ep_reset (ep); do { @@ -315,7 +315,7 @@ retry: if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ep_handle); + get_ep_connection_state (phead, ep); } /* Send Client_Info (using 2nd buffer in the pool) */ @@ -326,7 +326,7 @@ retry: sizeof (Client_Info_t)); DT_Client_Info_Endian ((Client_Info_t *) buffp); if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 1, DT_Bpool_GetBuffSize (bpool, 1))) @@ -343,7 +343,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 1), dto_cookie, "Client_Info_Send")) @@ -396,7 +396,7 @@ retry: /* Send the Command buffer */ if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 2, DT_Bpool_GetBuffSize (bpool, 2))) @@ -414,7 +414,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 2), dto_cookie, "Client_Cmd_Send")) @@ -430,7 +430,7 @@ retry: if (!DT_dto_event_wait (phead, recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 0), dto_cookie, "Server_Info_Recv")) @@ -476,7 +476,7 @@ retry: DT_Transaction_Cmd_PT_Print (phead, Transaction_Cmd); } status = DT_Transaction_Test_Client (pt_ptr, - ia_handle, + ia, server_netaddr); params_ptr->Client_Stats_T.bytes_rdma_read = pt_ptr->Client_Stats.bytes_rdma_read; @@ -508,7 +508,7 @@ retry: status = DT_Performance_Test_Client (params_ptr, pt_ptr, - ia_handle, + (struct dat_ia *)ia, server_netaddr); break; } @@ -521,13 +521,13 @@ client_exit: DT_Tdep_PT_Debug (1,(phead, "%s: Cleaning Up ...\n", module)); /* Disconnect the EP */ - if (ep_handle && try_connect) + if (ep && try_connect) { /* * graceful attempt might fail because we got here due to * some error above, so we may as well try harder. */ - ret = dat_ep_disconnect (ep_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ep_disconnect (ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -547,7 +547,7 @@ client_exit: DT_Bpool_Destroy (pt_ptr, phead, bpool); /* Free the EP */ - if (ep_handle) + if (ep) { struct dat_event event; /* @@ -560,7 +560,7 @@ client_exit: &event); } while (ret == DAT_SUCCESS); - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -614,9 +614,9 @@ client_exit: } /* Free the PZ */ - if (pz_handle) + if (pz) { - ret = dat_pz_free (pz_handle); + ret = dat_pz_free (pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -629,10 +629,10 @@ client_exit: } /* Close the IA */ - if (ia_handle) + if (ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -640,7 +640,7 @@ client_exit: module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, Index: linux-kernel/test/dapltest/test/dapl_fft_endpoint.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) @@ -36,59 +36,59 @@ int DT_endpoint_generic (Params_t *param boolean_t destroy_pz_early) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; u32 rc, wanted; int res; DT_Tdep_Print_Head *phead; res = 1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - evd_handle = NULL; - conn_evd_handle = NULL; - send_evd_handle = NULL; - recv_evd_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + evd = NULL; + conn_evd = NULL; + send_evd = NULL; + recv_evd = NULL; dev_name = cmd->device_name; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; phead = params_ptr->phead; - rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, &ia_handle); + rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); if (destroy_pz_early) { - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } } - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, - DAT_EVD_CONNECTION_FLAG, &conn_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, + DAT_EVD_CONNECTION_FLAG, &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, pz_handle, recv_evd_handle, send_evd_handle, - conn_evd_handle, NULL, &ep_handle); + rc = dat_ep_create (ia, pz, recv_evd, send_evd, + conn_evd, NULL, &ep); if (destroy_pz_early) { wanted = DAT_INVALID_HANDLE; @@ -100,39 +100,39 @@ int DT_endpoint_generic (Params_t *param DT_assert_dat (phead, DAT_GET_TYPE (rc) == wanted); cleanup: - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (!destroy_pz_early && pz_handle) + if (!destroy_pz_early && pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; @@ -163,10 +163,10 @@ int DT_endpoint_case1 (Params_t *params_ int DT_endpoint_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE send_evd, conn_evd, recv_evd, cr_evd; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_ep *ep; + struct dat_evd *send_evd, *conn_evd, *recv_evd, *cr_evd; + struct dat_pz *pz; struct dat_event event; Bpool *bpool; int res; @@ -177,39 +177,39 @@ int DT_endpoint_case2 (Params_t *params_ Description: try to destroy ep with descriptor still in working queue\n"); res = 1; bpool = NULL; - pz_handle = NULL; - ia_handle = NULL; - ep_handle = NULL; + pz = NULL; + ia = NULL; + ep = NULL; send_evd = NULL; conn_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name = cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, 4096, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, 4096, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); DT_assert (phead, DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, 4096) == TRUE); - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -229,14 +229,14 @@ cleanup: rc = DT_Bpool_Destroy (NULL, phead, bpool); DT_assert_clean (phead, rc != FALSE); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_util.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c (working copy) @@ -59,7 +59,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the recv */ - ret = dat_ep_post_recv ( ep_context[i].ep_handle, + ret = dat_ep_post_recv ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -85,7 +85,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H boolean_t DT_handle_send_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll) @@ -122,7 +122,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the send */ - ret = dat_ep_post_send ( ep_context[i].ep_handle, + ret = dat_ep_post_send ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -173,7 +173,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -188,7 +188,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Send")) @@ -201,7 +201,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -215,7 +215,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -235,8 +235,8 @@ DT_handle_send_op (DT_Tdep_Print_Head *p boolean_t DT_handle_recv_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -283,7 +283,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Receive: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -298,7 +298,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Recv")) @@ -315,7 +315,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -353,7 +353,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: "F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -374,11 +374,11 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p /* * If we have multiple EPs we can't guarantee the order of - * completions, so disable ep_handle check + * completions, so disable ep check */ if (!DT_dto_check (phead, &dto_stat, - num_eps == 1?ep_context[i].ep_handle: NULL, + num_eps == 1?ep_context[i].ep: NULL, op->num_segs * op->seg_size, dto_cookie, "Send-reaped-on-recv")) @@ -395,7 +395,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -411,7 +411,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -428,7 +428,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -463,7 +463,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -516,7 +516,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (opcode == RDMA_WRITE) { - ret = dat_ep_post_rdma_write (ep_context[i].ep_handle, + ret = dat_ep_post_rdma_write (ep_context[i].ep, op->num_segs, iov, cookie, @@ -527,7 +527,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p else /* opcode == RDMA_READ */ { - ret = dat_ep_post_rdma_read ( ep_context[i].ep_handle, + ret = dat_ep_post_rdma_read ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -574,7 +574,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: %s: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -588,7 +588,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, (opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD"))) @@ -601,7 +601,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: Secondary completion seen for endpoint 0x%p (%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -619,7 +619,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: No completion seen for endpoint 0x%p (#%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); Index: linux-kernel/test/dapltest/test/dapl_fft_pz.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) @@ -35,9 +35,9 @@ int DT_pz_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_evd *evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -47,25 +47,25 @@ int DT_pz_case0 ( Params_t *params_ptr, Description: Test if we can normally create pz and destroy it.\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + ia = NULL; + pz = NULL; + evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); cleanup: - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); } return res; @@ -75,10 +75,10 @@ cleanup: int DT_pz_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE conn_evd, send_evd, recv_evd, cr_evd; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *conn_evd, *send_evd, *recv_evd, *cr_evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -88,42 +88,42 @@ int DT_pz_case1 (Params_t *params_ptr, F Description: try to destroy pz with vi still associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; conn_evd = NULL; send_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } cleanup: /* corrrect order */ - if (ep_handle) + if (ep) { - rc=dat_ep_free (ep_handle); + rc=dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn_evd) @@ -141,15 +141,15 @@ cleanup: rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -160,8 +160,8 @@ cleanup: int DT_pz_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz *pz; Bpool *bpool; u32 rc; int res; @@ -174,26 +174,26 @@ int DT_pz_case2 (Params_t *params_ptr, F associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; bpool = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate and register bpool */ - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, BUFFSIZE, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } @@ -205,15 +205,15 @@ cleanup: DT_Tdep_PT_Printf (phead, "Warning: Destroy bpool fails, reboot for cleanup\n"); return 0; } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: linux-kernel/test/dapltest/test/dapl_fft_hwconn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (working copy) @@ -31,8 +31,8 @@ int DT_hwconn_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE nic_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * nic_handle; + struct dat_evd *evd; u32 rc; int res = 1; DT_Tdep_Print_Head *phead; @@ -43,9 +43,9 @@ int DT_hwconn_case0 ( Params_t *params_p dev_name= cmd->device_name; nic_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); @@ -58,9 +58,9 @@ cleanup: /*--------------------------------------------------------*/ int DT_hwconn_case1 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; char dev_name[100]; int i; DT_Tdep_Print_Head *phead; @@ -94,8 +94,8 @@ int DT_hwconn_case1 ( Params_t *params_p sprintf (dev_name, "%s", "34df"); /* number_letter */ } - evd_handle = DAT_HANDLE_NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + evd = NULL; + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); if (DAT_GET_TYPE (rc) != DAT_PROVIDER_NOT_FOUND) { const char *major_msg, *minor_msg; @@ -120,7 +120,7 @@ int DT_hwconn_case1 ( Params_t *params_p /*--------------------------------------------------------*/ int DT_hwconn_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; int res=1; DT_Tdep_Print_Head *phead; @@ -156,9 +156,9 @@ int DT_hwconn_case3 (Params_t *params_pt DT_fft_init_client (params_ptr, cmd, &conn); /* try to close nic when vi have not destroyed */ - if (conn.ia_handle) + if (conn.ia) { - rc= dat_ia_close (conn.ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc= dat_ia_close (conn.ia, DAT_CLOSE_ABRUPT_FLAG); if (rc !=DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Warning: dat_ia_close fails %s, reboot for cleanup\n", Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (revision 2545) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (working copy) @@ -34,7 +34,7 @@ int DT_dataxfer_generic ( DT_Tdep_Print_ u32 rc=0; int res=1; DT_fft_init_server (phead, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); DT_fft_listen (phead, &conn); @@ -81,7 +81,7 @@ cleanup: int DT_dataxfer_case0 ( DT_Tdep_Print_Head *phead, FFT_Cmd_t *cmd) { DT_Tdep_PT_Printf (phead, "\ - Description: Call dat_ep_post_send with null ep_handle.\n"); + Description: Call dat_ep_post_send with null ep.\n"); return DT_dataxfer_generic (phead, cmd, 0); } Index: linux-kernel/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_tdep.h (revision 2545) +++ linux-kernel/test/dapltest/include/dapl_tdep.h (working copy) @@ -31,7 +31,7 @@ #include "dapl_proto.h" #ifdef __KDAPL__ -typedef DAT_HANDLE DAT_CNO_HANDLE; +typedef void * DAT_CNO_HANDLE; #endif /* function prototypes */ @@ -45,21 +45,21 @@ int DT_Tdep_Execute_Test ( Params_t *params_ptr ) ; u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr); + struct dat_evd * *evd_ptr); u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle); +DT_Tdep_evd_free (struct dat_evd * evd); u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd * evd, unsigned long timeout, struct dat_event *event); u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd * evd, struct dat_event *event); #endif Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2545) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -54,7 +54,7 @@ typedef struct { typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL port; int pipeline_len; @@ -70,17 +70,17 @@ typedef struct boolean_t is_remote_little_endian; DAT_CONN_QUAL base_port; struct dat_ia_attr ia_attr; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; int reqt_evd_length; - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ int recv_evd_length; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ + struct dat_evd * recv_evd_hdl; /* receive */ int conn_evd_length; - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ + struct dat_evd * conn_evd_hdl; /* connect */ int creq_evd_length; - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_evd * creq_evd_hdl; /* "" request */ Performance_Ep_Context_t ep_context; } Performance_Test_t; Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2545) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -81,10 +81,10 @@ extern int g_status; /* dapl_bpool.c */ Bpool * DT_BpoolAlloc (Per_Test_Data_t * pt_ptr, DT_Tdep_Print_Head* phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_ep *ep, + struct dat_evd *rmr_evd, int seg_size, int num_segs, int alignment, @@ -106,7 +106,7 @@ void DT_Bpool_print (DT_Tdep_ /* dapl_cnxn.c */ int get_ep_connection_state (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle); + struct dat_ep *ep); /* dapl_client.c */ int DT_cs_Client (Params_t * params_ptr, @@ -236,7 +236,7 @@ void DT_Performance_Cmd_Endia /* dapl_performance_client.c */ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, struct sockaddr *remote); boolean_t DT_Performance_Test_Client_Connect ( @@ -261,7 +261,7 @@ boolean_t DT_Performance_Test /* dapl_performance_util.c */ boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -272,20 +272,20 @@ int DT_Performance_Test_Dest boolean_t is_server); boolean_t DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats); unsigned int DT_performance_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Mode_Type mode, Performance_Stats_t *stats); unsigned int DT_performance_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); unsigned int DT_performance_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); /* dapl_performance_stats.c */ @@ -346,69 +346,69 @@ void DT_Free_Per_Test_Data (Per_Test_D /* dapl_test_util.c */ boolean_t DT_query (Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle); + struct dat_ia * ia, + struct dat_ep * ep); boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_post_send_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_conn_event_wait (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep *ep, + struct dat_evd *evd, enum dat_event_number *event_number); boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ); + struct dat_evd *evd, + struct dat_ep * *ep ); boolean_t DT_cr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p); boolean_t DT_dto_event_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_ptr); boolean_t DT_dto_check ( DT_Tdep_Print_Head* phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message); boolean_t DT_rmr_check ( DT_Tdep_Print_Head* phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message); boolean_t DT_cr_check (DT_Tdep_Print_Head* phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp *psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr **crp, char *message); /* dapl_thread.c */ @@ -460,13 +460,13 @@ void DT_Transaction_Cmd_Endia boolean_t to_wire); /* dapl_transaction_test.c */ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote); void DT_Transaction_Test_Server (void *params); boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -491,15 +491,15 @@ boolean_t DT_handle_post_recv boolean_t DT_handle_send_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll); boolean_t DT_handle_recv_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -507,7 +507,7 @@ boolean_t DT_handle_recv_op ( boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -600,15 +600,15 @@ void DT_assert_fail (DT_Tdep_Print_H char *file, char *baseFile, int line); -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle); +int DT_ia_open (char *dev_name, struct dat_ia **ia); int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle); + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep); void DT_fft_init_conn_struct (FFT_Connection_t *conn); void DT_fft_init_client (Params_t *params_ptr, FFT_Cmd_t *cmd, Index: linux-kernel/test/dapltest/include/dapl_test_data.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_test_data.h (revision 2545) +++ linux-kernel/test/dapltest/include/dapl_test_data.h (working copy) @@ -46,16 +46,16 @@ typedef struct int NextPortNumber; int num_clients; spinlock_t num_clients_lock; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; - DAT_EVD_HANDLE reqt_evd_hdl; - DAT_EVD_HANDLE conn_evd_hdl; - DAT_EVD_HANDLE creq_evd_hdl; - DAT_EVD_HANDLE async_evd_hdl; - DAT_EVD_HANDLE rmr_evd_hdl; - DAT_EP_HANDLE ep_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_evd *recv_evd_hdl; + struct dat_evd *reqt_evd_hdl; + struct dat_evd *conn_evd_hdl; + struct dat_evd *creq_evd_hdl; + struct dat_evd *async_evd_hdl; + struct dat_evd *rmr_evd_hdl; + struct dat_ep *ep; + struct dat_sp *psp; Bpool *bpool; } Per_Server_Data_t; Index: linux-kernel/test/dapltest/include/dapl_bpool.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2545) +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) @@ -38,21 +38,21 @@ typedef struct Bpool_tag Bpool; struct Bpool_tag { unsigned char *alloc_ptr; uint32_t alloc_size; - DAT_PZ_HANDLE pz_handle; + struct dat_pz * pz; int seg_size; int num_segs; /* num segments */ unsigned char *buffer_start; /* Start of buffer area */ u64 buffer_size; /* Size of data buffer (rounded) */ u64 reg_addr; /* start of registered area */ u64 reg_size; /* size of registered area */ - DAT_EP_HANDLE ep_handle; /* EP area is registered to */ - DAT_LMR_HANDLE lmr_handle; /* local access */ + struct dat_ep * ep; /* EP area is registered to */ + struct dat_lmr * lmr; /* local access */ DAT_LMR_CONTEXT lmr_context; struct dat_lmr_triplet *tripl_start; /* local IOV */ boolean_t enable_rdma_write; /* remote access */ boolean_t enable_rdma_read; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; - DAT_EVD_HANDLE rmr_evd_handle; + struct dat_evd * rmr_evd; }; #endif Index: linux-kernel/test/dapltest/include/dapl_fft_util.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2545) +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) @@ -55,14 +55,14 @@ typedef struct { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_PSP_HANDLE psp_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE cr_evd, conn_evd, send_evd, recv_evd; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_sp *psp; + struct dat_ep *ep; + struct dat_evd *cr_evd, *conn_evd, *send_evd, *recv_evd; struct dat_event event; int count; - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; Bpool *bpool; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2545) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -49,13 +49,13 @@ typedef struct typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL ia_port; Bpool *bp; Transaction_Test_Op_t op[ MAX_OPS ]; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; } Ep_Context_t; @@ -75,7 +75,7 @@ typedef struct boolean_t is_server; boolean_t remote_is_little_endian; Per_Test_Data_t *pt_ptr; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; @@ -84,11 +84,11 @@ typedef struct Thread *thread; /* This group set up by each thread in DT_Transaction_Main() */ - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_pz * pz; + struct dat_evd * recv_evd_hdl; /* receive */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ + struct dat_evd * conn_evd_hdl; /* connect */ + struct dat_evd * creq_evd_hdl; /* "" request */ Ep_Context_t *ep_context; /* Statistics set by DT_Transaction_Run() */ Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2545) +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -39,7 +39,7 @@ typedef struct Tdep_Evd_Tag struct Tdep_Evd_Tag *evd_next; struct Tdep_Event_Tag *event_next; struct Tdep_Event_Tag *event_last; - DAT_EVD_HANDLE evd_handle; + struct dat_evd * evd; DT_WAIT_OBJECT wait_object; } Tdep_Evd; @@ -94,11 +94,11 @@ KDT_Evd_Destroy(void) } u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr) + struct dat_evd **evd_ptr_ptr) { u32 dat_status; @@ -120,12 +120,12 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha upcall.upcall_func = DT_Tdep_Event_Callback; upcall.instance_data = evd_ptr; - dat_status = dat_evd_kcreate (ia_handle, + dat_status = dat_evd_kcreate (ia, evd_min_qlen, DAT_UPCALL_SINGLE_INSTANCE, &upcall, evd_flags, - evd_handle_ptr); + evd_ptr_ptr); if (dat_status != DAT_SUCCESS) { goto error; @@ -134,7 +134,7 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha /* initialize evd_ptr */ evd_ptr->event_next = NULL; evd_ptr->event_last = NULL; - evd_ptr->evd_handle = *evd_handle_ptr; + evd_ptr->evd = *evd_ptr_ptr; DT_Mdep_wait_object_init (&evd_ptr->wait_object); /* add evd_ptr to front of evd list */ @@ -154,7 +154,7 @@ error: return (dat_status); } u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd *evd, struct dat_event *dat_event) { u32 dat_status; @@ -163,11 +163,11 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ unsigned long flags; dat_status = DAT_SUCCESS; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -181,7 +181,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } /* Get event */ spin_lock_irqsave (&DT_Evd_Lock, flags); - event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd_handle */ + event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd */ if (event) { evd_ptr->event_next = event->event_next; /* point to next event */ @@ -212,7 +212,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd *evd, unsigned long timeout, struct dat_event *dat_event) { @@ -223,11 +223,11 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han dat_status = DAT_CLASS_ERROR | DAT_INTERNAL_ERROR; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -290,7 +290,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han } u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle) +DT_Tdep_evd_free (struct dat_evd *evd) { Tdep_Evd *next; Tdep_Evd *last; @@ -298,13 +298,13 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han last = NULL; spin_lock_irq (&DT_Evd_Lock); next = DT_Evd_List; - if (next->evd_handle == evd_handle) + if (next->evd == evd) { DT_Evd_List = next->evd_next; } else { - while (next && next->evd_handle != evd_handle) + while (next && next->evd != evd) { last = next; next = next->evd_next; @@ -317,7 +317,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han spin_unlock_irq (&DT_Evd_Lock); DT_Mdep_Free (next); - return dat_evd_free(evd_handle); + return dat_evd_free(evd); } static void DT_Tdep_Event_Callback ( Index: linux-kernel/dat-provider/dapl_evd_cq_async_error_callb.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_cq_async_error_callb.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_cq_async_error_callb.c (working copy) @@ -61,11 +61,11 @@ void dapl_evd_cq_async_error_callback(st if (!evd) dapl_os_panic("NULL == context\n"); - async_evd = evd->header.owner_ia->async_error_evd; + async_evd = evd->common.owner_ia->async_error_evd; dat_status = dapl_evd_post_async_error_event(async_evd, DAT_ASYNC_ERROR_EVD_OVERFLOW, - async_evd->header.owner_ia); + async_evd->common.owner_ia); if (dat_status != DAT_SUCCESS) { dapl_os_panic("async EVD overflow\n"); Index: linux-kernel/dat-provider/dapl_srq.c =================================================================== --- linux-kernel/dat-provider/dapl_srq.c (revision 2545) +++ linux-kernel/dat-provider/dapl_srq.c (working copy) @@ -51,13 +51,9 @@ */ static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) { - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); - - /* reset magic to prevent reuse */ - srq_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); + dapl_ia_unlink_srq(srq_ptr->common.owner_ia, srq_ptr); dapl_cb_free(&srq_ptr->recv_buffer); - /* no need to destroy srq_ptr->header.lock */ + /* no need to destroy srq_ptr->common.lock */ kfree(srq_ptr); } @@ -91,19 +87,14 @@ static struct dapl_srq *dapl_srq_alloc(s /* zero the structure */ memset(srq_ptr, 0, sizeof *srq_ptr); - /* - * initialize the header - */ - srq_ptr->header.provider = ia_ptr->header.provider; - srq_ptr->header.magic = DAPL_MAGIC_SRQ; - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; - srq_ptr->header.owner_ia = ia_ptr; - srq_ptr->header.user_context.as_64 = 0; - srq_ptr->header.user_context.as_ptr = NULL; + srq_ptr->srq.provider = ia_ptr->ia.provider; + srq_ptr->common.owner_ia = ia_ptr; + srq_ptr->common.user_context.as_64 = 0; + srq_ptr->common.user_context.as_ptr = NULL; atomic_set(&srq_ptr->srq_ref_count, 0); - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); - spin_lock_init(&srq_ptr->header.lock); + dapl_llist_init_entry(&srq_ptr->common.ia_list_entry); + spin_lock_init(&srq_ptr->common.lock); /* * Initialize the body. @@ -130,15 +121,15 @@ bail: * dapl_srq_create * * Create an instance of a Shared Receive Queue that is provided to the - * consumer at srq_handle. + * consumer at srq. * * Input: - * ia_handle - * pz_handle + * ia + * pz * srq_attr * * Output: - * srq_handle + * srq * * Returns: * DAT_SUCCESS @@ -148,8 +139,8 @@ bail: * ?DAT_INVALID_ATTRIBUTE?? * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) +u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_srq_attr *srq_attr, struct dat_srq **srq) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; @@ -157,14 +148,14 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_create (%p, %p, %p, %p)\n", - ia_handle, pz_handle, srq_attr, srq_handle); + ia, pz, srq_attr, srq); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -175,12 +166,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - - if (srq_handle == NULL) { + if (srq == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -197,9 +183,9 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han goto bail; } - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; + srq_ptr->param.ia = (struct dat_ia *) ia_ptr; srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; - srq_ptr->param.pz_handle = pz_handle; + srq_ptr->param.pz = pz; /* * XXX Allocate provider resource here!!! @@ -211,7 +197,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han /* Link it onto the IA */ dapl_ia_link_srq(ia_ptr, srq_ptr); - *srq_handle = srq_ptr; + *srq = (struct dat_srq *)srq_ptr; bail: return status; @@ -223,7 +209,7 @@ bail: * Destroy an instance of an SRQ * * Input: - * srq_handle + * srq * * Output: * none @@ -233,22 +219,22 @@ bail: * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) +u32 dapl_srq_free(struct dat_srq *srq) { struct dapl_srq *srq_ptr; struct dapl_ia *ia_ptr; struct dat_srq_param *param; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq); - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; param = &srq_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { + if (!srq_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -263,17 +249,17 @@ u32 dapl_srq_free(DAT_SRQ_HANDLE srq_han goto bail; } - ia_ptr = srq_ptr->header.owner_ia; + ia_ptr = srq_ptr->common.owner_ia; /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_lock_irqsave(&srq_ptr->common.lock, srq_ptr->common.flags); /* Remove the SRQ from the IA */ dapl_ia_unlink_srq(ia_ptr, srq_ptr); - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_unlock_irqrestore(&srq_ptr->common.lock, srq_ptr->common.flags); /* * Finish tearing everything down. @@ -301,7 +287,7 @@ bail: * over a connection of any ep handle into local_iov * * Input: - * srq_handle + * srq * num_segments * local_iov * user_cookie @@ -318,7 +304,7 @@ bail: * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie) { @@ -328,14 +314,14 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_post_recv (%p, %d, %p, %P)\n", - srq_handle, num_segments, local_iov, user_cookie.as_64); + srq, num_segments, local_iov, user_cookie.as_64); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * Synchronization ok since this buffer is only used for receive @@ -369,16 +355,15 @@ bail: return status; } -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) +u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_srq_query (%p, %x, %p)\n", - srq_handle, srq_param); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", srq, + srq_param); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -387,7 +372,7 @@ u32 dapl_srq_query(DAT_SRQ_HANDLE srq_ha goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * XXX Need to calculate available_dto_count and outstanding_dto_count @@ -407,7 +392,7 @@ bail: * Modify the size fo the event queue of a Shared Recieve Queue * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -421,22 +406,22 @@ bail: * DAT_INVALID_STATE */ -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) +u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", - srq_handle, srq_max_recv_dto); + srq, srq_max_recv_dto); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; - ia_ptr = srq_ptr->header.owner_ia; + srq_ptr = (struct dapl_srq *)srq; + ia_ptr = srq_ptr->common.owner_ia; /* * Check for nonsense requests per the spec @@ -463,7 +448,7 @@ bail: * event if it is reached. * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -476,20 +461,20 @@ bail: * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) +u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", - srq_handle, low_watermark); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq, + low_watermark); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* XXX Put implementation here XXX */ Index: linux-kernel/dat-provider/dapl_ia_memtype_hint.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_memtype_hint.c (revision 2545) +++ linux-kernel/dat-provider/dapl_ia_memtype_hint.c (working copy) @@ -37,7 +37,7 @@ * Provide a hint about optimal memory alignment and size * * Input: - * ia_handle, + * ia, * mem_type, * length, * mem_optimization, @@ -53,7 +53,7 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, @@ -63,11 +63,11 @@ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE i u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", - ia_handle); + ia); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; Index: linux-kernel/dat-provider/dapl_cr_callback.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_callback.c (revision 2545) +++ linux-kernel/dat-provider/dapl_cr_callback.c (working copy) @@ -50,9 +50,9 @@ static u32 dapl_connection_request(struc struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; - cr_ptr = dapl_cr_alloc(sp_ptr->header.owner_ia); + cr_ptr = dapl_cr_alloc(sp_ptr->common.owner_ia); if (cr_ptr == NULL) { /* Invoking function will call dapl_ib_cm_reject() */ return DAT_INSUFFICIENT_RESOURCES; @@ -79,7 +79,7 @@ static u32 dapl_connection_request(struc } /* EP will be NULL unless RSP service point */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { /* @@ -89,7 +89,7 @@ static u32 dapl_connection_request(struc * EP we are out of resources and need to tell the * requestor that we cant help them. */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; ep_ptr = dapl_ep_alloc(ia_ptr, NULL); if (ep_ptr == NULL) { dapl_cr_free(cr_ptr); @@ -100,7 +100,7 @@ static u32 dapl_connection_request(struc dapl_ia_link_ep(ia_ptr, ep_ptr); } - cr_ptr->param.local_ep_handle = ep_ptr; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; if (ep_ptr != NULL) { /* Assign valid EP fields: RSP and PSP_PROVIDER_FLAG only */ @@ -109,12 +109,11 @@ static u32 dapl_connection_request(struc DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING; } else { /* RSP */ - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == DAT_SP_TYPE_RSP); ep_ptr->param.ep_state = DAT_EP_STATE_PASSIVE_CONNECTION_PENDING; } - ep_ptr->cm_handle = ib_cm_handle; + ep_ptr->cm = ib_cm_handle; } /* link the CR onto the SP so we can pick it up later */ @@ -122,16 +121,16 @@ static u32 dapl_connection_request(struc /* Post the event. */ /* assign sp_ptr to union to avoid typecast errors from some compilers */ - sp_handle.psp_handle = (DAT_PSP_HANDLE) sp_ptr; - dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp_handle, - (struct sockaddr *) &sp_ptr->header.owner_ia->hca_ptr->hca_address, - sp_ptr->conn_qual, (DAT_CR_HANDLE) cr_ptr); + sp = (struct dat_sp *)sp_ptr; + dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp, + (struct sockaddr *)&sp_ptr->common.owner_ia->hca_ptr->hca_address, + sp_ptr->conn_qual, (struct dat_cr *)cr_ptr); if (dat_status != DAT_SUCCESS) { /* Take the CR off the list, we can't use it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); (void)dapl_ib_reject_connection(ib_cm_handle); @@ -157,13 +156,7 @@ struct dapl_ep *dapl_get_sp_ep(struct da /* * acquire the lock, we may be racing with other threads here */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); - /* Verify under lock that the SP is still valid */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - return NULL; - } + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * There are potentially multiple connections in progress. Need to * go through the list and find the one we are interested @@ -172,16 +165,12 @@ struct dapl_ep *dapl_get_sp_ep(struct da */ cr_ptr = dapl_sp_search_cr(sp_ptr, ib_cm_handle); if (cr_ptr == NULL) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); return NULL; } - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - - /* Quick check to ensure our EP is still valid */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) - ep_ptr = NULL; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; /* The CR record is discarded in all except for the CONNECTED case, * as it will have no further relevance. @@ -211,31 +200,31 @@ struct dapl_ep *dapl_get_sp_ep(struct da "dapl_get_sp_ep disconnect dump sp: %p \n", sp_ptr); /* Decrement the ref count on the EVD */ - if (sp_ptr->evd_handle) { + if (sp_ptr->evd) { atomic_dec(&((struct dapl_evd *)sp_ptr-> - evd_handle)->evd_ref_count); - sp_ptr->evd_handle = NULL; + evd)->evd_ref_count); + sp_ptr->evd = NULL; } sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - (void)dapl_ib_remove_conn_listener(sp_ptr->header. + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + (void)dapl_ib_remove_conn_listener(sp_ptr->common. owner_ia, sp_ptr); dapl_ia_unlink_sp((struct dapl_ia *) - sp_ptr->header.owner_ia, + sp_ptr->common.owner_ia, sp_ptr); dapl_sp_dealloc(sp_ptr); dapl_cr_free(cr_ptr); goto skip_unlock; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); } else - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); skip_unlock: return ep_ptr; @@ -255,16 +244,6 @@ void dapl_cr_callback(struct dapl_cm_id ib_cm_handle, event, sp_ptr); /* - * The SP pointer could have been cleaned up in a racing - * CM callback, check to see if we should just exit here - */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - return; - } - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); - - /* * CONNECT_REQUEST events create an event on the PSP * EVD, which will trigger connection processing. The * sequence is: @@ -277,7 +256,7 @@ void dapl_cr_callback(struct dapl_cm_id */ if (event == DAT_CONNECTION_REQUEST_EVENT) { ep_ptr = NULL; - evd_ptr = sp_ptr->evd_handle; + evd_ptr = (struct dapl_evd *)sp_ptr->evd; } else { /* see if there is an EP connected with this CM handle */ ep_ptr = dapl_get_sp_ep(ib_cm_handle, sp_ptr, event); @@ -286,7 +265,7 @@ void dapl_cr_callback(struct dapl_cm_id if (ep_ptr == NULL) return; - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; /* if something has happened to our EVD, bail. */ if (evd_ptr == NULL) return; @@ -296,30 +275,29 @@ void dapl_cr_callback(struct dapl_cm_id switch (event) { case DAT_CONNECTION_REQUEST_EVENT: - { /* * Requests arriving on a disabled SP are immediately rejected */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); if (sp_ptr->listening == FALSE) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, "---> dapl_cr_callback: conn event on down SP\n"); (void)dapl_ib_reject_connection(ib_cm_handle); return; } - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_RSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_RSP) { /* * RSP connections only allow a single connection. Close * it down NOW so we reject any further connections. */ sp_ptr->listening = FALSE; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* * Only occurs on the passive side of a connection @@ -332,24 +310,20 @@ void dapl_cr_callback(struct dapl_cm_id evd_ptr = NULL; break; - } case DAT_CONNECTION_EVENT_ESTABLISHED: - { /* This is just a notification the connection is now * established, there isn't any private data to deal with. * * Update the EP state and cache a copy of the cm handle, * then let the user know we are ready to go. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->header.magic != DAPL_MAGIC_EP || - ep_ptr->param.ep_state != - DAT_EP_STATE_COMPLETION_PENDING) { + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING) { /* If someone pulled the plug on the EP or connection, * just exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; /* Set evd_ptr = NULL so we don't generate an event below */ evd_ptr = NULL; @@ -358,72 +332,63 @@ void dapl_cr_callback(struct dapl_cm_id } ep_ptr->param.ep_state = DAT_EP_STATE_CONNECTED; - ep_ptr->cm_handle = ib_cm_handle; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + ep_ptr->cm = ib_cm_handle; + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_DISCONNECTED: - { /* * EP is now fully disconnected; initiate any post processing * to reset the underlying QP and get the EP ready for * another connection */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { /* The disconnect has already occurred, we are now * cleaned up and ready to exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); return; } ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: - { /* * After posting an accept the requesting node has * stopped talking. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_BROKEN: - { - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } default: - { evd_ptr = NULL; dapl_os_assert(0); /* shouldn't happen */ break; } - } if (evd_ptr != NULL) dat_status = dapl_evd_post_connection_event(evd_ptr, event, - (DAT_HANDLE) ep_ptr, + (struct dat_ep *)ep_ptr, 0, NULL); if (dat_status != DAT_SUCCESS) { Index: linux-kernel/dat-provider/dapl_evd_dto_callb.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_dto_callb.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_dto_callb.c (working copy) @@ -74,7 +74,6 @@ void dapl_evd_dto_callback(struct ib_cq evd_ptr = (struct dapl_evd *)user_context; dapl_os_assert(evd_ptr->cq == cq); - dapl_os_assert(evd_ptr->header.magic == DAPL_MAGIC_EVD); /* Read once. */ state = *(volatile DAPL_EVD_STATE *)&evd_ptr->evd_state; @@ -99,9 +98,9 @@ void dapl_evd_dto_callback(struct ib_cq dat_status = dapl_ib_completion_notify(evd_ptr); if (DAT_SUCCESS != dat_status) (void)dapl_evd_post_async_error_event( - evd_ptr->header.owner_ia->async_error_evd, + evd_ptr->common.owner_ia->async_error_evd, DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR, - evd_ptr->header.owner_ia); + evd_ptr->common.owner_ia); dapl_cno_trigger(evd_ptr->cno_ptr, evd_ptr); } Index: linux-kernel/dat-provider/dapl_pz.c =================================================================== --- linux-kernel/dat-provider/dapl_pz.c (revision 2545) +++ linux-kernel/dat-provider/dapl_pz.c (working copy) @@ -44,18 +44,13 @@ static struct dapl_pz *dapl_pz_alloc(str /* zero the structure */ memset(pz, 0, sizeof *pz); - /* - * initialize the header - */ - pz->header.provider = ia->header.provider; - pz->header.magic = DAPL_MAGIC_PZ; - pz->header.handle_type = DAT_HANDLE_TYPE_PZ; - pz->header.owner_ia = ia; - pz->header.user_context.as_64 = 0; - pz->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&pz->header.ia_list_entry); + pz->pz.provider = ia->ia.provider; + pz->common.owner_ia = ia; + pz->common.user_context.as_64 = 0; + pz->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&pz->common.ia_list_entry); dapl_ia_link_pz(ia, pz); - spin_lock_init(&pz->header.lock); + spin_lock_init(&pz->common.lock); /* * initialize the body @@ -67,83 +62,81 @@ static struct dapl_pz *dapl_pz_alloc(str static void dapl_pz_dealloc(struct dapl_pz *pz) { - /* reset magic to prevent reuse */ - pz->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_pz(pz->header.owner_ia, pz); + dapl_ia_unlink_pz(pz->common.owner_ia, pz); kfree(pz); } -u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) +u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; int ib_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", - ia_handle, pz_handle); + ia, pz); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error1; } - if (NULL == pz_handle) { + if (NULL == pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto error1; } - ia = (struct dapl_ia *)ia_handle; + dapl_ia = (struct dapl_ia *)ia; - pz = dapl_pz_alloc(ia); - if (NULL == pz) { + dapl_pz = dapl_pz_alloc(dapl_ia); + if (!dapl_pz) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - pz->pd = ib_alloc_pd(ia->hca_ptr->ib_hca_handle); - if (IS_ERR(pz->pd)) { - ib_status = PTR_ERR(pz->pd); + dapl_pz->pd = ib_alloc_pd(dapl_ia->hca_ptr->ib_hca_handle); + if (IS_ERR(dapl_pz->pd)) { + ib_status = PTR_ERR(dapl_pz->pd); dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_alloc_pd failed: %X\n", ib_status); status = dapl_ib_status_convert(ib_status); goto error2; } - *pz_handle = pz; + *pz = (struct dat_pz *)dapl_pz; return DAT_SUCCESS; error2: - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); error1: - *pz_handle = NULL; + *pz = NULL; return status; } -u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) +u32 dapl_pz_free(struct dat_pz *pz) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; int ib_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - if (0 != atomic_read(&pz->pz_ref_count)) { + if (0 != atomic_read(&dapl_pz->pz_ref_count)) { status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_PZ_IN_USE); goto error; } - ib_status = ib_dealloc_pd(pz->pd); + ib_status = ib_dealloc_pd(dapl_pz->pd); if (ib_status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_dealloc_pd failed: %X\n", ib_status); @@ -151,22 +144,22 @@ u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle goto error; } - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); return DAT_SUCCESS; error: return status; } -u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) +u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_param) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query(%p, %x, %p)\n", - pz_handle, pz_param); + pz, pz_param); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } @@ -175,8 +168,8 @@ u32 dapl_pz_query(DAT_PZ_HANDLE pz_handl goto error; } - pz = (struct dapl_pz *)pz_handle; - pz_param->ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; + dapl_pz = (struct dapl_pz *)pz; + pz_param->ia = (struct dat_ia *)dapl_pz->common.owner_ia; return DAT_SUCCESS; error: Index: linux-kernel/dat-provider/dapl_evd_connection_callb.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_connection_callb.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_connection_callb.c (working copy) @@ -31,6 +31,7 @@ #include "dapl.h" #include "dapl_evd_util.h" +#include "dapl_openib_util.h" /* * Connection callback function for ACTIVE connection requests; callbacks @@ -38,7 +39,7 @@ * connect call. */ -void dapl_evd_connection_callback(struct dapl_cm_id *cm_handle, +void dapl_evd_connection_callback(struct dapl_cm_id *cm, const enum dat_event_number event, const void *private_data_ptr, struct dapl_ep *ep_ptr) @@ -48,18 +49,18 @@ void dapl_evd_connection_callback(struct u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, - "dapl_evd_connection_callback: cm_handle: %p event: %x " + "dapl_evd_connection_callback: cm: %p event: %x " "ep_ptr: %p\n", - cm_handle, event, ep_ptr); + cm, event, ep_ptr); - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; private_data_size = 0; /* * All operations effect the EP, so lock it once and unlock * when necessary */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: @@ -72,13 +73,13 @@ void dapl_evd_connection_callback(struct /* If someone pulled the plug on the connection, just * exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; break; } ep_ptr->param.ep_state = DAT_EP_STATE_CONNECTED; - ep_ptr->cm_handle = cm_handle; + ep_ptr->cm = cm; if (private_data_ptr == NULL) { private_data_size = 0; } else { @@ -87,8 +88,8 @@ void dapl_evd_connection_callback(struct private_data_ptr, private_data_size); } - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; } @@ -101,8 +102,8 @@ void dapl_evd_connection_callback(struct */ ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, TRUE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; } @@ -110,8 +111,8 @@ void dapl_evd_connection_callback(struct { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, TRUE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; } @@ -119,8 +120,8 @@ void dapl_evd_connection_callback(struct { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, TRUE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; } @@ -128,8 +129,8 @@ void dapl_evd_connection_callback(struct { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, TRUE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; } @@ -137,16 +138,16 @@ void dapl_evd_connection_callback(struct { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; } case DAT_CONNECTION_REQUEST_EVENT: default: { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); evd_ptr = NULL; dapl_os_assert(0); /* shouldn't happen */ @@ -160,7 +161,7 @@ void dapl_evd_connection_callback(struct */ if (evd_ptr != NULL) { dat_status = dapl_evd_post_connection_event(evd_ptr, event, - (DAT_HANDLE) ep_ptr, + (void *) ep_ptr, private_data_size, /* 0 except for CONNECTED */ ep_ptr->private_data); @@ -172,10 +173,10 @@ void dapl_evd_connection_callback(struct * expect a callback on an ABRUPT disconnect. */ dapl_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); } } Index: linux-kernel/dat-provider/Makefile =================================================================== --- linux-kernel/dat-provider/Makefile (revision 2545) +++ linux-kernel/dat-provider/Makefile (working copy) @@ -40,8 +40,6 @@ PROVIDER_MODULES := \ dapl_evd_resize \ dapl_evd_un_async_error_callb \ dapl_evd_util \ - dapl_get_consumer_context \ - dapl_get_handle_type \ dapl_hash \ dapl_hca_util \ dapl_ia_close \ @@ -58,7 +56,6 @@ PROVIDER_MODULES := \ dapl_ring_buffer_util \ dapl_rmr \ dapl_rsp \ - dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ dapl_util Index: linux-kernel/dat-provider/dapl_get_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_get_consumer_context.c (revision 2545) +++ linux-kernel/dat-provider/dapl_get_consumer_context.c (working copy) @@ -1,80 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_get_consumer_context - * - * DAPL Requirements Version xxx, 6.2.2.2 - * - * Gets the consumer context from the specified dat_object - * - * Input: - * dat_handle - * - * Output: - * context - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ -u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, union dat_context *context) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - if (context == NULL || ((unsigned long)(header) & 3)) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - *context = header->user_context; - - bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_evd_resize.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_resize.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_resize.c (working copy) @@ -40,7 +40,7 @@ * Modify the size fo the event queue of an Event Dispatcher * * Input: - * evd_handle + * evd * evd_qlen * * Output: @@ -53,7 +53,7 @@ * DAT_INVALID_STATE */ -u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) +u32 dapl_evd_resize(struct dat_evd *evd, int evd_qlen) { struct dapl_ia *ia_ptr; struct dapl_evd *evd_ptr; @@ -61,15 +61,15 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", - evd_handle, evd_qlen); + evd, evd_qlen); - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; - ia_ptr = evd_ptr->header.owner_ia; + evd_ptr = (struct dapl_evd *)evd; + ia_ptr = evd_ptr->common.owner_ia; if (evd_qlen == evd_ptr->qlen) { status = DAT_SUCCESS; @@ -81,7 +81,7 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; } - spin_lock_irqsave(&evd_ptr->header.lock, evd_ptr->header.flags); + spin_lock_irqsave(&evd_ptr->common.lock, evd_ptr->common.flags); pend_cnt = dapl_rbuf_count(&evd_ptr->pending_event_queue); if (pend_cnt > evd_qlen) { @@ -98,6 +98,6 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; bail: - spin_unlock_irqrestore(&evd_ptr->header.lock, evd_ptr->header.flags); + spin_unlock_irqrestore(&evd_ptr->common.lock, evd_ptr->common.flags); return status; } Index: linux-kernel/dat-provider/dapl_cr_util.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_util.c (revision 2545) +++ linux-kernel/dat-provider/dapl_cr_util.c (working copy) @@ -49,17 +49,12 @@ struct dapl_cr *dapl_cr_alloc(struct dap /* zero the structure */ memset(cr_ptr, 0, sizeof *cr_ptr); - /* - * initialize the header - */ - cr_ptr->header.provider = ia_ptr->header.provider; - cr_ptr->header.magic = DAPL_MAGIC_CR; - cr_ptr->header.handle_type = DAT_HANDLE_TYPE_CR; - cr_ptr->header.owner_ia = ia_ptr; - cr_ptr->header.user_context.as_64 = 0; - cr_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&cr_ptr->header.ia_list_entry); - spin_lock_init(&cr_ptr->header.lock); + cr_ptr->cr.provider = ia_ptr->ia.provider; + cr_ptr->common.owner_ia = ia_ptr; + cr_ptr->common.user_context.as_64 = 0; + cr_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&cr_ptr->common.ia_list_entry); + spin_lock_init(&cr_ptr->common.lock); return cr_ptr; } @@ -69,9 +64,5 @@ struct dapl_cr *dapl_cr_alloc(struct dap */ void dapl_cr_free(struct dapl_cr *cr_ptr) { - dapl_os_assert(cr_ptr->header.magic == DAPL_MAGIC_CR || - cr_ptr->header.magic == DAPL_MAGIC_CR_DESTROYED); - /* reset magic to prevent reuse */ - cr_ptr->header.magic = DAPL_MAGIC_INVALID; kfree(cr_ptr); } Index: linux-kernel/dat-provider/dapl_psp.c =================================================================== --- linux-kernel/dat-provider/dapl_psp.c (revision 2545) +++ linux-kernel/dat-provider/dapl_psp.c (working copy) @@ -45,13 +45,13 @@ * the user. * * Input: - * ia_handle - * evd_handle + * ia + * evd * psp_flags * * Output: * conn_qual - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -61,9 +61,9 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ struct dapl_ia *ia_ptr; @@ -72,19 +72,19 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia u32 status = DAT_SUCCESS; int i; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } @@ -93,7 +93,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -117,10 +117,10 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; - sp_ptr->evd_handle = evd_handle; + sp_ptr->ia = ia; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -153,7 +153,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia if (status != DAT_SUCCESS) { atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -168,7 +168,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia * Return handle to the user */ *conn_qual = sp_ptr->conn_qual; - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -183,13 +183,13 @@ bail: * in a notification event. * * Input: - * ia_handle + * ia * conn_qual - * evd_handle + * evd * psp_flags * * Output: - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -198,9 +198,9 @@ bail: * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -208,24 +208,24 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -261,11 +261,11 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -298,7 +298,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han * wrong! Decrements the EVD refcount & release it. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -313,7 +313,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -325,7 +325,7 @@ bail: * Destroy a specific instance of a Service Point. * * Input: - * psp_handle + * psp * * Output: * none @@ -334,26 +334,25 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) +u32 dapl_psp_free(struct dat_sp *psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; DAPL_SP_STATE save_state; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if it has been established * and there are no current connections in progress. @@ -361,18 +360,18 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * container until the last connection is disconnected, after * which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); sp_ptr->listening = FALSE; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -389,8 +388,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han sp_ptr->cr_list_count == 0) { save_state = sp_ptr->state; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -407,7 +406,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free: PSP PENDING\n"); } @@ -416,13 +416,12 @@ bail: return status; } -u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) +u32 dapl_psp_query(struct dat_sp *psp, struct dat_psp_param *psp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || - ((struct dapl_sp *)psp_handle)->listening != TRUE) { + if (!psp || ((struct dapl_sp *)psp)->listening != TRUE) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } @@ -432,11 +431,11 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha goto bail; } - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; - psp_param->ia_handle = sp_ptr->ia_handle; + psp_param->ia = sp_ptr->ia; psp_param->conn_qual = sp_ptr->conn_qual; - psp_param->evd_handle = sp_ptr->evd_handle; + psp_param->evd = sp_ptr->evd; psp_param->psp_flags = sp_ptr->psp_flags; status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_rsp.c =================================================================== --- linux-kernel/dat-provider/dapl_rsp.c (revision 2545) +++ linux-kernel/dat-provider/dapl_rsp.c (working copy) @@ -44,13 +44,13 @@ * event * * Input: - * ia_handle + * ia * conn_qual - * ep_handle - * evd_handle + * ep + * evd * * Output: - * rsp_handle + * rsp * * Returns: * DAT_SUCCESS @@ -59,9 +59,9 @@ * DAT_INVALID_STATE * DAT_CONN_QUAL_IN_USE */ -u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle) +u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -70,39 +70,39 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free conn_qual: %x EP: %p\n", - conn_qual, ep_handle); + conn_qual, ep); - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (rsp_handle == NULL) { + if (rsp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -126,11 +126,11 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Fill out the RSP args */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = 0; - sp_ptr->ep_handle = ep_handle; + sp_ptr->ep = ep; /* * Take a reference on the EVD handle @@ -162,7 +162,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han * the state to FREE, so we know the call failed. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; sp_ptr->state = DAPL_SP_STATE_FREE; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -178,7 +178,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *rsp_handle = (DAT_RSP_HANDLE)sp_ptr; + *rsp = (struct dat_sp *)sp_ptr; bail: return status; @@ -190,7 +190,7 @@ bail: * Destroy a specific instance of a Reserved Service Point. * * Input: - * rsp_handle + * rsp * * Output: * none @@ -199,50 +199,49 @@ bail: * DAT_SUCCESS * DAT_INVALID_HANDLE */ -u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) +u32 dapl_rsp_free(struct dat_sp *rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp_handle); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_RSP)) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp); + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if there are no connections. If * we defer removing the sp it becomes something of a zombie * container until disconnection, after which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * Make sure we don't leave a dangling EP. If the state is still * RESERVED then the RSP still owns it. */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -258,8 +257,8 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han /* This RSP has never been used. Clean it up */ sp_ptr->listening = FALSE; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -275,20 +274,20 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); } bail: return status; } -u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) +u32 dapl_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *rsp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { + if (!rsp) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } @@ -298,15 +297,15 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha goto bail; } - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Fill in the RSP params */ - rsp_param->ia_handle = sp_ptr->ia_handle; + rsp_param->ia = sp_ptr->ia; rsp_param->conn_qual = sp_ptr->conn_qual; - rsp_param->evd_handle = sp_ptr->evd_handle; - rsp_param->ep_handle = sp_ptr->ep_handle; + rsp_param->evd = sp_ptr->evd; + rsp_param->ep = sp_ptr->ep; status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_set_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_set_consumer_context.c (revision 2545) +++ linux-kernel/dat-provider/dapl_set_consumer_context.c (working copy) @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_set_consumer_context - * - * Set a consumer context in the provided dat_handle - * - * Input: - * dat_handle - * context - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - */ -u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, union dat_context context) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - header->user_context = context; - - bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_evd_post_se.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_post_se.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_post_se.c (working copy) @@ -39,7 +39,7 @@ * Post a software event to the Event Dispatcher event queue. * * Input: - * evd_handle + * evd * event * Output: * none @@ -49,14 +49,14 @@ * DAT_INVALID_PARAMETER */ -u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) +u32 dapl_evd_post_se(struct dat_evd *evd, const struct dat_event *event) { struct dapl_evd *evd_ptr; u32 dat_status = DAT_SUCCESS; - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } Index: linux-kernel/dat-provider/dapl_mr_util.c =================================================================== --- linux-kernel/dat-provider/dapl_mr_util.c (revision 2545) +++ linux-kernel/dat-provider/dapl_mr_util.c (working copy) @@ -59,7 +59,7 @@ u64 dapl_mr_get_address(DAT_REGION_DESCR case DAT_MEM_TYPE_VIRTUAL: return (u64) (unsigned long) desc.for_va; case DAT_MEM_TYPE_LMR: - lmr = (struct dapl_lmr *)desc.for_lmr_handle; + lmr = (struct dapl_lmr *)desc.for_lmr; /* Since this function is recoursive we cannot inline it */ return dapl_mr_get_address(lmr->param.region_desc, Index: linux-kernel/dat-provider/dapl_ia_query.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_query.c (revision 2545) +++ linux-kernel/dat-provider/dapl_ia_query.c (working copy) @@ -38,12 +38,12 @@ * Provide the consumer with Interface Adapter and Provider parameters. * * Input: - * ia_handle + * ia * ia_mask * provider_mask * * Output: - * async_evd_handle + * async_evd * ia_parameters * provider_parameters * @@ -51,7 +51,7 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE *async_evd_handle, +u32 dapl_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -66,21 +66,21 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_query (%p, %p, %p, %p)\n", - ia_handle, - async_evd_handle, + ia, + async_evd, ia_attr, provider_attr); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (NULL != async_evd_handle) { - *async_evd_handle = ia_ptr->async_error_evd; + if (NULL != async_evd) { + *async_evd = (struct dat_evd *)ia_ptr->async_error_evd; } if (NULL != ia_attr) @@ -88,7 +88,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl if (NULL != provider_attr) { strncpy(provider_attr->provider_name, - ia_ptr->header.provider->device_name, + ia_ptr->ia.provider->device_name, DAT_NAME_MAX_LENGTH); provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; Index: linux-kernel/dat-provider/dapl_evd_free.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_free.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_free.c (working copy) @@ -39,17 +39,17 @@ /* * Destroy a specific instance of the Event Dispatcher */ -u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) +u32 dapl_evd_free(struct dat_evd *evd) { struct dapl_evd *evd_ptr; struct dapl_cno *cno_ptr; u32 dat_status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_free (%p)\n", evd_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_free (%p)\n", evd); - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -65,11 +65,11 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han */ cno_ptr = evd_ptr->cno_ptr; - dapl_ia_unlink_evd(evd_ptr->header.owner_ia, evd_ptr); + dapl_ia_unlink_evd(evd_ptr->common.owner_ia, evd_ptr); dat_status = dapl_evd_dealloc(evd_ptr); if (dat_status != DAT_SUCCESS) { - dapl_ia_link_evd(evd_ptr->header.owner_ia, evd_ptr); + dapl_ia_link_evd(evd_ptr->common.owner_ia, evd_ptr); } if (cno_ptr) Index: linux-kernel/dat-provider/dapl_evd_modify_upcall.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_modify_upcall.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_modify_upcall.c (working copy) @@ -35,7 +35,7 @@ * Modify the upcall instance on an EVD */ -u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +u32 dapl_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy upcall_policy, /* ignore */ const struct dat_upcall_object *upcall) { @@ -44,12 +44,12 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_modify_upcall (%p)\n", - evd_handle); + evd); - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; cno_ptr = evd_ptr->cno_ptr; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } Index: linux-kernel/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_util.c (revision 2545) +++ linux-kernel/dat-provider/dapl_ia_util.c (working copy) @@ -66,17 +66,12 @@ struct dapl_ia *dapl_ia_alloc(struct dat /* zero the structure */ memset(ia_ptr, 0, sizeof *ia_ptr); - /* - * initialize the header - */ - ia_ptr->header.provider = provider; - ia_ptr->header.magic = DAPL_MAGIC_IA; - ia_ptr->header.handle_type = DAT_HANDLE_TYPE_IA; - ia_ptr->header.owner_ia = ia_ptr; - ia_ptr->header.user_context.as_64 = 0; - ia_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&ia_ptr->header.ia_list_entry); - spin_lock_init(&ia_ptr->header.lock); + ia_ptr->ia.provider = provider; + ia_ptr->common.owner_ia = ia_ptr; + ia_ptr->common.user_context.as_64 = 0; + ia_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&ia_ptr->common.ia_list_entry); + spin_lock_init(&ia_ptr->common.lock); /* * initialize the body @@ -128,9 +123,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rmr_list_head)); while (rmr_ptr != NULL) { next_rmr_ptr = dapl_llist_next_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header. + &rmr_ptr->common. ia_list_entry); - dat_status = dapl_rmr_free(rmr_ptr); + dat_status = dapl_rmr_free((struct dat_rmr *)rmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rmr_free(%p) returns %x\n", @@ -142,9 +137,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rsp_list_head)); while (sp_ptr != NULL) { next_sp_ptr = dapl_llist_next_entry(&ia_ptr->rsp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); - dat_status = dapl_rsp_free(sp_ptr); + dat_status = dapl_rsp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rsp_free(%p) returns %x\n", @@ -156,7 +151,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->ep_list_head)); while (ep_ptr != NULL) { next_ep_ptr = dapl_llist_next_entry(&ia_ptr->ep_list_head, - &ep_ptr->header. + &ep_ptr->common. ia_list_entry); /* * Issue a disconnect if the EP needs it @@ -164,21 +159,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - || ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING - || ep_ptr->param.ep_state == - DAT_EP_STATE_DISCONNECT_PENDING) { - dat_status = - dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + DAT_EP_STATE_ACTIVE_CONNECTION_PENDING || + ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { + dat_status = dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_disconnect(%p) returns %x\n", ep_ptr, dat_status); } /* force the EP into error state to force flush all posted DTOs. */ - (void) dapl_modify_qp_state_to_error(ep_ptr->qp); + (void)dapl_modify_qp_state_to_error(ep_ptr->qp); - dat_status = dapl_ep_free(ep_ptr); + dat_status = dapl_ep_free((struct dat_ep *)ep_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_free(%p) returns %x\n", @@ -190,9 +183,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->lmr_list_head)); while (lmr_ptr != NULL) { next_lmr_ptr = dapl_llist_next_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header. + &lmr_ptr->common. ia_list_entry); - dat_status = dapl_lmr_free(lmr_ptr); + dat_status = dapl_lmr_free((struct dat_lmr *)lmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): lmr_free(%p) returns %x\n", @@ -214,7 +207,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status); next_sp_ptr = dapl_llist_next_entry(&ia_ptr->psp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); /* Remove CR's from this PSP and clean them up */ @@ -223,18 +216,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia while (cr_ptr != NULL) { next_cr_ptr = dapl_llist_next_entry( &sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); /* Remove the CR from the queue & cleanup */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); cr_ptr = next_cr_ptr; } - dat_status = dapl_psp_free(sp_ptr); + dat_status = dapl_psp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): psp_free(%p) returns %x\n", @@ -247,9 +241,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->pz_list_head)); while (pz_ptr != NULL) { next_pz_ptr = dapl_llist_next_entry(&ia_ptr->pz_list_head, - &pz_ptr->header. + &pz_ptr->common. ia_list_entry); - dat_status = dapl_pz_free(pz_ptr); + dat_status = dapl_pz_free((struct dat_pz *)pz_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): pz_free(%p) returns %x\n", @@ -267,11 +261,11 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->evd_list_head)); while (evd_ptr != NULL) { next_evd_ptr = dapl_llist_next_entry(&ia_ptr->evd_list_head, - &evd_ptr->header. + &evd_ptr->common. ia_list_entry); if (evd_ptr != ia_ptr->async_error_evd) { /* it isn't the async EVD; delete it. */ - dat_status = dapl_evd_free(evd_ptr); + dat_status = dapl_evd_free((struct dat_evd *)evd_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): evd_free(%p) returns %x\n", @@ -290,7 +284,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status = dapl_ia_teardown_callbacks(ia_ptr); atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - dat_status = dapl_evd_free(ia_ptr->async_error_evd); + dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != dat_status) dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -405,7 +399,7 @@ u32 dapl_ia_graceful_close(struct dapl_i if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - cur_dat_status = dapl_evd_free(ia_ptr->async_error_evd); + cur_dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; @@ -425,8 +419,6 @@ bail: */ void dapl_ia_free(struct dapl_ia *ia_ptr) { - dapl_os_assert(ia_ptr->header.magic == DAPL_MAGIC_IA); - dapl_os_assert(ia_ptr->async_error_evd == NULL); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->lmr_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rmr_list_head)); @@ -435,12 +427,8 @@ void dapl_ia_free(struct dapl_ia *ia_ptr dapl_os_assert(dapl_llist_is_empty(&ia_ptr->psp_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rsp_list_head)); - /* - * deinitialize the header - */ dapl_hca_unlink_ia(ia_ptr->hca_ptr, ia_ptr); - ia_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - /* no need to destroy ia_ptr->header.lock */ + /* no need to destroy ia_ptr->common.lock */ kfree(ia_ptr); } @@ -450,10 +438,10 @@ void dapl_ia_free(struct dapl_ia *ia_ptr */ void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry, ep_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry, ep_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -461,10 +449,10 @@ void dapl_ia_link_ep(struct dapl_ia *ia_ */ void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -472,10 +460,10 @@ void dapl_ia_unlink_ep(struct dapl_ia *i */ void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry, srq_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry, srq_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -483,10 +471,10 @@ void dapl_ia_link_srq(struct dapl_ia *ia */ void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -494,10 +482,10 @@ void dapl_ia_unlink_srq(struct dapl_ia * */ void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry, lmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry, lmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -505,10 +493,10 @@ void dapl_ia_link_lmr(struct dapl_ia *ia */ void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -516,10 +504,10 @@ void dapl_ia_unlink_lmr(struct dapl_ia * */ void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry, rmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry, rmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -527,10 +515,10 @@ void dapl_ia_link_rmr(struct dapl_ia *ia */ void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -538,10 +526,10 @@ void dapl_ia_unlink_rmr(struct dapl_ia * */ void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry, pz_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry, pz_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -549,10 +537,10 @@ void dapl_ia_link_pz(struct dapl_ia *ia_ */ void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -560,10 +548,10 @@ void dapl_ia_unlink_pz(struct dapl_ia *i */ void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry, evd_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry, evd_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -571,10 +559,10 @@ void dapl_ia_link_evd(struct dapl_ia *ia */ void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -582,10 +570,10 @@ void dapl_ia_unlink_evd(struct dapl_ia * */ void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->psp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -595,17 +583,17 @@ void dapl_ia_unlink_sp(struct dapl_ia *i { struct dapl_llist_entry **list_head; - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_PSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_PSP) { list_head = &ia_ptr->psp_list_head; } else { - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == + DAT_SP_TYPE_RSP); list_head = &ia_ptr->rsp_list_head; } - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); - dapl_llist_remove_entry(list_head, &sp_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); + dapl_llist_remove_entry(list_head, &sp_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -635,7 +623,7 @@ struct dapl_sp *dapl_ia_sp_search(struct else list_head = &ia_ptr->rsp_list_head; - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); sp_ptr = (dapl_llist_is_empty(list_head) ? NULL : dapl_llist_peek_head(list_head)); @@ -645,10 +633,10 @@ struct dapl_sp *dapl_ia_sp_search(struct break; sp_ptr = dapl_llist_next_entry(list_head, - &sp_ptr->header.ia_list_entry); + &sp_ptr->common.ia_list_entry); } - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); return sp_ptr; } @@ -671,10 +659,10 @@ struct dapl_sp *dapl_ia_sp_search(struct */ void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rsp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, Index: linux-kernel/dat-provider/dapl_provider.c =================================================================== --- linux-kernel/dat-provider/dapl_provider.c (revision 2545) +++ linux-kernel/dat-provider/dapl_provider.c (working copy) @@ -71,9 +71,8 @@ static struct dat_provider g_dapl_provid .ia_close_func = &dapl_ia_close, .ia_memtype_hint_func = &dapl_ia_memtype_hint, - .set_consumer_context_func = &dapl_set_consumer_context, - .get_consumer_context_func = &dapl_get_consumer_context, - .get_handle_type_func = &dapl_get_handle_type, + .set_consumer_context_func = NULL, + .get_consumer_context_func = NULL, .cr_query_func = &dapl_cr_query, .cr_accept_func = &dapl_cr_accept, Index: linux-kernel/dat-provider/dapl_evd_dequeue.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_dequeue.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_dequeue.c (working copy) @@ -39,7 +39,7 @@ * Remove first element from an event dispatcher * * Input: - * evd_handle + * evd * * Output: * event @@ -52,18 +52,18 @@ * DAT_QUEUE_EMPTY */ -u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) +u32 dapl_evd_dequeue(struct dat_evd *evd, struct dat_event *event) { struct dapl_evd *evd_ptr; struct dat_event *local_event; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_evd_dequeue (%p, %p)\n", evd_handle, event); + "dapl_evd_dequeue (%p, %p)\n", evd, event); - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -78,7 +78,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ * restricts us from having multiple threads in CQ poll, and the * DAT API allows multiple threads in dat_evd_dequeue() */ - spin_lock_irqsave(&evd_ptr->header.lock, evd_ptr->header.flags); + spin_lock_irqsave(&evd_ptr->common.lock, evd_ptr->common.flags); /* * Make sure there are no other waiters and the evd is active. @@ -86,8 +86,8 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ */ if (evd_ptr->evd_state != DAPL_EVD_STATE_OPEN || evd_ptr->catastrophic_overflow) { - spin_unlock_irqrestore(&evd_ptr->header.lock, - evd_ptr->header.flags); + spin_unlock_irqrestore(&evd_ptr->common.lock, + evd_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, 0); goto bail; } @@ -107,7 +107,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ dat_status = DAT_ERROR(DAT_QUEUE_EMPTY, 0); } - spin_unlock_irqrestore(&evd_ptr->header.lock, evd_ptr->header.flags); + spin_unlock_irqrestore(&evd_ptr->common.lock, evd_ptr->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_dequeue () returns 0x%x\n", dat_status); Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2545) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -70,7 +70,7 @@ static void dapl_destroy_cm_id(struct da conn, conn->cm_id); ib_destroy_cm_id(conn->cm_id); if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } } @@ -172,7 +172,7 @@ static int dapl_cm_active_cb_handler(str spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -285,7 +285,7 @@ static int dapl_cm_passive_cb_handler(st spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -413,7 +413,7 @@ error: * Initiate a connection with the passive listener on another node * * Input: - * ep_handle + * ep * remote_ia_address * remote_conn_qual * private_data_size size of private data and structure @@ -428,7 +428,7 @@ error: * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data) @@ -438,7 +438,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han struct dapl_cm_id *conn; int status; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->qp == NULL) { printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); return DAT_INTERNAL_ERROR; @@ -460,9 +460,9 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han kfree(conn); return DAT_INTERNAL_ERROR; } - conn->ep->cm_handle = conn; + conn->ep->cm = conn; - ia_ptr = conn->ep->header.owner_ia; + ia_ptr = conn->ep->common.owner_ia; /* Setup QP/CM parameters */ memset(&conn->param, 0, sizeof conn->param); @@ -507,7 +507,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han * Disconnect an EP * * Input: - * ep_handle + * ep * close_flags * * Output: @@ -521,7 +521,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han */ u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags) { - struct dapl_cm_id *conn = ep_ptr->cm_handle; + struct dapl_cm_id *conn = ep_ptr->cm; int status; dapl_dbg_log(DAPL_DBG_TYPE_CM, @@ -537,7 +537,7 @@ u32 dapl_ib_disconnect(struct dapl_ep *e status = ib_send_cm_dreq(conn->cm_id, NULL, 0); if (status) printk(KERN_ERR "dapl_ib_disconnect: CM ID 0x%p " - "status %d\n", ep_ptr->cm_handle, status); + "status %d\n", ep_ptr->cm, status); } return DAT_SUCCESS; @@ -607,14 +607,14 @@ void dapl_ib_reinit_ep(struct dapl_ep *e dapl_dbg_log(DAPL_DBG_TYPE_CM, " >>> dapl_ib_reinit_ep: EP %p\n", ep_ptr); - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* * What if QP is connected? Do we need to destroy the cm_id * here as well, send a DREQ, REJ? What cm_id state are we in? * we may not care if we just destroy the cm_id - * ib_destroy_cm_id(ep_ptr->cm_handle); - * supporting this call may require setting the cm_handle to + * ib_destroy_cm_id(ep_ptr->cm); + * supporting this call may require setting the cm to * something like NULL wherever it is destroyed */ @@ -685,7 +685,7 @@ u32 dapl_ib_setup_conn_listener(struct d * Have the CM remove a connection listener. * * Input: - * ia_handle IA handle + * ia IA handle * ServiceID IB Channel Service ID * * Output: @@ -719,7 +719,7 @@ u32 dapl_ib_remove_conn_listener(struct * Perform necessary steps to reject a connection * * Input: - * cm_handle + * cm * * Output: * none @@ -730,24 +730,24 @@ u32 dapl_ib_remove_conn_listener(struct * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle) +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm) { int status; - if (cm_handle == NULL) { + if (cm == NULL) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_reject_connection: invalid cm handle\n"); return DAT_SUCCESS; } - status = ib_send_cm_rej(cm_handle->cm_id, IB_CM_REJ_CONSUMER_DEFINED, + status = ib_send_cm_rej(cm->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_reject_connection: " "ib_send_cm_rej failed: %d\n", status); return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); } - dapl_destroy_cm_id(cm_handle); + dapl_destroy_cm_id(cm); return DAT_SUCCESS; } @@ -757,8 +757,8 @@ u32 dapl_ib_reject_connection(struct dap * Perform necessary steps to accept a connection * * Input: - * cr_handle - * ep_handle + * cr + * ep * private_data_size * private_data * @@ -771,7 +771,7 @@ u32 dapl_ib_reject_connection(struct dap * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *prd_ptr) { @@ -783,9 +783,9 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN struct ib_cm_rep_param passive_params; struct dapl_cm_id *conn; - cr_ptr = (struct dapl_cr *)cr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; - ia_ptr = ep_ptr->header.owner_ia; + cr_ptr = (struct dapl_cr *)cr; + ep_ptr = (struct dapl_ep *)ep; + ia_ptr = ep_ptr->common.owner_ia; conn = cr_ptr->ib_cm_handle; /* Obtain size of private data structure & contents */ @@ -810,8 +810,8 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN } } - cr_ptr->param.local_ep_handle = ep_handle; - ep_ptr->cm_handle = conn; + cr_ptr->param.local_ep = ep; + ep_ptr->cm = conn; conn->ep = ep_ptr; memset(&passive_params, 0, sizeof passive_params); @@ -851,7 +851,7 @@ reject: /* * ib_cm_get_remote_gid */ -static int ib_cm_get_remote_gid(struct dapl_cm_id *cm_handle, u8 *remote_gid) +static int ib_cm_get_remote_gid(struct dapl_cm_id *cm, u8 *remote_gid) { return DAT_INTERNAL_ERROR; /* for now!!! */ } @@ -862,7 +862,7 @@ static int ib_cm_get_remote_gid(struct d * Obtain the remote IP address given a connection * * Input: - * cr_handle + * cr * private data structure handle (only if not using ATS) * * Output: @@ -874,7 +874,7 @@ static int ib_cm_get_remote_gid(struct d * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address) { struct dapl_ia *ia_ptr; @@ -883,7 +883,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da struct dapl_cr *cr_ptr = (struct dapl_cr *)dat_handle; - ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia_handle; + ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia; rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); if (rc < 0) Index: linux-kernel/dat-provider/dapl_cr_reject.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_reject.c (revision 2545) +++ linux-kernel/dat-provider/dapl_cr_reject.c (working copy) @@ -40,36 +40,36 @@ * Reject a connection request from the active remote side requesting * an Endpoint. */ -u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) +u32 dapl_cr_reject(struct dat_cr *cr) { struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; struct dapl_sp *sp_ptr; u32 dat_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* * Clean up provider created EP if there is one: only if * DAT_PSP_PROVIDER_FLAG was set on the PSP */ - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - entry_ep_handle = cr_ptr->param.local_ep_handle; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; + entry_ep = cr_ptr->param.local_ep; entry_ep_state = 0; if (ep_ptr) { entry_ep_state = ep_ptr->param.ep_state; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - cr_ptr->param.local_ep_handle = NULL; + cr_ptr->param.local_ep = NULL; } dat_status = dapl_ib_reject_connection(cr_ptr->ib_cm_handle); @@ -77,8 +77,8 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand if (ep_ptr != NULL) { /* Revert our state to the beginning */ ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; - cr_ptr->param.local_ep_handle = (DAT_EP_HANDLE) ep_ptr; + cr_ptr->param.local_ep = entry_ep; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; } } else { /* @@ -88,13 +88,13 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand */ sp_ptr = cr_ptr->sp_ptr; if (ep_ptr && sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) - (void)dapl_ep_free(ep_ptr); + (void)dapl_ep_free((struct dat_ep *)ep_ptr); /* Remove the CR from the queue, then free it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); } Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2545) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -172,7 +172,7 @@ u32 dapl_ib_close_hca(struct dapl_hca *h * Alloc a CQ * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * cqlen minimum QLen * @@ -205,7 +205,7 @@ u32 dapl_ib_cq_alloc(struct dapl_ia *ia_ * Alloc a CQ * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * cqlen minimum QLen * @@ -247,7 +247,7 @@ u32 dapl_ib_cq_resize(struct dapl_ia *ia * Set up CQ completion notifications * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * * Output: @@ -273,7 +273,7 @@ u32 dapl_set_cq_notify(struct dapl_ia *i * Register a virtual memory region * * Input: - * ia_handle IA handle + * ia IA handle * lmr pointer to dapl_lmr struct * virt_addr virtual address of beginning of mem region * length length of memory region @@ -320,7 +320,7 @@ u32 dapl_ib_mr_register_physical(struct iova = buf_list[0].addr; acl = dapl_ib_convert_mem_privileges(privileges); acl |= IB_ACCESS_MW_BIND; - mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz_handle)->pd, + mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz)->pd, buf_list, length, acl, &iova); kfree(buf_list); if (IS_ERR(mr)) { @@ -413,7 +413,7 @@ u32 dapl_ib_mw_alloc(struct dapl_rmr *rm int ib_status; struct ib_mw *mw; - mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz_handle)->pd); + mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz)->pd); if (IS_ERR(mw)) { ib_status = PTR_ERR(mw); dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -560,7 +560,7 @@ u32 dapl_ib_mw_unbind(struct dapl_rmr *r * Set up an asynchronous callbacks of various kinds * * Input: - * ia_handle IA handle + * ia IA handle * handler_type type of handler to set up * callback_handle handle param for completion callbacks * callback callback routine pointer Index: linux-kernel/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2545) +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) @@ -75,17 +75,17 @@ u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ attr = &ep_ptr->param.ep_attr; - dapl_os_assert(ep_ptr->param.pz_handle != NULL); - ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz_handle)->pd; + dapl_os_assert(ep_ptr->param.pz != NULL); + ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz)->pd; ib_hca_handle = ia_ptr->hca_ptr->ib_hca_handle; cq_recv = NULL; cq_send = NULL; cq_recv = dapl_get_dto_cq(ia_ptr, - (struct dapl_evd *)ep_ptr->param.recv_evd_handle); + (struct dapl_evd *)ep_ptr->param.recv_evd); cq_send = dapl_get_dto_cq(ia_ptr, (struct dapl_evd *)ep_ptr->param. - request_evd_handle); + request_evd); qp_attr.send_cq = cq_send; qp_attr.recv_cq = cq_recv; @@ -177,7 +177,7 @@ struct ib_cq *dapl_get_dto_cq(struct dap int cq_entries; int status; - if (evd_ptr != DAT_HANDLE_NULL) + if (evd_ptr != NULL) cq = evd_ptr->cq; else if (ia_ptr->hca_ptr->null_cq != NULL) cq = ia_ptr->hca_ptr->null_cq; Index: linux-kernel/dat-provider/dapl_evd_kcreate.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_kcreate.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_kcreate.c (working copy) @@ -48,28 +48,27 @@ * even if it is not required. However, it will not be armed. */ -u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, +u32 dapl_evd_kcreate(struct dat_ia *ia, int evd_min_qlen, enum dat_upcall_policy upcall_policy, /* ignore */ const struct dat_upcall_object *upcall, - enum dat_evd_flags evd_flags, DAT_EVD_HANDLE *evd_handle) + enum dat_evd_flags evd_flags, struct dat_evd **evd) { struct dapl_ia *ia_ptr; struct dapl_evd *evd_ptr; struct dapl_cno *cno_ptr; u32 dat_status = DAT_SUCCESS; - DAT_EVD_HANDLE evdhandle = (DAT_EVD_HANDLE) NULL; + struct dat_evd *evdhandle = (struct dat_evd *) NULL; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_kcreate (%p, %d, %x, upcall, 0x%x, %p)\n", - ia_handle, - evd_min_qlen, upcall_policy, evd_flags, evd_handle); + ia, evd_min_qlen, upcall_policy, evd_flags, evd); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; cno_ptr = NULL; evd_ptr = NULL; - *evd_handle = NULL; + *evd = NULL; - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -110,7 +109,7 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha if (dat_status != DAT_SUCCESS) goto bail; - evdhandle = *evd_handle = (DAT_EVD_HANDLE) evd_ptr; + evdhandle = *evd = (struct dat_evd *) evd_ptr; bail: if (dat_status != DAT_SUCCESS && cno_ptr != NULL) @@ -118,7 +117,7 @@ bail: if (dat_status != DAT_SUCCESS) if (evd_ptr) - dapl_evd_free(evd_ptr); + dapl_evd_free((struct dat_evd *)evd_ptr); return dat_status; } Index: linux-kernel/dat-provider/dapl_cno_util.c =================================================================== --- linux-kernel/dat-provider/dapl_cno_util.c (revision 2545) +++ linux-kernel/dat-provider/dapl_cno_util.c (working copy) @@ -50,16 +50,12 @@ struct dapl_cno *dapl_cno_alloc(struct d /* zero the structure */ memset(cno_ptr, 0, sizeof *cno_ptr); - /* - * Initialize the header. - */ - cno_ptr->header.provider = ia_ptr->header.provider; - cno_ptr->header.magic = DAPL_MAGIC_CNO; - cno_ptr->header.owner_ia = ia_ptr; - cno_ptr->header.user_context.as_64 = 0; - cno_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&cno_ptr->header.ia_list_entry); - spin_lock_init(&cno_ptr->header.lock); + cno_ptr->cno.provider = ia_ptr->ia.provider; + cno_ptr->common.owner_ia = ia_ptr; + cno_ptr->common.user_context.as_64 = 0; + cno_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&cno_ptr->common.ia_list_entry); + spin_lock_init(&cno_ptr->common.lock); /* * Initialize the body @@ -74,11 +70,6 @@ struct dapl_cno *dapl_cno_alloc(struct d */ void dapl_cno_dealloc(struct dapl_cno *cno_ptr) { - dapl_os_assert(cno_ptr->header.magic == DAPL_MAGIC_CNO); - - /* reset magic to prevent reuse */ - cno_ptr->header.magic = DAPL_MAGIC_INVALID; - kfree(cno_ptr); } @@ -104,9 +95,6 @@ void dapl_cno_trigger(struct dapl_cno *c u32 status = DAT_SUCCESS; struct dat_event event; - dapl_os_assert(cno->header.magic == DAPL_MAGIC_CNO); - dapl_os_assert(evd == NULL || evd->header.magic == DAPL_MAGIC_EVD); - /* Only process events if there is an enabled callback function. */ if ((cno->cno_upcall.upcall_func == (DAT_UPCALL_FUNC) NULL) || (cno->cno_upcall_policy == DAT_UPCALL_DISABLE)) { @@ -114,7 +102,7 @@ void dapl_cno_trigger(struct dapl_cno *c } for (;;) { - status = dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); + status = dapl_evd_dequeue((struct dat_evd *)evd, &event); if (DAT_SUCCESS != status) return; Index: linux-kernel/dat-provider/dapl_ia_close.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_close.c (revision 2545) +++ linux-kernel/dat-provider/dapl_ia_close.c (working copy) @@ -40,7 +40,7 @@ * Close a provider, clean up resources, etc. * * Input: - * ia_handle + * ia * * Output: * none @@ -50,30 +50,29 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags ia_flags) +u32 dapl_ia_close(struct dat_ia *ia, enum dat_close_flags ia_flags) { struct dapl_ia *ia_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); + "dapl_ia_close (%p, %d)\n", ia, ia_flags); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) { + if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) dat_status = dapl_ia_abrupt_close(ia_ptr); - } else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) { + else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) dat_status = dapl_ia_graceful_close(ia_ptr); - } else { + else dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_evd_kquery.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_kquery.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_kquery.c (working copy) @@ -38,7 +38,7 @@ #include "dapl.h" -u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) +u32 dapl_evd_kquery(struct dat_evd *evd, struct dat_evd_param *evd_param) { struct dapl_evd *evd_ptr; u32 dat_status = DAT_SUCCESS; @@ -49,16 +49,16 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h } /* Note: the spec. allows for events to be directed to a NULL EVD */ - /* with handle of type DAT_HANDLE_NULL. See 6.3.1 */ - if (DAT_HANDLE_NULL == evd_handle) { + /* with handle of type NULL. See 6.3.1 */ + if (NULL == evd) { memset(evd_param, 0, sizeof *evd_param); } else { - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; /* * We may be racing against the thread safe modify * calls here (dat_evd_{enable,disable,{set,clear}_unwaitable}). @@ -69,7 +69,7 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h * about reading the state variable atomically when we add * in waitable/unwaitable. */ - evd_param->ia_handle = evd_ptr->header.owner_ia; + evd_param->ia = (struct dat_ia *)evd_ptr->common.owner_ia; evd_param->evd_qlen = evd_ptr->qlen; evd_param->evd_flags = evd_ptr->evd_flags; if (evd_ptr->cno_ptr != NULL) { @@ -79,6 +79,6 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h } } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_openib_util.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.h (revision 2545) +++ linux-kernel/dat-provider/dapl_openib_util.h (working copy) @@ -59,7 +59,7 @@ u32 dapl_ib_qp_free(struct dapl_ia *ia_p u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, struct dat_ep_attr *ep_attr); -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data); @@ -70,12 +70,12 @@ u32 dapl_ib_setup_conn_listener(struct d u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr); -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, + struct dat_ep *ep, int private_data_size, const void *private_data); -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle); +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm); u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, DAPL_ASYNC_HANDLER_TYPE handler_type, @@ -126,14 +126,14 @@ u32 dapl_ib_completion_notify(struct dap enum dat_dto_completion_status dapl_ib_get_dto_status(struct ib_wc *cqe_ptr); -void dapl_ib_reinit_ep(struct dapl_ep * ep_ptr); +void dapl_ib_reinit_ep(struct dapl_ep *ep_ptr); -void dapl_ib_disconnect_clean(struct dapl_ep * ep_ptr, boolean_t passive); +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t passive); u32 dapl_ib_get_async_event(struct ib_event *cause_ptr, enum dat_event_number *async_event); -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address); static inline u32 dapl_ib_status_convert(int32_t ib_status) Index: linux-kernel/dat-provider/dapl_evd_qp_async_error_callb.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_qp_async_error_callb.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_qp_async_error_callb.c (working copy) @@ -55,7 +55,7 @@ void dapl_evd_qp_async_error_callback(st u32 dat_status; ep_ptr = (struct dapl_ep *)context; - async_evd = (struct dapl_evd *)ep_ptr->header.owner_ia->async_error_evd; + async_evd = (struct dapl_evd *)ep_ptr->common.owner_ia->async_error_evd; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback: " @@ -80,7 +80,7 @@ void dapl_evd_qp_async_error_callback(st * we are not interested in. */ (void)dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback () returns\n"); Index: linux-kernel/dat-provider/dapl_cr_accept.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_accept.c (revision 2545) +++ linux-kernel/dat-provider/dapl_cr_accept.c (working copy) @@ -39,42 +39,39 @@ * Establish a connection between active side (remote endpoint) * and passive side (local endpoint). */ -u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - int private_data_size, const void *private_data) +u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, + const void *private_data) { struct dapl_ep *ep_ptr; u32 dat_status; struct dapl_cr *cr_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_cr_accept (%p, %p, %d, %p)\n", - cr_handle, ep_handle, private_data_size, private_data); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_accept (%p, %p, %d, %p)\n", + cr, ep, private_data_size, private_data); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* - * Return an error if we have an ep_handle and the CR already has an + * Return an error if we have an ep and the CR already has an * EP, indicating this is an RSP connection or PSP_PROVIDER_FLAG was * specified. */ - if (ep_handle != NULL && - (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP) || - cr_ptr->param.local_ep_handle != NULL)) { + if (ep != NULL && cr_ptr->param.local_ep != NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* Make sure we have an EP handle in one place or another */ - if (ep_handle == NULL && cr_ptr->param.local_ep_handle == NULL) { + if (ep == NULL && cr_ptr->param.local_ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -86,84 +83,71 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand } /* - * ep_handle is NULL if the user specified DAT_PSP_PROVIDER_FLAG + * ep is NULL if the user specified DAT_PSP_PROVIDER_FLAG * OR this is an RSP connection; retrieve it from the cr. */ - if (ep_handle == NULL) { - ep_handle = cr_ptr->param.local_ep_handle; - if ((((struct dapl_ep *)ep_handle)->param.ep_state != + if (ep == NULL) { + ep = cr_ptr->param.local_ep; + if ((((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) - && (((struct dapl_ep *)ep_handle)->param.ep_state != + && (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING)) { return DAT_INVALID_STATE; } } else { /* ensure this EP isn't connected or in use */ - if (((struct dapl_ep *)ep_handle)->param.ep_state != + if (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_UNCONNECTED) { return DAT_INVALID_STATE; } } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required * We do all verification and state change under lock, at which * point the EP state should protect us from most races. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* The qp must be attached by this point! */ - if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) - { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } entry_ep_state = ep_ptr->param.ep_state; - entry_ep_handle = cr_ptr->param.local_ep_handle; + entry_ep = cr_ptr->param.local_ep; ep_ptr->param.ep_state = DAT_EP_STATE_COMPLETION_PENDING; - ep_ptr->cm_handle = cr_ptr->ib_cm_handle; + ep_ptr->cm = cr_ptr->ib_cm_handle; ep_ptr->cr_ptr = cr_ptr; ep_ptr->param.remote_ia_address_ptr = cr_ptr->param.remote_ia_address_ptr; - cr_ptr->param.local_ep_handle = ep_handle; + cr_ptr->param.local_ep = ep; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); - dat_status = dapl_ib_accept_connection(cr_handle, - ep_handle, + dat_status = dapl_ib_accept_connection(cr, + ep, private_data_size, private_data); /* @@ -176,16 +160,14 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand * system error, app termination, perhaps other reasons. */ dat_status = - dapl_evd_post_connection_event(ep_ptr->param. - request_evd_handle, + dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. + request_evd, DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR, - (DAT_HANDLE) ep_ptr, + (void *) ep_ptr, 0, NULL); - - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } else { ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; + cr_ptr->param.local_ep = entry_ep; ep_ptr->cr_ptr = NULL; ep_ptr->param.remote_ia_address_ptr = NULL; } @@ -199,14 +181,8 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); } - } else { - /* Make this CR invalid. We need to hang on to it until - * the connection terminates, but it's destroyed from - * the app point of view. - */ - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_evd_un_async_error_callb.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_un_async_error_callb.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_un_async_error_callb.c (working copy) @@ -65,7 +65,7 @@ void dapl_evd_un_async_error_callback(st */ dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_un_async_error_callback () returns\n"); Index: linux-kernel/dat-provider/dapl_cr_handoff.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_handoff.c (revision 2545) +++ linux-kernel/dat-provider/dapl_cr_handoff.c (working copy) @@ -37,7 +37,7 @@ * Hand the connection request to another Sevice pont specified by the * Connectin Qualifier. */ -u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) +u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL cr_handoff) { /* handoff */ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); } Index: linux-kernel/dat-provider/dapl_lmr.c =================================================================== --- linux-kernel/dat-provider/dapl_lmr.c (revision 2545) +++ linux-kernel/dat-provider/dapl_lmr.c (working copy) @@ -36,7 +36,7 @@ static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_desc, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags mem_priv) { struct dapl_lmr *lmr; @@ -49,27 +49,22 @@ static struct dapl_lmr *dapl_lmr_alloc(s /* zero the structure */ memset(lmr, 0, sizeof *lmr); - /* - * initialize the header - */ - lmr->header.provider = ia->header.provider; - lmr->header.magic = DAPL_MAGIC_LMR; - lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; - lmr->header.owner_ia = ia; - lmr->header.user_context.as_64 = 0; - lmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&lmr->header.ia_list_entry); + lmr->lmr.provider = ia->ia.provider; + lmr->common.owner_ia = ia; + lmr->common.user_context.as_64 = 0; + lmr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&lmr->common.ia_list_entry); dapl_ia_link_lmr(ia, lmr); - spin_lock_init(&lmr->header.lock); + spin_lock_init(&lmr->common.lock); /* * initialize the body */ - lmr->param.ia_handle = (DAT_IA_HANDLE) ia; + lmr->param.ia = (struct dat_ia *)ia; lmr->param.mem_type = mem_type; lmr->param.region_desc = region_desc; lmr->param.length = length; - lmr->param.pz_handle = pz_handle; + lmr->param.pz = pz; lmr->param.mem_priv = mem_priv; atomic_set(&lmr->lmr_ref_count, 0); @@ -78,9 +73,7 @@ static struct dapl_lmr *dapl_lmr_alloc(s static void dapl_lmr_dealloc(struct dapl_lmr *lmr) { - /* reset magic to prevent reuse */ - lmr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); + dapl_ia_unlink_lmr(lmr->common.owner_ia, lmr); kfree(lmr); } @@ -88,49 +81,50 @@ static void dapl_lmr_dealloc(struct dapl static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, u64 length, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; reg_desc.for_va = virt_addr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, + (struct dat_pz *)pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register(ia, lmr, virt_addr, length, privileges); + status = dapl_ib_mr_register(ia, new_lmr, virt_addr, length, + privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64)(unsigned long) virt_addr; if (registered_length) @@ -139,9 +133,9 @@ static inline u32 dapl_lmr_create_virtua return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; @@ -151,26 +145,26 @@ static inline u32 dapl_lmr_create_physic DAT_REGION_DESCRIPTION phys_addr, u64 page_count, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; u64 *array = phys_addr.for_array; u32 status; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, page_count, - (DAT_PZ_HANDLE) pz, privileges); + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, + page_count, (struct dat_pz *) pz, privileges); - if (NULL == lmr) { + if (NULL == new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, + status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr.for_array, page_count, privileges); if (DAT_SUCCESS != status) @@ -178,23 +172,23 @@ static inline u32 dapl_lmr_create_physic /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = array[0]; if (registered_length) @@ -203,9 +197,9 @@ static inline u32 dapl_lmr_create_physic return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } @@ -214,13 +208,13 @@ static inline u32 dapl_lmr_create_lmr(st struct dapl_lmr *original_lmr, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; @@ -230,39 +224,39 @@ static inline u32 dapl_lmr_create_lmr(st if (status != DAT_SUCCESS) goto error1; - reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; + reg_desc.for_lmr = (struct dat_lmr *) original_lmr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, + (struct dat_pz *) pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_shared(ia, lmr, privileges); + status = dapl_ib_mr_register_shared(ia, new_lmr, privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64) (unsigned long) original_lmr->param.region_desc.for_va; @@ -272,73 +266,72 @@ static inline u32 dapl_lmr_create_lmr(st return DAT_SUCCESS; error3: - dapl_ib_mr_deregister(lmr); + dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } -u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags privileges, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - u64 *registered_length, u64 *registered_address) + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, + u64 *registered_address) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", - ia_handle, mem_type); + ia, mem_type); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - ia = (struct dapl_ia *)ia_handle; - pz = (struct dapl_pz *)pz_handle; + dapl_ia = (struct dapl_ia *)ia; + dapl_pz = (struct dapl_pz *)pz; switch (mem_type) { case DAT_MEM_TYPE_VIRTUAL: - status = dapl_lmr_create_virtual(ia, region_description.for_va, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_virtual(dapl_ia, + region_description.for_va, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_PHYSICAL: - status = dapl_lmr_create_physical(ia, region_description, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_physical(dapl_ia, region_description, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_LMR: { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; - if (DAPL_BAD_HANDLE(region_description.for_lmr_handle, - DAPL_MAGIC_LMR)) { + if (!region_description.for_lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto bail; } - lmr = (struct dapl_lmr *)region_description.for_lmr_handle; + dapl_lmr = (struct dapl_lmr *)region_description.for_lmr; - status = dapl_lmr_create_lmr(ia, lmr, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_lmr(dapl_ia, dapl_lmr, dapl_pz, + privileges, lmr, lmr_context, rmr_context, registered_length, registered_address); break; @@ -357,45 +350,45 @@ bail: return status; } -u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) +u32 dapl_lmr_free(struct dat_lmr *lmr) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } - lmr = (struct dapl_lmr *)lmr_handle; + dapl_lmr = (struct dapl_lmr *)lmr; - switch (lmr->param.mem_type) { + switch (dapl_lmr->param.mem_type) { case DAT_MEM_TYPE_PHYSICAL: case DAT_MEM_TYPE_VIRTUAL: case DAT_MEM_TYPE_LMR: { struct dapl_pz *pz; - if (0 != atomic_read(&lmr->lmr_ref_count)) + if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) return DAT_INVALID_STATE; status = dapl_hash_remove( - lmr->header.owner_ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + dapl_lmr->common.owner_ia->hca_ptr->lmr_hash_table, + dapl_lmr->param.lmr_context, NULL); if (status != DAT_SUCCESS) goto error; - status = dapl_ib_mr_deregister(lmr); + status = dapl_ib_mr_deregister(dapl_lmr); if (status == DAT_SUCCESS) { - pz = (struct dapl_pz *)lmr->param.pz_handle; + pz = (struct dapl_pz *)dapl_lmr->param.pz; atomic_dec(&pz->pz_ref_count); - dapl_lmr_dealloc(lmr); - } else /* failure; put lmr back in hash table */ - dapl_hash_insert(lmr->header.owner_ia-> + dapl_lmr_dealloc(dapl_lmr); + } else /* failure; put dapl_lmr back in hash table */ + dapl_hash_insert(dapl_lmr->common.owner_ia-> hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + dapl_lmr->param.lmr_context, dapl_lmr); break; } @@ -412,15 +405,15 @@ error: return status; } -u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) +u32 dapl_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *lmr_param) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", - lmr_handle, lmr_param); + lmr, lmr_param); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } @@ -429,8 +422,8 @@ u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_ha goto error; } - lmr = (struct dapl_lmr *)lmr_handle; - memcpy(lmr_param, &lmr->param, sizeof *lmr_param); + dapl_lmr = (struct dapl_lmr *)lmr; + memcpy(lmr_param, &dapl_lmr->param, sizeof *lmr_param); status = DAT_SUCCESS; error: return status; @@ -441,7 +434,7 @@ error: * will sync the cache with memory. */ -u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -449,11 +442,11 @@ u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDL u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_read(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } @@ -462,7 +455,7 @@ error: return status; } -u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -470,11 +463,11 @@ u32 dapl_lmr_sync_rdma_write(DAT_IA_HAND u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_write(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } Index: linux-kernel/dat-provider/dapl_get_handle_type.c =================================================================== --- linux-kernel/dat-provider/dapl_get_handle_type.c (revision 2545) +++ linux-kernel/dat-provider/dapl_get_handle_type.c (working copy) @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_get_handle_type - * - * Gets the handle type for the given dat_handle - * - * Input: - * dat_handle - * - * Output: - * handle_type - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ - -u32 dapl_get_handle_type(DAT_HANDLE dat_handle, enum dat_handle_type *type) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - *type = header->handle_type; - -bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_rmr.c =================================================================== --- linux-kernel/dat-provider/dapl_rmr.c (revision 2545) +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) @@ -61,24 +61,19 @@ static struct dapl_rmr *dapl_rmr_alloc(s /* zero the structure */ memset(rmr, 0, sizeof *rmr); - /* - * initialize the header - */ - rmr->header.provider = pz->header.provider; - rmr->header.magic = DAPL_MAGIC_RMR; - rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; - rmr->header.owner_ia = pz->header.owner_ia; - rmr->header.user_context.as_64 = 0; - rmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&rmr->header.ia_list_entry); - dapl_ia_link_rmr(rmr->header.owner_ia, rmr); - spin_lock_init(&rmr->header.lock); + rmr->rmr.provider = pz->pz.provider; + rmr->common.owner_ia = pz->common.owner_ia; + rmr->common.user_context.as_64 = 0; + rmr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&rmr->common.ia_list_entry); + dapl_ia_link_rmr(rmr->common.owner_ia, rmr); + spin_lock_init(&rmr->common.lock); /* * initialize the body */ - rmr->param.ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; - rmr->param.pz_handle = (DAT_PZ_HANDLE)pz; + rmr->param.ia = (struct dat_ia *)pz->common.owner_ia; + rmr->param.pz = (struct dat_pz *)pz; rmr->param.lmr_triplet.lmr_context = 0; rmr->param.lmr_triplet.pad = 0; rmr->param.lmr_triplet.virtual_address = 0; @@ -93,11 +88,8 @@ static struct dapl_rmr *dapl_rmr_alloc(s static void dapl_rmr_dealloc(struct dapl_rmr *rmr) { - /* reset magic to prevent reuse */ - rmr->header.magic = DAPL_MAGIC_INVALID; - - dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); - /* no need to destroy rmr->header.lock */ + dapl_ia_unlink_rmr(rmr->common.owner_ia, rmr); + /* no need to destroy rmr->common.lock */ kfree(rmr); } @@ -115,7 +107,7 @@ static inline u32 dapl_rmr_bind_fuse(str u32 status; boolean_t is_signaled; - status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, + status = dapl_hash_search(rmr->common.owner_ia->hca_ptr->lmr_hash_table, lmr_triplet->lmr_context, (DAPL_HASH_DATA *) &lmr); if (DAT_SUCCESS != status) { @@ -150,8 +142,8 @@ static inline u32 dapl_rmr_bind_fuse(str } /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ - if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || - (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { + if ((ep_ptr->param.pz != lmr->param.pz) || + (ep_ptr->param.pz != rmr->param.pz)) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -240,7 +232,7 @@ static inline u32 dapl_rmr_bind_unfuse(s } /* If the RMR and EP are not in the same PZ, there is an error */ - if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { + if (ep_ptr->param.pz != rmr->param.pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail1; } @@ -306,9 +298,9 @@ bail1: * Input: * Output: */ -u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, const struct dat_lmr_triplet *lmr_triplet, - enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, + enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) @@ -316,14 +308,14 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han struct dapl_rmr *rmr; struct dapl_ep *ep_ptr; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) + if (!rmr_handle) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) + if (!ep) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); rmr = (struct dapl_rmr *)rmr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* if the rmr should be bound */ if (0 != lmr_triplet->segment_length) @@ -341,7 +333,7 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * Create a remote memory region for the specified protection zone * * Input: - * pz_handle + * pz * * Output: * rmr_handle @@ -351,20 +343,20 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) +u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr) { - struct dapl_pz *pz; - struct dapl_rmr *rmr; + struct dapl_pz *dapl_pz; + struct dapl_rmr *dapl_rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - rmr = dapl_rmr_alloc(pz); + dapl_rmr = dapl_rmr_alloc(dapl_pz); if (rmr == NULL) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, @@ -372,18 +364,18 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han goto bail; } - status = dapl_ib_mw_alloc(rmr); + status = dapl_ib_mw_alloc(dapl_rmr); if (status != DAT_SUCCESS) { - dapl_rmr_dealloc(rmr); + dapl_rmr_dealloc(dapl_rmr); status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY_REGION); goto bail; } - atomic_inc(&pz->pz_ref_count); + atomic_inc(&dapl_pz->pz_ref_count); - *rmr_handle = rmr; + *rmr = (struct dat_rmr *)dapl_rmr; bail: return status; @@ -404,12 +396,12 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) +u32 dapl_rmr_free(struct dat_rmr *rmr_handle) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } @@ -438,12 +430,12 @@ bail: return status; } -u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) +u32 dapl_rmr_query(struct dat_rmr *rmr_handle, struct dat_rmr_param *rmr_param) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2545) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -43,11 +43,6 @@ static void dapl_ep_dealloc(struct dapl_ep *ep_ptr) { - dapl_os_assert(ep_ptr->header.magic == DAPL_MAGIC_EP); - - /* reset magic to prevent reuse */ - ep_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_cb_free(&ep_ptr->req_buffer); dapl_cb_free(&ep_ptr->recv_buffer); @@ -73,18 +68,12 @@ struct dapl_ep *dapl_ep_alloc(struct dap /* zero the structure */ memset(ep_ptr, 0, sizeof *ep_ptr + sizeof (struct sockaddr)); - /* - * initialize the header - */ - ep_ptr->header.provider = ia_ptr->header.provider; - ep_ptr->header.magic = DAPL_MAGIC_EP; - ep_ptr->header.handle_type = DAT_HANDLE_TYPE_EP; - ep_ptr->header.owner_ia = ia_ptr; - ep_ptr->header.user_context.as_64 = 0; - ep_ptr->header.user_context.as_ptr = NULL; - - dapl_llist_init_entry(&ep_ptr->header.ia_list_entry); - spin_lock_init(&ep_ptr->header.lock); + ep_ptr->ep.provider = ia_ptr->ia.provider; + ep_ptr->common.owner_ia = ia_ptr; + ep_ptr->common.user_context.as_64 = 0; + ep_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&ep_ptr->common.ia_list_entry); + spin_lock_init(&ep_ptr->common.lock); /* * Initialize the body @@ -105,7 +94,7 @@ struct dapl_ep *dapl_ep_alloc(struct dap ep_ptr->qp = NULL; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - ep_ptr->cm_handle = NULL; + ep_ptr->cm = NULL; atomic_set(&ep_ptr->req_count, 0); atomic_set(&ep_ptr->recv_count, 0); @@ -165,7 +154,7 @@ static u32 dapl_ep_check_recv_completion return DAT_SUCCESS; } -static u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, +static u32 dapl_ep_post_send_req(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -177,13 +166,13 @@ static u32 dapl_ep_post_send_req(DAT_EP_ struct dapl_cookie *cookie; u32 dat_status; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Synchronization ok since this buffer is only used for send @@ -281,11 +270,11 @@ enum dat_return_subtype dapl_ep_state_su return dat_status; } -u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) +u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + const struct dat_ep_attr *ep_attr, struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -294,18 +283,18 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create (%p, %p, %p, %p, %p, %p, %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -317,44 +306,34 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != DAT_HANDLE_NULL && - DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -370,12 +349,12 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -413,25 +392,25 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (recv_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_recv_dtos = 0; - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (request_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_request_dtos = 0; /* @@ -440,9 +419,9 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -450,7 +429,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -460,36 +439,36 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free - * evd_handles: + * pz: dat_pz_free + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = (struct dat_ep *)ep_ptr; bail: return dat_status; } -u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +u32 dapl_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *ep_attr, - DAT_EP_HANDLE *ep_handle) + struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -498,18 +477,18 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create_with_srq (%p, %p, %p, %p, %p, %p, %p %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, srq_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, srq, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -521,35 +500,27 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)-> + evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); @@ -560,13 +531,13 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * Verify the SRQ handle. It is an error to invoke this call with * a NULL handle */ - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -582,12 +553,12 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -636,26 +607,26 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (recv_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_recv_dtos = 0; } - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (request_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_request_dtos = 0; } @@ -665,9 +636,9 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -675,7 +646,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -685,24 +656,24 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free, - * evd_handles: + * pz: dat_pz_free, + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = ep_ptr; #endif /* notdef */ @@ -710,7 +681,7 @@ bail: return dat_status; } -u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_free(struct dat_ep *ep) { struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; @@ -718,15 +689,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle enum ib_qp_state save_qp_state; u32 dat_status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; param = &ep_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -744,7 +715,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle goto bail; } - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* If we are connected, issue a disconnect. If we are in the * disconnect_pending state, disconnect with the ABRUPT flag @@ -754,14 +725,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle /* * Invoke ep_disconnect to clean up outstanding connections */ - (void)dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + (void)dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); dapl_os_assert(ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_UNCONNECTED); /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); @@ -771,24 +743,24 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle * to set handles to NULL. Set handles to NULL so this routine * is idempotent. */ - if (param->pz_handle != NULL) { - atomic_dec(&((struct dapl_pz *)param->pz_handle)->pz_ref_count); - param->pz_handle = NULL; + if (param->pz != NULL) { + atomic_dec(&((struct dapl_pz *)param->pz)->pz_ref_count); + param->pz = NULL; } - if (param->recv_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->recv_evd_handle)-> + if (param->recv_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->recv_evd)-> evd_ref_count); - param->recv_evd_handle = NULL; + param->recv_evd = NULL; } - if (param->request_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->request_evd_handle)-> + if (param->request_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->request_evd)-> evd_ref_count); - param->request_evd_handle = NULL; + param->request_evd = NULL; } - if (param->connect_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->connect_evd_handle)-> + if (param->connect_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->connect_evd)-> evd_ref_count); - param->connect_evd_handle = NULL; + param->connect_evd = NULL; } /* @@ -804,7 +776,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ save_qp_state = ep_ptr->qp_state; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Free the QP. If the EP has never been used, the QP is invalid */ if (save_qp_state != DAPL_QP_STATE_UNATTACHED) { @@ -828,7 +800,7 @@ bail: } -u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, @@ -839,7 +811,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_connect(%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", - ep_handle, + ep, remote_ia_address->sa_data[2], remote_ia_address->sa_data[3], remote_ia_address->sa_data[4], @@ -847,19 +819,19 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han remote_conn_qual, timeout, private_data_size, private_data, qos, connect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state. The connection handle must be good * at this point. */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD)) { + if (!ep_ptr->param.connect_evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; @@ -889,9 +861,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * * Assign the QP and release the lock. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ)) - { + if (ep_ptr->param.pz == NULL) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); @@ -899,7 +869,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } alloc_ep = *ep_ptr; - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, &alloc_ep, ep_ptr); if (dat_status != DAT_SUCCESS) { dat_status = @@ -908,16 +878,16 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han goto bail; } - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * PZ shouldn't have changed since we're only racing with * dapl_cr_accept() */ if (ep_ptr->qp_state != DAPL_QP_STATE_UNATTACHED) { /* Bail, cleaning up. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); - dat_status = dapl_ib_qp_free(ep_ptr->header.owner_ia, + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); + dat_status = dapl_ib_qp_free(ep_ptr->common.owner_ia, &alloc_ep); if (dat_status != DAT_SUCCESS) { dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -933,42 +903,30 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->qp = alloc_ep.qp; ep_ptr->qp_state = alloc_ep.qp_state; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); } /* * We do state checks and transitions under lock. * The only code we're racing against is dapl_cr_accept. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required - */ - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL - && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required + */ + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); goto bail; @@ -978,12 +936,12 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * we need to attach one now. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_TEP); @@ -992,8 +950,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1004,8 +962,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han /* * At this point we only support one QOS level */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_MODEL_NOT_SUPPORTED, 0); goto bail; } @@ -1019,7 +977,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * At this point we're committed, and done with the endpoint * except for the connect, so we can drop the lock. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Copy the connection qualifiers */ memcpy(ep_ptr->param.remote_ia_address_ptr, remote_ia_address, @@ -1027,7 +985,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->param.remote_port_qual = remote_conn_qual; memcpy(ep_ptr->private_data,private_data,private_data_size); - dat_status = dapl_ib_connect(ep_ptr, remote_ia_address, + dat_status = dapl_ib_connect((struct dat_ep *)ep_ptr, remote_ia_address, remote_conn_qual, timeout, private_data_size, ep_ptr->private_data); @@ -1044,10 +1002,10 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han if (dat_status == DAT_ERROR(DAT_INTERNAL_ERROR, 1)) { dapl_evd_post_connection_event((struct dapl_evd *) ep_ptr->param. - connect_evd_handle, + connect_evd, DAT_CONNECTION_EVENT_UNREACHABLE, - (DAT_HANDLE) ep_ptr, 0, - NULL); + (struct dat_ep *)ep_ptr, + 0, NULL); dat_status = DAT_SUCCESS; } } @@ -1059,7 +1017,7 @@ bail: return dat_status; } -u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, +u32 dapl_ep_dup_connect(struct dat_ep *ep, struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { @@ -1068,13 +1026,13 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep DAT_CONN_QUAL remote_conn_qual; u32 dat_status; - ep_dup_ptr = (struct dapl_ep *)ep_dup_handle; + ep_dup_ptr = (struct dapl_ep *)ep_dup; /* * Verify the dup handle, which must be connected. All other * parameters will be verified by dapl_ep_connect */ - if (DAPL_BAD_HANDLE(ep_dup_handle, DAPL_MAGIC_EP)) { + if (!ep_dup) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1090,10 +1048,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep * there is a problem. Do this under a lock and pull out * the connection parameters for atomicity. */ - spin_lock_irqsave(&ep_dup_ptr->header.lock, ep_dup_ptr->header.flags); + spin_lock_irqsave(&ep_dup_ptr->common.lock, ep_dup_ptr->common.flags); if (ep_dup_ptr->param.ep_state != DAT_EP_STATE_CONNECTED) { - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_dup_ptr)); @@ -1101,10 +1059,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep } remote_ia_address_ptr = ep_dup_ptr->param.remote_ia_address_ptr; remote_conn_qual = ep_dup_ptr->param.remote_port_qual; - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); - dat_status = dapl_ep_connect(ep_handle, + dat_status = dapl_ep_connect(ep, remote_ia_address_ptr, remote_conn_qual, timeout, @@ -1115,7 +1073,7 @@ bail: return dat_status; } -u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags disconnect_flags) { struct dapl_ep *ep_ptr; @@ -1125,14 +1083,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_disconnect (%p, %x)\n", - ep_handle, disconnect_flags); + ep, disconnect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1142,12 +1100,12 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * Do the verification of parameters and the state change * atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Disconnecting a disconnected EP is a no-op. */ if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; goto bail; } @@ -1161,8 +1119,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ ep_ptr->param.ep_state != DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECT_PENDING) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1174,8 +1132,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * If in state DISCONNECT_PENDING then this must be an * ABRUPT disconnect */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } @@ -1188,18 +1146,19 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ */ ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); /* disconnect and make sure we get no callbacks */ (void)dapl_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); /* clean up connection state */ dapl_sp_remove_ep(ep_ptr); - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; dapl_evd_post_connection_event(evd_ptr, DAT_CONNECTION_EVENT_DISCONNECTED, - (DAT_HANDLE) ep_ptr, 0, NULL); + (struct dat_ep *)ep_ptr, 0, + NULL); dat_status = DAT_SUCCESS; goto bail; } @@ -1217,14 +1176,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECT_PENDING; } - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); dat_status = dapl_ib_disconnect(ep_ptr, disconnect_flags); /* * Reacquire the lock and make sure we didn't get a callback * that cleaned up. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (disconnect_flags == DAT_CLOSE_ABRUPT_FLAG && ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { /* @@ -1235,8 +1194,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * on the way out. */ cr_ptr = ep_ptr->cr_ptr; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); if (cr_ptr != NULL) { dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, @@ -1247,13 +1206,13 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ DAT_CONNECTION_EVENT_DISCONNECTED, NULL, cr_ptr->sp_ptr); } else { - dapl_evd_connection_callback(ep_ptr->cm_handle, + dapl_evd_connection_callback(ep_ptr->cm, DAT_CONNECTION_EVENT_DISCONNECTED, NULL, ep_ptr); } } else - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, @@ -1262,7 +1221,7 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ return dat_status; } -u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, enum dat_ep_state *ep_state, +u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle) { struct dapl_ep *ep_ptr; @@ -1270,14 +1229,14 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_get_status (%p, %p, %p, %p)\n", - ep_handle, ep_state, in_dto_idle, out_dto_idle); + ep, ep_state, in_dto_idle, out_dto_idle); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1305,15 +1264,15 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ } static inline u32 -dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, +dapl_ep_modify_validate_parameters(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param, struct dapl_ia **ia_ptr, struct dapl_ep **ep_ptr, struct dat_ep_attr *ep_attr_ptr) { - struct dapl_ia *ia; - struct dapl_ep *ep; + struct dapl_ia *dapl_ia; + struct dapl_ep *dapl_ep; struct dat_ep_attr ep_attr; struct dat_ep_attr ep_attr_limit; struct dat_ep_attr ep_attr_request; @@ -1322,14 +1281,14 @@ dapl_ep_modify_validate_parameters(DAT_E *ia_ptr = NULL; *ep_ptr = NULL; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep = (struct dapl_ep *)ep_handle; - ia = ep->header.owner_ia; + dapl_ep = (struct dapl_ep *)ep; + dapl_ia = dapl_ep->common.owner_ia; /* * Verify parameters valid in current EP state @@ -1349,10 +1308,10 @@ dapl_ep_modify_validate_parameters(DAT_E * TENTATIVE_CONNECTION_PENDING (psp PROVIDER allocated EP) */ if ((mask & DAT_EP_FIELD_PZ_HANDLE) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } @@ -1368,41 +1327,21 @@ dapl_ep_modify_validate_parameters(DAT_E DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS | DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV | DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV)) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_RESERVED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_RESERVED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } /* * Validate handles being modified */ - if (mask & DAT_EP_FIELD_PZ_HANDLE) { - if (ep_param->pz_handle != NULL && - DAPL_BAD_HANDLE(ep_param->pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { - if (ep_param->recv_evd_handle != NULL && - (DAPL_BAD_HANDLE(ep_param->recv_evd_handle, DAPL_MAGIC_EVD) - || !((struct dapl_evd *)ep_param->recv_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - - if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) { - if (ep_param->request_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->request_evd_handle, - DAPL_MAGIC_EVD)) { + if (ep_param->recv_evd != NULL && + !((struct dapl_evd *)ep_param->recv_evd)-> + evd_flags & DAT_EVD_DTO_FLAG) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; @@ -1410,10 +1349,8 @@ dapl_ep_modify_validate_parameters(DAT_E } if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) { - if (ep_param->connect_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->connect_evd_handle, - DAPL_MAGIC_EVD) - && !(((struct dapl_evd *)ep_param->connect_evd_handle)-> + if (ep_param->connect_evd != NULL && + !(((struct dapl_evd *)ep_param->connect_evd)-> evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1424,8 +1361,8 @@ dapl_ep_modify_validate_parameters(DAT_E /* * Validate the attributes against the HCA limits */ - ep_attr = ep->param.ep_attr; - ep_attr_limit = ia->hca_ptr->ep_attr; + ep_attr = dapl_ep->param.ep_attr; + ep_attr_limit = dapl_ia->hca_ptr->ep_attr; ep_attr_request = ep_param->ep_attr; if (mask & DAT_EP_FIELD_EP_ATTR_MAX_MESSAGE_SIZE) { @@ -1472,7 +1409,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS) { if (ep_attr_request.max_recv_dtos > ep_attr_limit.max_recv_dtos - || (ep_param->recv_evd_handle == DAT_HANDLE_NULL + || (ep_param->recv_evd == NULL && (ep_attr_request.max_recv_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1485,7 +1422,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS) { if (ep_attr_request.max_request_dtos > ep_attr_limit.max_request_dtos - || (ep_param->request_evd_handle == DAT_HANDLE_NULL + || (ep_param->request_evd == NULL && (ep_attr_request.max_request_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1518,8 +1455,8 @@ dapl_ep_modify_validate_parameters(DAT_E } } - *ia_ptr = ia; - *ep_ptr = ep; + *ia_ptr = dapl_ia; + *ep_ptr = dapl_ep; *ep_attr_ptr = ep_attr; bail: @@ -1535,7 +1472,7 @@ dapl_ep_modify_validate_parameters(DAT_E * the Endpoint. * * Input: - * ep_handle + * ep * ep_args_mask * * Output: @@ -1547,7 +1484,7 @@ dapl_ep_modify_validate_parameters(DAT_E * DAT_INVALID_ATTRIBUTE * DAT_INVALID_STATE */ -u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, +u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param) { struct dapl_ia *ia; @@ -1573,7 +1510,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand boolean_t rqst_iov_used = FALSE; boolean_t recv_iov_used = FALSE; - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep1, &ep_attr1); @@ -1587,16 +1524,16 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep = *ep1; alloc_ep.param.ep_attr = ep_attr1; if (mask & DAT_EP_FIELD_PZ_HANDLE) - alloc_ep.param.pz_handle = ep_param->pz_handle; + alloc_ep.param.pz = ep_param->pz; if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) - alloc_ep.param.recv_evd_handle = ep_param->recv_evd_handle; + alloc_ep.param.recv_evd = ep_param->recv_evd; if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) - alloc_ep.param.request_evd_handle = ep_param->request_evd_handle; + alloc_ep.param.request_evd = ep_param->request_evd; if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) - alloc_ep.param.connect_evd_handle = ep_param->connect_evd_handle; + alloc_ep.param.connect_evd = ep_param->connect_evd; /* * Allocate everything that might be needed. @@ -1649,18 +1586,18 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * There's a lot of work done here, but there should be no * allocation or blocking. */ - spin_lock_irqsave(&ep1->header.lock, ep1->header.flags); + spin_lock_irqsave(&ep1->common.lock, ep1->common.flags); /* * Revalidate parameters; make sure that races haven't * changed anything important. */ - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep2, &ep_attr2); if (DAT_SUCCESS != dat_status) { - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); goto bail; } @@ -1689,10 +1626,10 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * is stuff changed by dapl_cr_accept, and neither PZ nor EVD is in that * list. */ - new_ep.param.pz_handle = alloc_ep.param.pz_handle; - new_ep.param.recv_evd_handle = alloc_ep.param.recv_evd_handle; - new_ep.param.request_evd_handle = alloc_ep.param.request_evd_handle; - new_ep.param.connect_evd_handle = alloc_ep.param.connect_evd_handle; + new_ep.param.pz = alloc_ep.param.pz; + new_ep.param.recv_evd = alloc_ep.param.recv_evd; + new_ep.param.request_evd = alloc_ep.param.request_evd; + new_ep.param.connect_evd = alloc_ep.param.connect_evd; /* Deal with each of the allocation fields. */ if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS @@ -1754,50 +1691,50 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand if (DAPL_QP_STATE_UNATTACHED != new_ep.qp_state) { dat_status = dapl_ib_qp_modify(ia, ep2, &ep_attr2); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep2->header.lock, - ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, + ep2->common.flags); goto bail; } } *ep2 = new_ep; - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); /* * Modify reference counts, incrementing new ones * and then decrementing old ones (so if they're the same * the refcount never drops to zero). */ - tmp_pz = (struct dapl_pz *)new_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)new_ep.param.pz; if (tmp_pz) atomic_inc(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.request_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); /* decreament the old reference counts */ - tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz; if (tmp_pz) atomic_dec(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); @@ -1845,7 +1782,7 @@ bail: return dat_status; } -u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1856,12 +1793,12 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_read (%p, %d, %p, %p, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1876,7 +1813,7 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE return dat_status; } -u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1886,12 +1823,12 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_write (%p, %d, %p, %P, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1906,7 +1843,7 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL return dat_status; } -u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1917,15 +1854,15 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_recv (%p, %d, %p, %P, %x)\n", - ep_handle, num_segments, local_iov, user_cookie.as_64, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *) ep_handle; + ep_ptr = (struct dapl_ep *) ep; /* * Synchronization ok since this buffer is only used for receive @@ -1955,7 +1892,7 @@ bail: return status; } -u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1965,11 +1902,11 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_send (%p, %d, %p, %P, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1983,21 +1920,21 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h return dat_status; } -u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) +u32 dapl_ep_query(struct dat_ep *ep, struct dat_ep_param *ep_param) { struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_query (%p, %p)\n", - ep_handle, ep_param); + ep, ep_param); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } @@ -2013,12 +1950,10 @@ u32 dapl_ep_query(DAT_EP_HANDLE ep_handl * N.B. Just slam all values into the user structure, there * is nothing to be gained by checking for each bit. */ - if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { + if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) /* obtain the remote IP address */ - status = - dapl_ib_cm_remote_addr((DAT_HANDLE) ep_handle, - &ep_ptr->remote_ia_address); - } + status = dapl_ib_cm_remote_addr(ep, &ep_ptr->remote_ia_address); + ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *)&ep_ptr->remote_ia_address; *ep_param = ep_ptr->param; @@ -2027,21 +1962,21 @@ bail: return status; } -u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, int *nbufs_allocate, +u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocate, int *bufs_alloc_span) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", - ep_handle, nbufs_allocate, bufs_alloc_span); + ep, nbufs_allocate, bufs_alloc_span); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2054,17 +1989,17 @@ bail: } -u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_reset(struct dat_ep *ep) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2086,7 +2021,7 @@ bail: return dat_status; } -u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -2094,14 +2029,14 @@ u32 dapl_ep_set_watermark(DAT_EP_HANDLE u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", - ep_handle, soft_high_watermark, hard_high_watermark); + ep, soft_high_watermark, hard_high_watermark); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; Index: linux-kernel/dat-provider/dapl_ia_open.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_open.c (revision 2545) +++ linux-kernel/dat-provider/dapl_ia_open.c (working copy) @@ -45,18 +45,18 @@ * and calls this function. */ u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr) + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr) { u32 dat_status = DAT_SUCCESS; struct dat_provider *provider; struct dapl_hca *hca_ptr = NULL; - struct dapl_ia *ia_ptr = NULL; - struct dapl_evd *evd_ptr; + struct dapl_ia *ia = NULL; + struct dapl_evd *evd; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_open (%s, %d, %p, %p)\n", - name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); + name, async_evd_qlen, async_evd_ptr, ia_ptr); dat_status = dapl_provider_list_search(name, &provider); if (DAT_SUCCESS != dat_status) { @@ -64,25 +64,25 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - /* ia_handle_ptr and async_evd_handle_ptr cannot be NULL */ - if (ia_handle_ptr == NULL) { + /* ia_ptr and async_evd_ptr cannot be NULL */ + if (ia_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } - if (async_evd_handle_ptr == NULL) { + if (async_evd_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; } /* initialize the caller's param */ - *ia_handle_ptr = DAT_HANDLE_NULL; + *ia_ptr = NULL; hca_ptr = (struct dapl_hca *)provider->extension; atomic_inc(&hca_ptr->handle_ref_count); /* Allocate and initialize ia structure */ - ia_ptr = dapl_ia_alloc(provider, hca_ptr); - if (!ia_ptr) { + ia = dapl_ia_alloc(provider, hca_ptr); + if (!ia) { atomic_dec(&hca_ptr->handle_ref_count); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -93,17 +93,16 @@ u32 dapl_ia_open(const char *name, int a * use the one passed in (if non-NULL) or create one */ - evd_ptr = (struct dapl_evd *)*async_evd_handle_ptr; - if (evd_ptr) { - if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || - !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { + evd = (struct dapl_evd *)*async_evd_ptr; + if (evd) { + if (!(evd->evd_flags & DAT_EVD_ASYNC_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - if (evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle != + if (evd->common.owner_ia->hca_ptr->ib_hca_handle != hca_ptr->ib_hca_handle) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, @@ -111,8 +110,8 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - ia_ptr->cleanup_async_error_evd = FALSE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = FALSE; + ia->async_error_evd = evd; } else { /* Verify we have >0 length, and let the provider check the size */ if (async_evd_qlen <= 0) { @@ -120,33 +119,32 @@ u32 dapl_ia_open(const char *name, int a DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } - dat_status = dapl_evd_internal_create(ia_ptr, - NULL, /* CNO ptr */ + dat_status = dapl_evd_internal_create(ia, NULL, /* CNO ptr */ async_evd_qlen, DAT_EVD_ASYNC_FLAG, - &evd_ptr); + &evd); if (dat_status != DAT_SUCCESS) goto bail; - atomic_inc(&evd_ptr->evd_ref_count); + atomic_inc(&evd->evd_ref_count); /* Register the handlers associated with the async EVD. */ - dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); + dat_status = dapl_ia_setup_callbacks(ia, evd); /* Assign the EVD so it gets cleaned up */ - ia_ptr->cleanup_async_error_evd = TRUE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = TRUE; + ia->async_error_evd = evd; if (dat_status != DAT_SUCCESS) goto bail; } dat_status = DAT_SUCCESS; - *ia_handle_ptr = ia_ptr; - *async_evd_handle_ptr = evd_ptr; + *ia_ptr = (struct dat_ia *)ia; + *async_evd_ptr = (struct dat_evd *)evd; - bail: - if (dat_status != DAT_SUCCESS) { - if (ia_ptr) - dapl_ia_close(ia_ptr, DAT_CLOSE_ABRUPT_FLAG); - } +bail: + if (dat_status != DAT_SUCCESS) + if (ia) + dapl_ia_close((struct dat_ia *)ia, + DAT_CLOSE_ABRUPT_FLAG); dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_ia_open () returns 0x%x\n", dat_status); Index: linux-kernel/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_util.c (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_util.c (working copy) @@ -131,17 +131,12 @@ struct dapl_evd *dapl_evd_alloc(struct d /* zero the structure */ memset(evd_ptr, 0, sizeof *evd_ptr); - /* - * initialize the header - */ - evd_ptr->header.provider = ia_ptr->header.provider; - evd_ptr->header.magic = DAPL_MAGIC_EVD; - evd_ptr->header.handle_type = DAT_HANDLE_TYPE_EVD; - evd_ptr->header.owner_ia = ia_ptr; - evd_ptr->header.user_context.as_64 = 0; - evd_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&evd_ptr->header.ia_list_entry); - spin_lock_init(&evd_ptr->header.lock); + evd_ptr->evd.provider = ia_ptr->ia.provider; + evd_ptr->common.owner_ia = ia_ptr; + evd_ptr->common.user_context.as_64 = 0; + evd_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&evd_ptr->common.ia_list_entry); + spin_lock_init(&evd_ptr->common.lock); /* * Initialize the body @@ -257,7 +252,6 @@ u32 dapl_evd_dealloc(struct dapl_evd *ev u32 dat_status = DAT_SUCCESS; struct dapl_ia *ia_ptr; - dapl_os_assert(evd_ptr->header.magic == DAPL_MAGIC_EVD); dapl_os_assert(atomic_read(&evd_ptr->evd_ref_count) == 0); /* @@ -265,7 +259,7 @@ u32 dapl_evd_dealloc(struct dapl_evd *ev * up from it. */ if (evd_ptr->cq != NULL) { - ia_ptr = evd_ptr->header.owner_ia; + ia_ptr = evd_ptr->common.owner_ia; dat_status = dapl_ib_cq_free(ia_ptr, evd_ptr); if (dat_status != DAT_SUCCESS) { @@ -273,12 +267,6 @@ u32 dapl_evd_dealloc(struct dapl_evd *ev } } - /* - * We should now be safe to invalidate the EVD; reset the - * magic to prevent reuse. - */ - evd_ptr->header.magic = DAPL_MAGIC_INVALID; - /* Release reference on the CNO if it exists */ if (evd_ptr->cno_ptr != NULL) evd_ptr->cno_ptr = NULL; @@ -289,9 +277,8 @@ u32 dapl_evd_dealloc(struct dapl_evd *ev dapl_rbuf_destroy(&evd_ptr->free_event_queue); dapl_rbuf_destroy(&evd_ptr->pending_event_queue); - if (evd_ptr->events) { + if (evd_ptr->events) kfree(evd_ptr->events); - } kfree(evd_ptr); @@ -369,15 +356,15 @@ static struct dat_event *dapl_evd_get_ev struct dat_event *event; if (evd_ptr->evd_producer_locking_needed) { - spin_lock_irqsave(&evd_ptr->header.lock, evd_ptr->header.flags); + spin_lock_irqsave(&evd_ptr->common.lock, evd_ptr->common.flags); } event = (struct dat_event *) dapl_rbuf_remove(&evd_ptr->free_event_queue); /* Release the lock if it was taken and the call failed. */ if (!event && evd_ptr->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd_ptr->header.lock, - evd_ptr->header.flags); + spin_unlock_irqrestore(&evd_ptr->common.lock, + evd_ptr->common.flags); return event; } @@ -410,8 +397,8 @@ static void dapl_evd_post_event(struct d cno_to_trigger = evd_ptr->cno_ptr; if (evd_ptr->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd_ptr->header.lock, - evd_ptr->header.flags); + spin_unlock_irqrestore(&evd_ptr->common.lock, + evd_ptr->common.flags); if (cno_to_trigger != NULL) dapl_cno_trigger (cno_to_trigger, evd_ptr); @@ -426,12 +413,12 @@ static void dapl_evd_format_overflow_eve { struct dapl_ia *ia_ptr; - ia_ptr = evd_ptr->header.owner_ia; + ia_ptr = evd_ptr->common.owner_ia; - event_ptr->evd_handle = (DAT_EVD_HANDLE) evd_ptr; + event_ptr->evd = (struct dat_evd *) evd_ptr; event_ptr->event_number = DAT_ASYNC_ERROR_EVD_OVERFLOW; event_ptr->event_data.asynch_error_event_data.dat_handle = - (DAT_HANDLE) ia_ptr; + (void *) ia_ptr; } /* @@ -471,10 +458,10 @@ static struct dat_event *dapl_evd_get_an event_ptr = dapl_evd_get_event(evd_ptr); if (!event_ptr) - dapl_evd_post_overflow_event(evd_ptr->header.owner_ia-> + dapl_evd_post_overflow_event(evd_ptr->common.owner_ia-> async_error_evd, evd_ptr); else { - event_ptr->evd_handle = (DAT_EVD_HANDLE) evd_ptr; + event_ptr->evd = (struct dat_evd *) evd_ptr; event_ptr->event_number = evno; } @@ -482,10 +469,10 @@ static struct dat_event *dapl_evd_get_an } u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd_ptr, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp, struct sockaddr *ia_address_ptr, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle) + struct dat_cr *cr) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, @@ -500,11 +487,11 @@ u32 dapl_evd_post_cr_arrival_event(struc return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event_ptr->event_data.cr_arrival_event_data.sp_handle = sp_handle; + event_ptr->event_data.cr_arrival_event_data.sp = sp; event_ptr->event_data.cr_arrival_event_data.local_ia_address_ptr = ia_address_ptr; event_ptr->event_data.cr_arrival_event_data.conn_qual = conn_qual; - event_ptr->event_data.cr_arrival_event_data.cr_handle = cr_handle; + event_ptr->event_data.cr_arrival_event_data.cr = cr; dapl_evd_post_event(evd_ptr, event_ptr); @@ -513,7 +500,7 @@ u32 dapl_evd_post_cr_arrival_event(struc u32 dapl_evd_post_connection_event(struct dapl_evd *evd_ptr, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, int private_data_size, void *private_data) { @@ -530,7 +517,7 @@ u32 dapl_evd_post_connection_event(struc DAT_RESOURCE_MEMORY); } - event_ptr->event_data.connect_event_data.ep_handle = ep_handle; + event_ptr->event_data.connect_event_data.ep = ep; event_ptr->event_data.connect_event_data.private_data_size = private_data_size; event_ptr->event_data.connect_event_data.private_data = private_data; @@ -557,7 +544,7 @@ u32 dapl_evd_post_async_error_event(stru DAT_RESOURCE_MEMORY); event_ptr->event_data.asynch_error_event_data.dat_handle = - (DAT_HANDLE) ia_ptr; + (void *) ia_ptr; dapl_evd_post_event(evd_ptr, event_ptr); @@ -620,12 +607,8 @@ static void dapl_evd_cqe_to_event(struct ep_ptr = cookie->ep; dapl_os_assert(NULL != ep_ptr); - if (ep_ptr->header.magic != DAPL_MAGIC_EP) { - /* ep may have been freed, just return */ - return; - } - event_ptr->evd_handle = (DAT_EVD_HANDLE) evd_ptr; + event_ptr->evd = (struct dat_evd *) evd_ptr; switch (cookie->type) { case DAPL_COOKIE_TYPE_DTO: @@ -644,7 +627,7 @@ static void dapl_evd_cqe_to_event(struct } event_ptr->event_number = DAT_DTO_COMPLETION_EVENT; - event_data->ep_handle = cookie->ep; + event_data->ep = (struct dat_ep *)cookie->ep; event_data->user_cookie = cookie->val.dto.cookie; event_data->status = dto_status; @@ -687,7 +670,7 @@ static void dapl_evd_cqe_to_event(struct event_ptr->event_number = DAT_RMR_BIND_COMPLETION_EVENT; event_data = &event_ptr->event_data.rmr_completion_event_data; - event_data->rmr_handle = cookie->val.rmr.rmr; + event_data->rmr = (struct dat_rmr *)cookie->val.rmr.rmr; event_data->user_cookie = cookie->val.rmr.cookie; if (dto_status == DAT_DTO_SUCCESS) { Index: linux-kernel/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel/dat-provider/dapl_sp_util.c (revision 2545) +++ linux-kernel/dat-provider/dapl_sp_util.c (working copy) @@ -65,22 +65,16 @@ struct dapl_sp *dapl_sp_alloc(struct dap /* zero the structure */ memset(sp_ptr, 0, sizeof *sp_ptr); - /* - * initialize the header - */ - sp_ptr->header.provider = ia_ptr->header.provider; - if (is_psp) { - sp_ptr->header.magic = DAPL_MAGIC_PSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_PSP; - } else { - sp_ptr->header.magic = DAPL_MAGIC_RSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_RSP; - } - sp_ptr->header.owner_ia = ia_ptr; - sp_ptr->header.user_context.as_64 = 0; - sp_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&sp_ptr->header.ia_list_entry); - spin_lock_init(&sp_ptr->header.lock); + sp_ptr->sp.provider = ia_ptr->ia.provider; + if (is_psp) + sp_ptr->sp.type = DAT_SP_TYPE_PSP; + else + sp_ptr->sp.type = DAT_SP_TYPE_RSP; + sp_ptr->common.owner_ia = ia_ptr; + sp_ptr->common.user_context.as_64 = 0; + sp_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&sp_ptr->common.ia_list_entry); + spin_lock_init(&sp_ptr->common.lock); /* * Initialize the Body (set to NULL above) @@ -107,12 +101,8 @@ struct dapl_sp *dapl_sp_alloc(struct dap */ void dapl_sp_dealloc(struct dapl_sp *sp_ptr) { - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); dapl_os_assert(dapl_llist_is_empty(&sp_ptr->cr_list_head)); - sp_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - kfree(sp_ptr); } @@ -134,11 +124,11 @@ void dapl_sp_dealloc(struct dapl_sp *sp_ */ void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) { - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_llist_add_tail(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry, cr_ptr); + &cr_ptr->common.ia_list_entry, cr_ptr); sp_ptr->cr_list_count++; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, sp_ptr->common.flags); } /* @@ -174,7 +164,7 @@ struct dapl_cr *dapl_sp_search_cr(struct if (cr_ptr->ib_cm_handle == ib_cm_handle) return cr_ptr; - cr_ptr = cr_ptr->header.ia_list_entry.flink->data; + cr_ptr = cr_ptr->common.ia_list_entry.flink->data; } while ((void *)cr_ptr != (void *)sp_ptr->cr_list_head->data); return NULL; @@ -208,7 +198,7 @@ void dapl_sp_remove_cr(struct dapl_sp *s } dapl_llist_remove_entry(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); sp_ptr->cr_list_count--; } @@ -238,13 +228,13 @@ void dapl_sp_remove_ep(struct dapl_ep *e if (cr_ptr != NULL) { sp_ptr = cr_ptr->sp_ptr; - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* Remove the CR from the queue */ dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); Index: linux-kernel/dat-provider/dapl_evd_util.h =================================================================== --- linux-kernel/dat-provider/dapl_evd_util.h (revision 2545) +++ linux-kernel/dat-provider/dapl_evd_util.h (working copy) @@ -59,14 +59,14 @@ u32 dapl_evd_event_realloc(struct dapl_e */ u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd_ptr, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp, struct sockaddr *ia_address_ptr, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle); + struct dat_cr *cr); u32 dapl_evd_post_connection_event(struct dapl_evd *evd_ptr, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, int private_data_size, void *private_data); @@ -83,7 +83,7 @@ u32 dapl_evd_post_software_event(struct *************************************/ /* connection verb callback */ -extern void dapl_evd_connection_callback(struct dapl_cm_id *ib_cm_handle, +extern void dapl_evd_connection_callback(struct dapl_cm_id *cm, const enum dat_event_number event_number, const void *instant_data_p, struct dapl_ep *ep_ptr); Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2545) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -46,23 +46,6 @@ * * *********************************************************************/ -typedef enum dapl_magic { - /* magic number values for verification & debug */ - DAPL_MAGIC_IA = 0xCafeF00d, - DAPL_MAGIC_EVD = 0xFeedFace, - DAPL_MAGIC_EP = 0xDeadBabe, - DAPL_MAGIC_LMR = 0xBeefCafe, - DAPL_MAGIC_RMR = 0xABadCafe, - DAPL_MAGIC_PZ = 0xDeafBeef, - DAPL_MAGIC_PSP = 0xBeadeD0c, - DAPL_MAGIC_RSP = 0xFab4Feed, - DAPL_MAGIC_SRQ = 0xC001Babe, - DAPL_MAGIC_CR = 0xBe12Cee1, - DAPL_MAGIC_CR_DESTROYED = 0xB12bDead, - DAPL_MAGIC_CNO = 0xDeadF00d, - DAPL_MAGIC_INVALID = 0xFFFFFFFF -} DAPL_MAGIC; - typedef enum dapl_evd_state { DAPL_EVD_STATE_OPEN, DAPL_EVD_STATE_DEAD = 0xDEAD @@ -92,23 +75,6 @@ typedef enum dapl_evd_completion { /********************************************************************* * * - * Macros * - * * - *********************************************************************/ - -/* - * Simple macro to verify a handle is bad. Conditions: - * - pointer is NULL - * - pointer is not word aligned - * - pointer's magic number is wrong - */ -#define DAPL_BAD_HANDLE(h, magicNum) ( \ - ((h) == NULL) || \ - ((unsigned long)(h) & 3) || \ - (((struct dapl_header *)(h))->magic != (magicNum))) - -/********************************************************************* - * * * Typedefs * * * *********************************************************************/ @@ -170,11 +136,8 @@ struct dapl_hca { struct dat_ep_attr ep_attr; }; -/* DAPL Objects always have the following header */ -struct dapl_header { - struct dat_provider *provider; /* required by DAT - must be first */ - DAPL_MAGIC magic; /* magic number for verification */ - enum dat_handle_type handle_type; +/* DAPL Objects always have the following items */ +struct dapl_common { struct dapl_ia *owner_ia; struct dapl_llist_entry ia_list_entry; union dat_context user_context; /* user context - opaque to DAPL */ @@ -182,9 +145,9 @@ struct dapl_header { unsigned long flags; /* saved lock flag values */ }; -/* struct dapl_ia maps to DAT_IA_HANDLE */ struct dapl_ia { - struct dapl_header header; /* WARNING: must be first */ + struct dat_ia ia; + struct dapl_common common; struct dapl_hca *hca_ptr; struct dapl_evd *async_error_evd; boolean_t cleanup_async_error_evd; @@ -200,16 +163,16 @@ struct dapl_ia { struct dapl_llist_entry *srq_list_head; /* SRQ queue */ }; -/* DAPL_CNO maps to DAT_CNO_HANDLE */ struct dapl_cno { - struct dapl_header header; /* WARNING: must be first */ - struct dat_upcall_object cno_upcall; - enum dat_upcall_policy cno_upcall_policy; + struct dat_cno cno; + struct dapl_common common; + struct dat_upcall_object cno_upcall; + enum dat_upcall_policy cno_upcall_policy; }; -/* struct dapl_evd maps to DAT_EVD_HANDLE */ struct dapl_evd { - struct dapl_header header; /* WARNING: must be first */ + struct dat_evd evd; + struct dapl_common common; DAPL_EVD_STATE evd_state; enum dat_evd_flags evd_flags; @@ -236,9 +199,9 @@ struct dapl_evd { struct dapl_cno *cno_ptr; }; -/* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { - struct dapl_header header; + struct dat_ep ep; + struct dapl_common common; /* What the DAT Consumer asked for */ struct dat_ep_param param; @@ -247,7 +210,7 @@ struct dapl_ep { enum ib_qp_state qp_state; /* communications manager handle */ - struct dapl_cm_id *cm_handle; + struct dapl_cm_id *cm; /* store the remote IA address here, reference from the param * struct which only has a pointer, no storage */ @@ -273,33 +236,33 @@ struct dapl_ep { int send_iov_num; }; -/* struct dapl_srq maps to DAT_SRQ_HANDLE */ struct dapl_srq { - struct dapl_header header; /* WARNING: must be first */ + struct dat_srq srq; + struct dapl_common common; struct dat_srq_param param; atomic_t srq_ref_count; struct dapl_cookie_buffer recv_buffer; atomic_t recv_count; }; -/* struct dapl_pz maps to DAT_PZ_HANDLE */ struct dapl_pz { - struct dapl_header header; /* WARNING: must be first */ + struct dat_pz pz; + struct dapl_common common; struct ib_pd *pd; atomic_t pz_ref_count; }; -/* struct dapl_lmr maps to DAT_LMR_HANDLE */ struct dapl_lmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_lmr lmr; + struct dapl_common common; struct dat_lmr_param param; struct ib_mr *mr; atomic_t lmr_ref_count; }; -/* struct dapl_rmr maps to DAT_RMR_HANDLE */ struct dapl_rmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_rmr rmr; + struct dapl_common common; struct dat_rmr_param param; struct dapl_ep *ep; struct dapl_pz *pz; @@ -316,17 +279,17 @@ typedef enum dapl_sp_state { DAPL_SP_STATE_RSP_PENDING } DAPL_SP_STATE; -/* struct dapl_sp maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ struct dapl_sp { - struct dapl_header header; /* WARNING: must be first */ + struct dat_sp sp; + struct dapl_common common; DAPL_SP_STATE state; /* type and queue of the SP */ /* PSP/RSP PARAM fields */ - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; /* maintenance fields */ boolean_t listening; /* PSP is registered & active */ @@ -335,9 +298,9 @@ struct dapl_sp { int cr_list_count; /* count of CRs on queue */ }; -/* struct dapl_cr maps to DAT_CR_HANDLE */ struct dapl_cr { - struct dapl_header header; + struct dat_cr cr; + struct dapl_common common; /* for convenience the data is kept as a struct dat_cr_param. * however, the "local_endpoint" field is always NULL * so this wastes a pointer. This is probably ok to @@ -402,91 +365,88 @@ struct dapl_cookie { */ extern u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr); + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr); -extern u32 dapl_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dapl_ia_close(struct dat_ia *, enum dat_close_flags); -extern u32 dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, +extern u32 dapl_ia_query(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); /* helper functions */ -extern u32 dapl_set_consumer_context(DAT_HANDLE handle, +extern u32 dapl_set_consumer_context(void *handle, union dat_context context); -extern u32 dapl_get_consumer_context(DAT_HANDLE handle, +extern u32 dapl_get_consumer_context(void *handle, union dat_context *context); -extern u32 dapl_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type); - /* CR Functions */ -extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); +extern u32 dapl_cr_query(struct dat_cr *, struct dat_cr_param *); -extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +extern u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data); -extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); +extern u32 dapl_cr_reject(struct dat_cr *cr); -extern u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL handoff); +extern u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff); /* EVD Functions */ -extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, +extern u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, u64 *suggested_alignment); -extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, +extern u32 dapl_evd_kcreate(struct dat_ia *ia, int evd_min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE * evd_handle); + struct dat_evd ** evd); -extern u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_kquery(struct dat_evd *evd, struct dat_evd_param *evd_args); -extern u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall); -extern u32 dapl_evd_enable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_enable(struct dat_evd *evd); -extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_disable(struct dat_evd *evd); -extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen); +extern u32 dapl_evd_resize(struct dat_evd *evd, int evd_qlen); -extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_post_se(struct dat_evd *evd, const struct dat_event *event); -extern u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event); +extern u32 dapl_evd_dequeue(struct dat_evd *evd, struct dat_event *event); -extern u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_free(struct dat_evd *evd); -extern u32 dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_set_unwaitable(struct dat_evd *evd); -extern u32 dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_clear_unwaitable(struct dat_evd *evd); /* EP functions */ -extern u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE in_dto_completion_evd_handle, - DAT_EVD_HANDLE out_dto_completion_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, +extern u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_dto_completion_evd, + struct dat_evd *out_dto_completion_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *ep_parameters, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); +extern u32 dapl_ep_query(struct dat_ep *, struct dat_ep_param *); -extern u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask ep_args_mask, const struct dat_ep_param *ep_args); -extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, @@ -495,161 +455,161 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); -extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, +extern u32 dapl_ep_dup_connect(struct dat_ep *ep, + struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); -extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags close_flags); -extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle); -extern u32 dapl_ep_free(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_free(struct dat_ep *ep); -extern u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_reset(struct dat_ep *ep); -extern u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_ep_create_with_srq(struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq_handle, const struct dat_ep_attr *ep_attributes, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocated, int *bufs_alloc_span); -extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark); /* LMR functions */ -extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +extern u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address); -extern u32 dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); +extern u32 dapl_lmr_query(struct dat_lmr *, struct dat_lmr_param *); -extern u32 dapl_lmr_free(DAT_LMR_HANDLE); +extern u32 dapl_lmr_free(struct dat_lmr *); -extern u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); -extern u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); /* RMR Functions */ -extern u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle); +extern u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr); -extern u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_query(struct dat_rmr *rmr, struct dat_rmr_param *rmr_args); -extern u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_bind(struct dat_rmr *rmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *context); -extern u32 dapl_rmr_free(DAT_RMR_HANDLE); +extern u32 dapl_rmr_free(struct dat_rmr *rmr); /* PSP Functions */ -extern u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, +extern u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, +extern u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); +extern u32 dapl_psp_query(struct dat_sp *, struct dat_psp_param *); -extern u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle); +extern u32 dapl_psp_free(struct dat_sp *psp); /* RSP Functions */ -extern u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle); +extern u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp); -extern u32 dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); +extern u32 dapl_rsp_query(struct dat_sp *, struct dat_rsp_param *); -extern u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle); +extern u32 dapl_rsp_free(struct dat_sp *rsp); /* PZ Functions */ -extern u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle); +extern u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz); -extern u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_args); +extern u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_args); -extern u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle); +extern u32 dapl_pz_free(struct dat_pz *pz); /* SRQ functions */ -extern u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, +extern u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *srq_attr, - DAT_SRQ_HANDLE *srq_handle); + struct dat_srq **srq); -extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); +extern u32 dapl_srq_free(struct dat_srq *srq); -extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +extern u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie); -extern u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param); -extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto); -extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark); +extern u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark); /* * DAPL internal utility function prototypes Index: linux-kernel/dat-provider/dapl_cr_query.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_query.c (revision 2545) +++ linux-kernel/dat-provider/dapl_cr_query.c (working copy) @@ -34,16 +34,16 @@ #include "dapl.h" #include "dapl_openib_util.h" -u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) +u32 dapl_cr_query(struct dat_cr *cr, struct dat_cr_param *cr_param) { struct dapl_cr *cr_ptr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", - cr_handle, cr_param); + cr, cr_param); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (cr == NULL) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } @@ -53,10 +53,10 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* obtain the remote IP address */ - (void) dapl_ib_cm_remote_addr((DAT_HANDLE) cr_handle, + (void) dapl_ib_cm_remote_addr((void *) cr, &cr_ptr->remote_ia_address); /* since the arguments are easily accessible, ignore the mask */ Index: linux-kernel/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel/patches/alt_dat_provider_makefile (revision 2545) +++ linux-kernel/patches/alt_dat_provider_makefile (working copy) @@ -34,8 +34,6 @@ PROVIDER_MODULES := \ dapl_evd_resize \ dapl_evd_un_async_error_callb \ dapl_evd_util \ - dapl_get_consumer_context \ - dapl_get_handle_type \ dapl_hash \ dapl_hca_util \ dapl_ia_close \ @@ -52,7 +50,6 @@ PROVIDER_MODULES := \ dapl_ring_buffer_util \ dapl_rmr \ dapl_rsp \ - dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ dapl_util Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2545) +++ linux-kernel/dat/api.c (working copy) @@ -350,7 +350,7 @@ static u32 dat_strerror_minor(u32 value, } u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, + struct dat_evd **async_event_handle, struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; @@ -387,13 +387,12 @@ u32 dat_ia_openv(const char *name, int a return status; } - return ia_open_func(name, async_event_qlen, async_event_handle, - ia_handle); + return ia_open_func(name, async_event_qlen, async_event_handle, ia); } EXPORT_SYMBOL(dat_ia_openv); -u32 dat_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags flags) +u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; @@ -401,17 +400,16 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle const char *ia_name; dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "%s(ia:%x, flags:%x)\n", __func__, ia_handle, flags); + "%s(ia:%x, flags:%x)\n", __func__, ia, flags); - provider = DAT_HANDLE_TO_PROVIDER(ia_handle); + provider = DAT_HANDLE_TO_PROVIDER(ia); ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia_handle, NULL, NULL, + if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); - else if (DAT_SUCCESS != (status = provider->ia_close_func(ia_handle, - flags))) + else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "close function for IA %s failed\n", ia_name); else { Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2545) +++ linux-kernel/dat/dat.h (working copy) @@ -333,20 +333,6 @@ enum dat_return_subtype { #define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) #define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) -/* dat handle types */ -enum dat_handle_type { - DAT_HANDLE_TYPE_CR, - DAT_HANDLE_TYPE_EP, - DAT_HANDLE_TYPE_EVD, - DAT_HANDLE_TYPE_IA, - DAT_HANDLE_TYPE_LMR, - DAT_HANDLE_TYPE_PSP, - DAT_HANDLE_TYPE_PZ, - DAT_HANDLE_TYPE_RMR, - DAT_HANDLE_TYPE_RSP, - DAT_HANDLE_TYPE_SRQ -}; - #define DAT_NAME_MAX_LENGTH 256 struct dat_named_attr { @@ -385,21 +371,6 @@ enum dat_completion_flags { #define DAT_TIMEOUT_MAX ULONG_MAX -/* dat handles */ -typedef void *DAT_HANDLE; -typedef DAT_HANDLE DAT_CR_HANDLE; -typedef DAT_HANDLE DAT_EP_HANDLE; -typedef DAT_HANDLE DAT_EVD_HANDLE; -typedef DAT_HANDLE DAT_IA_HANDLE; -typedef DAT_HANDLE DAT_LMR_HANDLE; -typedef DAT_HANDLE DAT_PSP_HANDLE; -typedef DAT_HANDLE DAT_PZ_HANDLE; -typedef DAT_HANDLE DAT_RMR_HANDLE; -typedef DAT_HANDLE DAT_RSP_HANDLE; -typedef DAT_HANDLE DAT_SRQ_HANDLE; - -#define DAT_HANDLE_NULL ((DAT_HANDLE)NULL) - typedef u64 DAT_CONN_QUAL; typedef u64 DAT_PORT_QUAL; @@ -541,8 +512,8 @@ enum dat_mem_priv_flags { /* RMR Arguments */ struct dat_rmr_param { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; struct dat_lmr_triplet lmr_triplet; enum dat_mem_priv_flags mem_priv; DAT_RMR_CONTEXT rmr_context; @@ -588,17 +559,17 @@ enum dat_ep_state { }; struct dat_ep_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_ep_state ep_state; struct sockaddr *local_ia_address_ptr; DAT_PORT_QUAL local_port_qual; struct sockaddr *remote_ia_address_ptr; DAT_PORT_QUAL remote_port_qual; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EVD_HANDLE request_evd_handle; - DAT_EVD_HANDLE connect_evd_handle; - DAT_SRQ_HANDLE srq_handle; + struct dat_pz *pz; + struct dat_evd *recv_evd; + struct dat_evd *request_evd; + struct dat_evd *connect_evd; + struct dat_srq *srq; struct dat_ep_attr ep_attr; }; @@ -654,9 +625,9 @@ struct dat_srq_attr { }; struct dat_srq_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_srq_state srq_state; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; int max_recv_dtos; int max_recv_iov; int low_watermark; @@ -667,25 +638,25 @@ struct dat_srq_param { /* PZ Parameters */ struct dat_pz_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; }; /* PSP Parameters */ struct dat_psp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; }; /* RSP Parameters */ struct dat_rsp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_ep *ep; }; /* Connection Request Parameters. @@ -712,11 +683,11 @@ struct dat_cr_param { /* The local Endpoint provided by the Service Point for the requested * connection. It is the only Endpoint that can accept a Connection - * Request on this Service Point. The value DAT_HANDLE_NULL + * Request on this Service Point. The value NULL * represents that there is no associated local Endpoint for the * requested connection. */ - DAT_EP_HANDLE local_ep_handle; + struct dat_ep *local_ep; }; /************************** Events ******************************/ @@ -753,7 +724,7 @@ enum dat_dto_completion_status { /* DTO completion event data */ /* transfered_length is not defined if status is not DAT_SUCCESS */ struct dat_dto_completion_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; DAT_DTO_COOKIE user_cookie; enum dat_dto_completion_status status; u64 transfered_length; @@ -761,25 +732,20 @@ struct dat_dto_completion_event_data { /* RMR bind completion event data */ struct dat_rmr_bind_completion_event_data { - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr *rmr; DAT_RMR_COOKIE user_cookie; enum dat_dto_completion_status status; }; -typedef union dat_sp_handle { - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; -} DAT_SP_HANDLE; - /* Connection Request Arrival event data */ struct dat_cr_arrival_event_data { /* Handle to the Service Point that received the Connection Request * from the remote side. If the Service Point was Reserved, sp is - * DAT_HANDLE_NULL because the reserved Service Point is + * NULL because the reserved Service Point is * automatically destroyed upon generating this event. Can be PSP * or RSP. */ - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; /* Address of the IA on which the Connection Request arrived. */ struct sockaddr *local_ia_address_ptr; @@ -791,30 +757,30 @@ struct dat_cr_arrival_event_data { /* The Connection Request instance created by a Provider for the * arrived Connection Request. Consumers can find out private_data - * passed by a remote Consumer from cr_handle. It is up to a Consumer + * passed by a remote Consumer from cr. It is up to a Consumer * to dat_cr_accept or dat_cr_reject of the Connection Request. */ - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; }; /* Connection event data */ struct dat_connection_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; int private_data_size; void *private_data; }; /* Async Error event data */ -/* For unaffiliated asynchronous event dat_handle is ia_handle. For - * Endpoint affiliated asynchronous event dat_handle is ep_handle. For - * EVD affiliated asynchronous event dat_handle is evd_handle. For SRQ +/* For unaffiliated asynchronous event dat_handle is ia. For + * Endpoint affiliated asynchronous event dat_handle is ep. For + * EVD affiliated asynchronous event dat_handle is evd. For SRQ * affiliated asynchronous event dat_handle is srq_handle. For Memory - * affiliated asynchronous event dat_handle is either lmr_handle, - * rmr_handle or pz_handle. + * affiliated asynchronous event dat_handle is either lmr, + * rmr_handle or pz. */ struct dat_asynch_error_event_data { - DAT_HANDLE dat_handle; /* either IA, EP, EVD, SRQ, */ + void *dat_handle; /* either IA, EP, EVD, SRQ, */ /* LMR, RMR, or PZ handle */ int reason; /* object specific */ }; @@ -894,7 +860,7 @@ typedef union dat_event_data { struct dat_event { enum dat_event_number event_number; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; DAT_EVENT_DATA event_data; }; @@ -934,7 +900,7 @@ struct dat_provider_info { }; struct dat_evd_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; int evd_qlen; enum dat_upcall_policy upcall_policy; struct dat_upcall_object upcall; @@ -960,7 +926,7 @@ enum dat_mem_optimize_flags { typedef union dat_region_description { void *for_va; - DAT_LMR_HANDLE for_lmr_handle; + struct dat_lmr *for_lmr; void *for_pointer; /* For kdapl only */ void *for_array; /* For kdapl only */ u64 for_pa; /* For kdapl only */ @@ -968,11 +934,11 @@ typedef union dat_region_description { /* LMR Arguments */ struct dat_lmr_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_mem_type mem_type; DAT_REGION_DESCRIPTION region_desc; u64 length; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; enum dat_mem_priv_flags mem_priv; DAT_LMR_CONTEXT lmr_context; DAT_RMR_CONTEXT rmr_context; @@ -1083,178 +1049,176 @@ struct dat_provider_attr { * ****************************************************************/ -typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *); +typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **); -typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *, u32, u32, boolean_t); +typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **, u32, u32, boolean_t); -typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum dat_close_flags); -typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_HANDLE, DAT_EVD_HANDLE *, +typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); - -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(void *, union dat_context); -typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, enum dat_handle_type *); +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(void *, union dat_context *); -typedef u32 (*DAT_CR_QUERY_FUNC)(DAT_CR_HANDLE, struct dat_cr_param *); +typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct dat_cr_param *); -typedef u32 (*DAT_CR_ACCEPT_FUNC)(DAT_CR_HANDLE, DAT_EP_HANDLE, int, +typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct dat_ep *, int, const void*); -typedef u32 (*DAT_CR_REJECT_FUNC)(DAT_CR_HANDLE); +typedef u32 (*DAT_CR_REJECT_FUNC)(struct dat_cr *); -typedef u32 (*DAT_CR_HANDOFF_FUNC)(DAT_CR_HANDLE, DAT_CONN_QUAL); +typedef u32 (*DAT_CR_HANDOFF_FUNC)(struct dat_cr *, DAT_CONN_QUAL); -typedef u32 (*DAT_EVD_RESIZE_FUNC)(DAT_EVD_HANDLE, int); +typedef u32 (*DAT_EVD_RESIZE_FUNC)(struct dat_evd *, int); -typedef u32 (*DAT_EVD_POST_SE_FUNC)(DAT_EVD_HANDLE, const struct dat_event *); +typedef u32 (*DAT_EVD_POST_SE_FUNC)(struct dat_evd *, const struct dat_event *); -typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(DAT_EVD_HANDLE, struct dat_event *); +typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(struct dat_evd *, struct dat_event *); -typedef u32 (*DAT_EVD_FREE_FUNC)(DAT_EVD_HANDLE); +typedef u32 (*DAT_EVD_FREE_FUNC)(struct dat_evd *); -typedef u32 (*DAT_EVD_QUERY_FUNC)(DAT_EVD_HANDLE, struct dat_evd_param *); +typedef u32 (*DAT_EVD_QUERY_FUNC)(struct dat_evd *, struct dat_evd_param *); -typedef u32 (*DAT_EP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - const struct dat_ep_attr *, DAT_EP_HANDLE *); +typedef u32 (*DAT_EP_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, struct dat_evd *, + struct dat_evd *, struct dat_evd *, + const struct dat_ep_attr *, struct dat_ep **); -typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_SRQ_HANDLE, +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_evd *, struct dat_evd *, + struct dat_evd *, struct dat_srq *, const struct dat_ep_attr *, - DAT_EP_HANDLE *); + struct dat_ep **); -typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_HANDLE, struct dat_ep_param *); +typedef u32 (*DAT_EP_QUERY_FUNC)(struct dat_ep *, struct dat_ep_param *); -typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, enum dat_ep_param_mask, +typedef u32 (*DAT_EP_MODIFY_FUNC)(struct dat_ep *, enum dat_ep_param_mask, const struct dat_ep_param *); -typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, +typedef u32 (*DAT_EP_CONNECT_FUNC)(struct dat_ep *, struct sockaddr *, DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); -typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(struct dat_ep *, struct dat_ep *, unsigned long timeout, int, const void *, enum dat_qos); -typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_EP_DISCONNECT_FUNC)(struct dat_ep *, enum dat_close_flags); -typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_SEND_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RECV_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RECV_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, enum dat_ep_state *, +typedef u32 (*DAT_EP_GET_STATUS_FUNC)(struct dat_ep *, enum dat_ep_state *, boolean_t *, boolean_t *); -typedef u32 (*DAT_EP_FREE_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_FREE_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RESET_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_RESET_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(DAT_EP_HANDLE, int *, int *); +typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(struct dat_ep *, int *, int *); -typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(DAT_EP_HANDLE, int, int); +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(struct dat_ep *, int, int); -typedef u32 (*DAT_LMR_FREE_FUNC)(DAT_LMR_HANDLE); +typedef u32 (*DAT_LMR_FREE_FUNC)(struct dat_lmr *); -typedef u32 (*DAT_LMR_QUERY_FUNC)(DAT_LMR_HANDLE, struct dat_lmr_param *); +typedef u32 (*DAT_LMR_QUERY_FUNC)(struct dat_lmr *, struct dat_lmr_param *); -typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_PZ_HANDLE, DAT_RMR_HANDLE *); +typedef u32 (*DAT_RMR_CREATE_FUNC)(struct dat_pz *, struct dat_rmr * *); -typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RMR_HANDLE, struct dat_rmr_param *); +typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); -typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, - enum dat_mem_priv_flags, DAT_EP_HANDLE, +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, + enum dat_mem_priv_flags, struct dat_ep *, DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); -typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); +typedef u32 (*DAT_RMR_FREE_FUNC)(struct dat_rmr *); -typedef u32 (*DAT_PSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, - enum dat_psp_flags, DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_evd *, + enum dat_psp_flags, struct dat_sp **); -typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL *, - DAT_EVD_HANDLE, enum dat_psp_flags, - DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(struct dat_ia *, DAT_CONN_QUAL *, + struct dat_evd *, enum dat_psp_flags, + struct dat_sp **); -typedef u32 (*DAT_PSP_QUERY_FUNC)(DAT_PSP_HANDLE, struct dat_psp_param *); +typedef u32 (*DAT_PSP_QUERY_FUNC)(struct dat_sp *, struct dat_psp_param *); -typedef u32 (*DAT_PSP_FREE_FUNC)(DAT_PSP_HANDLE); +typedef u32 (*DAT_PSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_RSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, - DAT_EVD_HANDLE, DAT_RSP_HANDLE *); +typedef u32 (*DAT_RSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_ep *, + struct dat_evd *, struct dat_sp **); -typedef u32 (*DAT_RSP_QUERY_FUNC)(DAT_RSP_HANDLE, struct dat_rsp_param *); +typedef u32 (*DAT_RSP_QUERY_FUNC)(struct dat_sp *, struct dat_rsp_param *); -typedef u32 (*DAT_RSP_FREE_FUNC)(DAT_RSP_HANDLE); +typedef u32 (*DAT_RSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_PZ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE *); +typedef u32 (*DAT_PZ_CREATE_FUNC)(struct dat_ia *, struct dat_pz **); -typedef u32 (*DAT_PZ_QUERY_FUNC)(DAT_PZ_HANDLE, struct dat_pz_param *); +typedef u32 (*DAT_PZ_QUERY_FUNC)(struct dat_pz *, struct dat_pz_param *); -typedef u32 (*DAT_PZ_FREE_FUNC)(DAT_PZ_HANDLE); +typedef u32 (*DAT_PZ_FREE_FUNC)(struct dat_pz *); -typedef u32 (*DAT_SRQ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - struct dat_srq_attr *, DAT_SRQ_HANDLE *); +typedef u32 (*DAT_SRQ_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_srq_attr *, struct dat_srq **); -typedef u32 (*DAT_SRQ_SET_LW_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_SET_LW_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_FREE_FUNC)(DAT_SRQ_HANDLE); +typedef u32 (*DAT_SRQ_FREE_FUNC)(struct dat_srq *); -typedef u32 (*DAT_SRQ_QUERY_FUNC)(DAT_SRQ_HANDLE, struct dat_srq_param *); +typedef u32 (*DAT_SRQ_QUERY_FUNC)(struct dat_srq *, struct dat_srq_param *); -typedef u32 (*DAT_SRQ_RESIZE_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_RESIZE_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DAT_SRQ_HANDLE, int, +typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(struct dat_srq *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, +typedef u32 (*DAT_LMR_KCREATE_FUNC)(struct dat_ia *, enum dat_mem_type, DAT_REGION_DESCRIPTION, u64, - DAT_PZ_HANDLE, enum dat_mem_priv_flags, + struct dat_pz *, enum dat_mem_priv_flags, enum dat_mem_optimize_flags, - DAT_LMR_HANDLE *, + struct dat_lmr **, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, u64 *, u64 *); -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, u64, +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(struct dat_ia *, enum dat_mem_type, u64, enum dat_mem_optimize_flags, u64 *, u64 *); -typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, enum dat_upcall_policy, +typedef u32 (*DAT_EVD_KCREATE_FUNC) (struct dat_ia *, int, enum dat_upcall_policy, const struct dat_upcall_object *, - enum dat_evd_flags, DAT_EVD_HANDLE *); + enum dat_evd_flags, struct dat_evd **); -typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(struct dat_evd *, enum dat_upcall_policy, const struct dat_upcall_object *); @@ -1269,7 +1233,6 @@ struct dat_provider { DAT_SET_CONSUMER_CONTEXT_FUNC set_consumer_context_func; DAT_GET_CONSUMER_CONTEXT_FUNC get_consumer_context_func; - DAT_GET_HANDLE_TYPE_FUNC get_handle_type_func; DAT_CR_QUERY_FUNC cr_query_func; DAT_CR_ACCEPT_FUNC cr_accept_func; @@ -1338,6 +1301,52 @@ struct dat_provider { DAT_SRQ_SET_LW_FUNC srq_set_lw_func; }; +struct dat_ep { + struct dat_provider *provider; +}; + +struct dat_ia { + struct dat_provider *provider; +}; + +struct dat_evd { + struct dat_provider *provider; +}; + +struct dat_cno { + struct dat_provider *provider; +}; + +struct dat_srq { + struct dat_provider *provider; +}; + +struct dat_pz { + struct dat_provider *provider; +}; + +struct dat_lmr { + struct dat_provider *provider; +}; + +struct dat_rmr { + struct dat_provider *provider; +}; + +enum dat_sp_type { + DAT_SP_TYPE_RSP, + DAT_SP_TYPE_PSP +}; + +struct dat_sp { + struct dat_provider *provider; + enum dat_sp_type type; +}; + +struct dat_cr { + struct dat_provider *provider; +}; + /*************************************************************** * * FUNCTION PROTOTYPES @@ -1359,8 +1368,8 @@ extern u32 dat_registry_remove_provider( * DAT registry functions for consumers */ extern u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, - DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, + struct dat_evd **async_event_handle, + struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety); #define dat_ia_open(name, qlen, async_evd, ia) \ @@ -1368,7 +1377,7 @@ extern u32 dat_ia_openv(const char *name DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ DAT_THREADSAFE) -extern u32 dat_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); extern u32 dat_strerror(u32, const char **, const char **); @@ -1383,27 +1392,21 @@ extern u32 dat_registry_list_providers(i #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) -static inline u32 dat_set_consumer_context(DAT_HANDLE handle, +static inline u32 dat_set_consumer_context(void *handle, union dat_context context) { return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, context); } -static inline u32 dat_get_consumer_context(DAT_HANDLE handle, +static inline u32 dat_get_consumer_context(void *handle, union dat_context *context) { return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, context); } -static inline u32 dat_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type) -{ - return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); -} - -static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, +static inline u32 dat_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimize, @@ -1415,7 +1418,7 @@ static inline u32 dat_ia_memtype_hint(DA preferred_alignment); } -static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE *async_evd, +static inline u32 dat_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -1423,49 +1426,49 @@ static inline u32 dat_ia_query(DAT_IA_HA ia_query_func, ia, async_evd, ia_attr, provider_attr); } -static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, +static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data) { return DAT_CALL_PROVIDER_FUNC( cr_accept_func, cr, ep, private_data_size, private_data); } -static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) +static inline u32 dat_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff) { return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); } -static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) +static inline u32 dat_cr_query(struct dat_cr *cr, struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } -static inline u32 dat_cr_reject(DAT_CR_HANDLE cr) +static inline u32 dat_cr_reject(struct dat_cr *cr) { return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); } -static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) +static inline u32 dat_evd_dequeue(struct dat_evd *evd, struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } -static inline u32 dat_evd_free(DAT_EVD_HANDLE evd) +static inline u32 dat_evd_free(struct dat_evd *evd) { return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); } -static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, +static inline u32 dat_evd_kcreate(struct dat_ia *ia, int qlen, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall, enum dat_evd_flags flags, - DAT_EVD_HANDLE * evd) + struct dat_evd ** evd) { return DAT_CALL_PROVIDER_FUNC(evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); } -static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall) { @@ -1473,23 +1476,23 @@ static inline u32 dat_evd_modify_upcall( upcall); } -static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_post_se(struct dat_evd *evd, const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } -static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) +static inline u32 dat_evd_query(struct dat_evd *evd, struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } -static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, int qlen) +static inline u32 dat_evd_resize(struct dat_evd *evd, int qlen) { return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, +static inline u32 dat_ep_connect(struct dat_ep *ep, struct sockaddr *ia_addr, DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, @@ -1500,36 +1503,36 @@ static inline u32 dat_ep_connect(DAT_EP_ qos, flags); } -static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, +static inline u32 dat_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, struct dat_evd *out_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_func, ia, pz, in_evd, out_evd, connect_evd, attr, ep); } -static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, - DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, - DAT_SRQ_HANDLE srq, +static inline u32 dat_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, + struct dat_evd *out_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_with_srq_func, ia, pz, in_evd, out_evd, connect_evd, srq, attr, ep); } -static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_disconnect(struct dat_ep *ep, enum dat_close_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } -static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, +static inline u32 dat_ep_dup_connect(struct dat_ep *ep, struct dat_ep *dup_ep, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) @@ -1538,25 +1541,25 @@ static inline u32 dat_ep_dup_connect(DAT private_data_size, private_data, qos); } -static inline u32 dat_ep_free(DAT_EP_HANDLE ep) +static inline u32 dat_ep_free(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, enum dat_ep_state *state, +static inline u32 dat_ep_get_status(struct dat_ep *ep, enum dat_ep_state *state, boolean_t *recv_idle, boolean_t *req_idle) { return DAT_CALL_PROVIDER_FUNC(ep_get_status_func, ep, state, recv_idle, req_idle); } -static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, enum dat_ep_param_mask mask, +static inline u32 dat_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } -static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_read(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1566,7 +1569,7 @@ static inline u32 dat_ep_post_rdma_read( local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_write(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1576,7 +1579,7 @@ static inline u32 dat_ep_post_rdma_write local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_recv(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, enum dat_completion_flags flags) @@ -1585,7 +1588,7 @@ static inline u32 dat_ep_post_recv(DAT_E cookie, flags); } -static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_send(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1595,24 +1598,24 @@ static inline u32 dat_ep_post_send(DAT_E ep_post_send_func, ep, size, local_iov, cookie, flags); } -static inline u32 dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) +static inline u32 dat_ep_query(struct dat_ep *ep, struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } -static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, int *bufs_alloc, +static inline u32 dat_ep_recv_query(struct dat_ep *ep, int *bufs_alloc, int *bufs_avail) { return DAT_CALL_PROVIDER_FUNC(ep_recv_query_func, ep, bufs_alloc, bufs_avail); } -static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) +static inline u32 dat_ep_reset(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); } -static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, +static inline u32 dat_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -1620,17 +1623,17 @@ static inline u32 dat_ep_set_watermark(D soft_high_watermark, hard_high_watermark); } -static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) +static inline u32 dat_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } -static inline u32 dat_lmr_free(DAT_LMR_HANDLE lmr) +static inline u32 dat_lmr_free(struct dat_lmr *lmr) { return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); } -static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1638,7 +1641,7 @@ static inline u32 dat_lmr_sync_rdma_read num_iovs); } -static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1646,20 +1649,20 @@ static inline u32 dat_lmr_sync_rdma_writ num_iovs); } -static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE *rmr) +static inline u32 dat_rmr_create(struct dat_pz *pz, struct dat_rmr * *rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline u32 dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) +static inline u32 dat_rmr_query(struct dat_rmr * rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, +static inline u32 dat_rmr_bind(struct dat_rmr * rmr, const struct dat_lmr_triplet *iov, enum dat_mem_priv_flags mem_flags, - DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, + struct dat_ep *ep, DAT_RMR_COOKIE cookie, enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) { @@ -1667,84 +1670,84 @@ static inline u32 dat_rmr_bind(DAT_RMR_H cookie, comp_flags, context); } -static inline u32 dat_rmr_free(DAT_RMR_HANDLE rmr) +static inline u32 dat_rmr_free(struct dat_rmr * rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) +static inline u32 dat_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd, +static inline u32 dat_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_any_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) +static inline u32 dat_psp_query(struct dat_sp *psp, struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } -static inline u32 dat_psp_free(DAT_PSP_HANDLE psp) +static inline u32 dat_psp_free(struct dat_sp *psp) { return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); } -static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, - DAT_RSP_HANDLE *rsp) +static inline u32 dat_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_create_func, ia, conn_qual, ep, evd, rsp); } -static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) +static inline u32 dat_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } -static inline u32 dat_rsp_free(DAT_RSP_HANDLE rsp) +static inline u32 dat_rsp_free(struct dat_sp *rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); } -static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE *pz) +static inline u32 dat_pz_create(struct dat_ia *ia, struct dat_pz **pz) { return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) +static inline u32 dat_pz_query(struct dat_pz *pz, struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } -static inline u32 dat_pz_free(DAT_PZ_HANDLE pz) +static inline u32 dat_pz_free(struct dat_pz *pz) { return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); } -static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, +static inline u32 dat_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *attr, - DAT_SRQ_HANDLE *srq) + struct dat_srq **srq) { return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); } -static inline u32 dat_srq_free(DAT_SRQ_HANDLE srq) +static inline u32 dat_srq_free(struct dat_srq *srq) { return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); } -static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, int num_iovs, +static inline u32 dat_srq_post_recv(struct dat_srq *srq, int num_iovs, struct dat_lmr_triplet *iovs, DAT_DTO_COOKIE cookie) { @@ -1752,34 +1755,34 @@ static inline u32 dat_srq_post_recv(DAT_ cookie); } -static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) +static inline u32 dat_srq_query(struct dat_srq *srq, struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } -static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, int max_recv_dtos) +static inline u32 dat_srq_resize(struct dat_srq *srq, int max_recv_dtos) { return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); } -static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, int low_watermark) +static inline u32 dat_srq_set_lw(struct dat_srq *srq, int low_watermark) { return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, enum dat_mem_type type, +static inline u32 dat_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type type, DAT_REGION_DESCRIPTION region, u64 len, - DAT_PZ_HANDLE pz, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { return DAT_CALL_PROVIDER_FUNC(lmr_kcreate_func, ia, type, region, len, - pz, privileges, optimization, lmr_handle, + pz, privileges, optimization, lmr, lmr_context, rmr_context, registered_length, registered_address); } From surs at cse.ohio-state.edu Fri Jun 3 15:43:19 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Fri, 3 Jun 2005 18:43:19 -0400 Subject: [openib-general] low number of CQ entries allowed In-Reply-To: References: <20050603214400.GA28357@cse.ohio-state.edu> Message-ID: <20050603224317.GA28534@cse.ohio-state.edu> To add to this, I cannot run `pingpong' on r2539. Because of QP creation error. Thanks, Sayantan. [surs at x5:examples] ./a.out --help ./a.out: unrecognized option `--help' Usage: ./a.out start a server and wait for connection ./a.out connect to server at Options: -p, --port= listen on/connect to port (default 18515) -d, --ib-dev= use IB device (default first device found) -i, --ib-port= use port of IB device (default 1) -s, --size= size of message to exchange (default 4096) -r, --rx-depth= number of receives to post at a time (default 500) -n, --iters= number of exchanges (default 1000) -e, --events sleep on CQ events (default poll) [surs at x5:examples] ./a.out Couldn't create QP ^^^^^^^^^^^^^^^^^^ * On Jun,2 Shirley Ma wrote : > > I have no problem to create 65000 QPs on r2519. > > Shirley Ma > IBM Linux Technology Center > 15300 SW Koll Parkway > Beaverton, OR 97006-6063 > Phone(Fax): (503) 578-7638 > -- http://www.cse.ohio-state.edu/~surs From iod00d at hp.com Fri Jun 3 17:33:01 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 3 Jun 2005 17:33:01 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602234519.GA29972@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> Message-ID: <20050604003301.GB2538@esmail.cup.hp.com> On Fri, Jun 03, 2005 at 02:45:19AM +0300, Michael S. Tsirkin wrote: ... > Its possible to link libibverbs and libmthca statically. > I did it once. While I can get "mthca.a" to link statically, I get warnings and the binary doesn't work. -LOADLIBES += -libverbs +LOADLIBES += -static -libverbs -lpthread -lsysfs -ldl $ make cc -Wall -O2 -g -D_GNU_SOURCE rdma_lat.c get_clock.c -static -libverbs -lpthread -lsysfs -ldl -L/usr/lib/infiniband/ -lmthca -o rdma_lat /usr/lib/gcc-lib/ia64-linux/3.3.5/../../../libibverbs.a(src_libibverbs_la-init.o)(.text+0x22): In function `load_driver': : warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /tmp/ccxfeerJ.o(.text+0x132): In function `pp_connect_sock': /home/src/openib_gen2/src/userspace/perftest/rdma_lat.c:120: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking ... $ ldd rdma_lat not a dynamic executable grundler at gsyprf3$ taskset 1 ./rdma_lat -C -n 100000 libibverbs: Warning: no userspace device-specific driver found for uverbs0 driver search path: /usr/lib/infiniband No IB devices found Segmentation fault Sounds like libibverbs *must* dynamically load mthca.so in order to operate. I'll look at libibverbs source but this smells like a rat hole. I'm also wondering why "-dn" and "-Bstatic" do NOT produce static binary. "man ld" suggests those are equivalents to "-static". thanks, grant From xma at us.ibm.com Fri Jun 3 17:52:12 2005 From: xma at us.ibm.com (Shirley Ma) Date: Fri, 3 Jun 2005 17:52:12 -0700 Subject: [openib-general] low number of CQ entries allowed In-Reply-To: <20050603220941.GA28413@cse.ohio-state.edu> Message-ID: FYI. r2545 kernel + r2519 userspace works ok. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Postmaster at mx.bigpond.com Fri Jun 3 18:58:02 2005 From: Postmaster at mx.bigpond.com (Mail Administrator) Date: Sat, 4 Jun 2005 01:58:02 +0000 Subject: [openib-general] Mail System Error - Returned Mail Message-ID: <20050604015802.WLVI4114.omta05sl.mx.bigpond.com@omta05sl> This Message was undeliverable due to the following reason: Each of the following recipients was rejected by a remote mail server. The reasons given by the server are included to help you determine why each recipient was rejected. Recipient: Reason: : Recipient address rejected: User unknown in virtual alias table Please reply to if you feel this message to be in error. -------------- next part -------------- An embedded message was scrubbed... From: openib-general at openib.org Subject: something for you Date: Sat, 4 Jun 2005 09:55:31 +0800 Size: 49480 URL: From Administrator at stargate.chelsio.com Fri Jun 3 18:58:14 2005 From: Administrator at stargate.chelsio.com (Administrator at stargate.chelsio.com) Date: Fri, 3 Jun 2005 18:58:14 -0700 Subject: [openib-general] [MailServer Notification]To Recipient file blocking settings matched and action taken. Message-ID: <002f01c568a8$de859ce0$0fa0c00a@asicdesigners.com> ScanMail for Microsoft Exchange has blocked an attachment. Sender = openib-general-bounces at openib.org Recipient(s) = openib-general at openib.org Subject = [openib-general] Mail System Error - Returned Mail Scanning time = 6/3/2005 6:58:14 PM Action on file blocking: The attachment DOCUMENT.exe matches the file blocking settings. ScanMail has Quarantine failed it. The attachment was quarantined to C:\Program Files\Trend\Smex\Alert\DOCUMENT42a10ab619.exe_. Warning to Recipient: Action taken by attachment blocking. From hch at lst.de Sat Jun 4 03:30:58 2005 From: hch at lst.de (Christoph Hellwig) Date: Sat, 4 Jun 2005 12:30:58 +0200 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e0050603115887db8fa@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> <469958e00506030430687795a8@mail.gmail.com> <20050603172727.GA3888@lst.de> <469958e0050603115887db8fa@mail.gmail.com> Message-ID: <20050604103058.GB17004@lst.de> On Fri, Jun 03, 2005 at 11:58:50AM -0700, Caitlin Bestler wrote: > Since you haven't bothered to learn much about RDMA, let me > explain that unlike typical devices an RDMA device can be > pefrforming IO even when the OS is not aware of it. We have similar issues with spurious dma transfers and interrupts on all device classes. The ->remove callback must make sure the device is in a quiscent state, by whatever means are nessecary. This is absolutely nessecary for hot removals to work. > That is > why the Quadrics extensions for zero placement are not > applicable to an RDMA device. That extension, for example, > only remaps memory when the IO device is not actively > doing a transfer. Something that the kernel does not > know with an RDMA device. How is memory remapping related to device removal? From hch at lst.de Sat Jun 4 04:52:01 2005 From: hch at lst.de (Christoph Hellwig) Date: Sat, 4 Jun 2005 13:52:01 +0200 Subject: [openib-general] [PATCH] kDAPL: remove use of HANDLEs In-Reply-To: <1117837957.16881.21.camel@duffman> References: <1117837957.16881.21.camel@duffman> Message-ID: <20050604115201.GA18264@lst.de> On Fri, Jun 03, 2005 at 03:32:37PM -0700, Tom Duffy wrote: > This patch is HUGE. Sorry. Not much I could do to split it. > > It makes a superclass for all the dat objects, and subclasses > dat-provider (dapl) off of it. No more opaque handles. No more magic > checking. I actually found a few bugs from the compiler because of it > (more than reported to openib). > > The dapl_header struct is renamed dapl_common. Most of the _handle > variables are renamed removing _handle. It removes the > get_handle_type() function since this is no longer used. Also, it > unimplements [get|set]_consumer_context (for now) since nobody was using > it. DAT_BAD_HANDLE() goes as it becomes a simple NULL check. And event that should go away in most cases. In the kernel we should't check for NULL pointers all over the place, if client programmers don't stick to the APIs they deserve to get panics. From roland at topspin.com Fri Jun 3 16:37:47 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 03 Jun 2005 16:37:47 -0700 Subject: [openib-general] low number of CQ entries allowed In-Reply-To: <20050603214400.GA28357@cse.ohio-state.edu> (Sayantan Sur's message of "Fri, 3 Jun 2005 17:44:02 -0400") References: <20050603214400.GA28357@cse.ohio-state.edu> Message-ID: <52y89rroz8.fsf@topspin.com> Sayantan> Hi, Today I upgraded Gen2 drivers to revision 2539. I am Sayantan> seeing that there are CQ & QP creation problems with Sayantan> larger number of CQ entries. Are your kernel and userspace versions in sync? If so, can you send the output of strace -etrace=write -ewrite=all for some that fails? Thanks, Roland From mst at mellanox.co.il Sat Jun 4 11:12:37 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sat, 4 Jun 2005 21:12:37 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050604003301.GB2538@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> Message-ID: <20050604181237.GA18586@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [PATCH] rdma_lat-09 and results > > On Fri, Jun 03, 2005 at 02:45:19AM +0300, Michael S. Tsirkin wrote: > ... > > Its possible to link libibverbs and libmthca statically. > > I did it once. > > While I can get "mthca.a" to link statically, I get warnings > and the binary doesn't work. > > -LOADLIBES += -libverbs > +LOADLIBES += -static -libverbs -lpthread -lsysfs -ldl > > $ make > cc -Wall -O2 -g -D_GNU_SOURCE rdma_lat.c get_clock.c -static -libverbs -lpthread -lsysfs -ldl -L/usr/lib/infiniband/ -lmthca -o rdma_lat > /usr/lib/gcc-lib/ia64-linux/3.3.5/../../../libibverbs.a(src_libibverbs_la-init.o)(.text+0x22): In function `load_driver': > : warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > /tmp/ccxfeerJ.o(.text+0x132): In function `pp_connect_sock': > /home/src/openib_gen2/src/userspace/perftest/rdma_lat.c:120: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > ... > $ ldd rdma_lat > not a dynamic executable > grundler at gsyprf3$ taskset 1 ./rdma_lat -C -n 100000 > libibverbs: Warning: no userspace device-specific driver found for uverbs0 > driver search path: /usr/lib/infiniband > No IB devices found > Segmentation fault > > Sounds like libibverbs *must* dynamically load mthca.so in order to operate. > I'll look at libibverbs source but this smells like a rat hole. libibverbs uses dlsym(openib_driver_init) to get entry point into low level driver. dlsym documentation says: If the executable was linked with the flag "-rdynamic", then the global symbols in the executable will also be used to resolve references in a dynamically loaded library. So you need to link with -rdynamic. Maybe you also shall list mthca.a on linker's command line, and pass in -u openib_driver_init, I dont remember, exactly. -- MST From surs at cse.ohio-state.edu Sat Jun 4 12:15:18 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Sat, 4 Jun 2005 15:15:18 -0400 Subject: [openib-general] low number of CQ entries allowed In-Reply-To: <52y89rroz8.fsf@topspin.com> References: <20050603214400.GA28357@cse.ohio-state.edu> <52y89rroz8.fsf@topspin.com> Message-ID: <20050604191517.GA9257@cse.ohio-state.edu> Roland, * On Jun,8 Roland Dreier wrote : > Sayantan> Hi, Today I upgraded Gen2 drivers to revision 2539. I am > Sayantan> seeing that there are CQ & QP creation problems with > Sayantan> larger number of CQ entries. > > Are your kernel and userspace versions in sync? > > If so, can you send the output of > > strace -etrace=write -ewrite=all > > for some that fails? Thanks for your reply. I have made sure the kernel & userspace versions are in sync. Whenever I update drivers, I update for both. Info about code branches: [surs at q1:userspace] svn info Path: . URL: https://openib.org/svn/gen2/trunk/src/userspace Repository UUID: 21a7a0b7-18d7-0310-8e21-e8b31bdbf5cd Revision: 2546 Node Kind: directory Schedule: normal Last Changed Author: halr Last Changed Rev: 2546 Last Changed Date: 2005-06-04 05:28:20 -0400 (Sat, 04 Jun 2005) [surs at q1:linux-kernel] svn info Path: . URL: https://openib.org/svn/gen2/trunk/src/linux-kernel Repository UUID: 21a7a0b7-18d7-0310-8e21-e8b31bdbf5cd Revision: 2546 Node Kind: directory Schedule: normal Last Changed Author: halr Last Changed Rev: 2533 Last Changed Date: 2005-06-03 09:36:27 -0400 (Fri, 03 Jun 2005) ---------------------------------------------------------------- The ib_pingpong test included with the source fails like this: [surs at x1:examples] strace -etrace=write -ewrite=all ./ibv_pingpong write(3, "\0\0\0\0\4\0\1\0\324\364\377\377\377\177\0\0", 16) = 16 | 00000 00 00 00 00 04 00 01 00 d4 f4 ff ff ff 7f 00 00 ........ ........ | write(3, "\1\0\0\0\6\0\2\0 \364\377\377\377\177\0\0x\364\377\377"..., 24) = 24 | 00000 01 00 00 00 06 00 02 00 20 f4 ff ff ff 7f 00 00 ........ ....... | | 00010 78 f4 ff ff ff 7f 00 00 x....... | write(3, "\6\0\0\0\6\0\1\0\4\364\377\377\377\177\0\0(\364\377\377"..., 24) = 24 | 00000 06 00 00 00 06 00 01 00 04 f4 ff ff ff 7f 00 00 ........ ........ | | 00010 28 f4 ff ff ff 7f 00 00 (....... | write(3, "\6\0\0\0\6\0\1\0\224\364\377\377\377\177\0\0\270\364\377"..., 24) = 24 | 00000 06 00 00 00 06 00 01 00 94 f4 ff ff ff 7f 00 00 ........ ........ | | 00010 b8 f4 ff ff ff 7f 00 00 ........ | write(3, "\f\0\0\0\f\0\3\0P\364\377\377\377\177\0\0\0\260P\0\0\0"..., 48) = 48 | 00000 0c 00 00 00 0c 00 03 00 50 f4 ff ff ff 7f 00 00 ........ P....... | | 00010 00 b0 50 00 00 00 00 00 00 10 00 00 00 00 00 00 ..P..... ........ | | 00020 00 b0 50 00 00 00 00 00 01 00 00 00 01 00 00 00 ..P..... ........ | write(3, "\f\0\0\0\f\0\3\0\260\363\377\377\377\177\0\0\0\340P\0\0"..., 48) = 48 | 00000 0c 00 00 00 0c 00 03 00 b0 f3 ff ff ff 7f 00 00 ........ ........ | | 00010 00 e0 50 00 00 00 00 00 00 40 00 00 00 00 00 00 ..P..... . at ...... | | 00020 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ........ ........ | write(3, "\24\0\0\0\22\0\2\0@\364\377\377\377\177\0\0\240\207P\0"..., 72) = 72 | 00000 14 00 00 00 12 00 02 00 40 f4 ff ff ff 7f 00 00 ........ @....... | | 00010 a0 87 50 00 00 00 00 00 ff 01 00 00 39 00 00 00 ..P..... ....9... | | 00020 68 f4 ff ff ff 7f 00 00 00 27 00 08 07 00 02 00 h....... .'...... | | 00030 00 50 51 00 00 00 00 00 00 30 51 00 00 00 00 00 .PQ..... .0Q..... | | 00040 00 00 00 00 ff 7f 00 00 ........ | write(3, "\f\0\0\0\f\0\3\0\260\363\377\377\377\177\0\0\0pQ\0\0\0"..., 48) = -1 ENOMEM (Cannot allocate memory) write(2, "Couldn\'t create QP\n", 19Couldn't create QP ) = 19 | 00000 43 6f 75 6c 64 6e 27 74 20 63 72 65 61 74 65 20 Couldn't create | | 00010 51 50 0a QP. | Thanks, Sayantan. > > Thanks, > Roland -- http://www.cse.ohio-state.edu/~surs From roland at topspin.com Sat Jun 4 13:50:19 2005 From: roland at topspin.com (Roland Dreier) Date: Sat, 04 Jun 2005 13:50:19 -0700 Subject: [openib-general] low number of CQ entries allowed In-Reply-To: <20050604191517.GA9257@cse.ohio-state.edu> (Sayantan Sur's message of "Sat, 4 Jun 2005 15:15:18 -0400") References: <20050603214400.GA28357@cse.ohio-state.edu> <52y89rroz8.fsf@topspin.com> <20050604191517.GA9257@cse.ohio-state.edu> Message-ID: <527jh9sv78.fsf@topspin.com> Ah, I know what's happening -- the latest uverbs code now correctly checks against the user's limits on locked memory. I suspect if you do "ulimit -l" you'll see a relatively low number like "32". If you're running as root you should just be able to raise your ulimit by doing "ulimit -l unlimited". For regular users, your probably have to edit /etc/security/limits.conf to add a line like * soft memlock 32000 (And you may have to do some fiddling to get it to work with ssh login, eg https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=115442) - R. From surs at cse.ohio-state.edu Sat Jun 4 14:28:33 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Sat, 4 Jun 2005 17:28:33 -0400 Subject: [openib-general] low number of CQ entries allowed In-Reply-To: <527jh9sv78.fsf@topspin.com> References: <20050603214400.GA28357@cse.ohio-state.edu> <52y89rroz8.fsf@topspin.com> <20050604191517.GA9257@cse.ohio-state.edu> <527jh9sv78.fsf@topspin.com> Message-ID: <20050604212830.GA9858@cse.ohio-state.edu> Roland, * On Jun,11 Roland Dreier wrote : > Ah, I know what's happening -- the latest uverbs code now correctly > checks against the user's limits on locked memory. I suspect if you > do "ulimit -l" you'll see a relatively low number like "32". > > If you're running as root you should just be able to raise your ulimit > by doing "ulimit -l unlimited". For regular users, your probably have > to edit /etc/security/limits.conf to add a line like > > * soft memlock 32000 > > (And you may have to do some fiddling to get it to work with ssh > login, eg https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=115442) Thanks!! As usual, you are on target. This was the problem. I have set the locked memory limits to phys. memory limits now. Is there any documentation for OpenIB developers where such quirks are noted down? Thanks, Sayantan. > > - R. -- http://www.cse.ohio-state.edu/~surs From mst at mellanox.co.il Sat Jun 4 14:34:22 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 5 Jun 2005 00:34:22 +0300 Subject: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050601232123.GN25321@esmail.cup.hp.com> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> <20050601232123.GN25321@esmail.cup.hp.com> Message-ID: <20050604213422.GA23153@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest > > On Wed, May 25, 2005 at 07:57:27AM -0700, mst at openib.org wrote: > > gen2/trunk/src/userspace/perftest/rdma_bw.c > > Log: > > Bandwidth test for RDMA write > > > Michael, > > I finally had a chance to test rdma_bw....and it didn't work :^( > This was running r2519 IB kernel drivers, libibverbs, libmthca. > > Here is the rdma_bw output on HP IA64 ZX1 (rx2600) machines: > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw > local address: LID 0x27, QPN 0x70406, PSN 0xac2e45 RKey 0x2a0434 VAddr 0x6000000000014001 > remote address: LID 0x25, QPN 0x60406, PSN 0xf0f7a, RKey 0x240434 VAddr 0x6000000000014001 > server read: Address family not supported by protocol > 0/45: Couldn't read remote address Weird. What does this error mean? Could you try debugging it? > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw 10.0.0.51 > local address: LID 0x25, QPN 0x60406, PSN 0xf0f7a RKey 0x240434 VAddr 0x6000000000014001 > remote address: LID 0x27, QPN 0x70406, PSN 0xac2e45, RKey 0x2a0434 VAddr 0x6000000000014001 > Couldn't post send: scnt=0 -- MST From caitlin.bestler at gmail.com Sat Jun 4 16:33:12 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Sat, 4 Jun 2005 16:33:12 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <20050604103058.GB17004@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> <469958e00506030430687795a8@mail.gmail.com> <20050603172727.GA3888@lst.de> <469958e0050603115887db8fa@mail.gmail.com> <20050604103058.GB17004@lst.de> Message-ID: <469958e005060416332c9f4f9b@mail.gmail.com> On 6/4/05, Christoph Hellwig wrote: > > > That is > > why the Quadrics extensions for zero placement are not > > applicable to an RDMA device. That extension, for example, > > only remaps memory when the IO device is not actively > > doing a transfer. Something that the kernel does not > > know with an RDMA device. > > How is memory remapping related to device removal? > > Different problem, but the required solutions overlap. To remap a pinned memory region the OS must tell the RDMA device to suspend it's access to that memory. In doing so it must wait for in-progress transfers to complete. The RDMA Device is then in a state where it defers all activities associated with the suspended memory. Work Reqeusts that reference it are effectively fenced. Inbound packets that refernce it must be deferred (by dropping it, RNRs or interim buffering). After the OS has migrated the pages it supplies the new mapping to the RDMA Device, which lets it resume the Memory Region (after it has updated the PBLEs). The Quadrics patch informed zero copy devices of new mappings, but does so when the device is not actively doing work. That latter piece of information is not available for an RDMA device. For normal devices the OS has a reasonable expectation on when the device is performing DMA, because it is doing so in response to a request that the OS knows about. For an RDMA device a DMA transfer can be in progress as as a result of an RDMA Read -- something the host processor knows nothing about. From halr at voltaire.com Sat Jun 4 16:28:47 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 Jun 2005 19:28:47 -0400 Subject: [openib-general] [PATCH] [TRIVIAL] kdapl: Eliminate some dapl_evd_connection_callb.c compile warnings Message-ID: <1117927726.4493.5025.camel@hal.voltaire.com> kdapl: Eliminate some dapl_evd_connection_callb.c compile warnings Index: dapl_evd_connection_callb.c =================================================================== --- dapl_evd_connection_callb.c (revision 2546) +++ dapl_evd_connection_callb.c (working copy) @@ -31,6 +31,7 @@ #include "dapl.h" #include "dapl_evd_util.h" +#include "dapl_openib_util.h" /* * Connection callback function for ACTIVE connection requests; callbacks From support at ebay.com Sat Jun 4 17:55:40 2005 From: support at ebay.com (ebay) Date: Sun, 05 Jun 2005 01:55:40 +0100 Subject: [openib-general] Dear ebay Member Message-ID: An HTML attachment was scrubbed... URL: From info at dankichi3.net Sat Jun 4 20:47:07 2005 From: info at dankichi3.net (info at dankichi3.net) Date: 5 Jun 2005 12:47:07 +0900 Subject: [openib-general] $BB`2q<+M3$G$9(B Message-ID: <20050605034707.15335.qmail@mail.dankichi3.net> $B%(%C%A$@$1$N4X78!"1|MM$H$N5U!{8r:]!"=c?h$JK\5$$NNx0&$r5a$a$kJ}!"(B $B=O$J1|%5%^!"?'9aN)$D Hi All, I was running today some kdapltest and after about 10 tests i got this oops. the test was : ./kdapltest -T T -s 11.1.4.43 -D mthca0a -d -t 2 -w 8 -i 10 client SR server SR (the same test pass before it got the oops) my system is : dual x86 xeon 2G with 1GB memory + suse9.1 + k2.6.11 + svn rev 2546 + kdapl my kernel is with CONFIG_DEBUG_PAGEALLOC Jun 5 13:45:32 mtvs42 kernel: Unable to handle kernel paging request at virtual address ec298ff4 Jun 5 13:45:32 mtvs42 kernel: printing eip: Jun 5 13:45:32 mtvs42 kernel: f8ad3c15 Jun 5 13:45:32 mtvs42 kernel: *pde = 0047c067 Jun 5 13:45:32 mtvs42 kernel: Oops: 0002 [#1] Jun 5 13:45:32 mtvs42 kernel: SMP DEBUG_PAGEALLOC Jun 5 13:45:32 mtvs42 kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at dat ib_umad ib_ipoib ib_sa edd joydev st sd_mod sr_mod scsi_mod ide_cd cdrom nvram usbserial parport_pc lp parport autofs thermal processor fan button ipv6 battery ac e100 mii ib_mthca ib_mad ib_core i2c_i801 i2c_core uhci_hcd usbcore hw_random evdev dm_mod ext3 jbd Jun 5 13:45:32 mtvs42 kernel: CPU: 0 Jun 5 13:45:32 mtvs42 kernel: EIP: 0060:[pg0+946297877/1069155328] Not tainted VLI Jun 5 13:45:32 mtvs42 kernel: EIP: 0060:[] Not tainted VLI Jun 5 13:45:32 mtvs42 kernel: EFLAGS: 00010246 (2.6.11-gen2) Jun 5 13:45:32 mtvs42 kernel: EIP is at resolve_path+0xa5/0x100 [ib_at] Jun 5 13:45:32 mtvs42 kernel: eax: 00000000 ebx: ec298f3c ecx: f21b7e80 edx: 00000000 Jun 5 13:45:32 mtvs42 kernel: esi: ec79dae8 edi: 00000064 ebp: ec79dae8 esp: ec79daa8 Jun 5 13:45:32 mtvs42 kernel: ds: 007b es: 007b ss: 0068 Jun 5 13:45:32 mtvs42 kernel: Process DT_Mdep_Thread_ (pid: 9737, threadinfo=ec79c000 task=ebd18ad0) Jun 5 13:45:32 mtvs42 kernel: Stack: 000080fe 00000000 00c90200 51d00001 000080fe 00000000 01c90200 71f79d07 Jun 5 13:45:32 mtvs42 kernel: 00000000 00000000 00000000 00000000 00000000 ffff0001 00000000 00000000 Jun 5 13:45:32 mtvs42 kernel: 000080fe 00000000 00c90200 51d00001 000080fe 00000000 01c90200 71f79d07 Jun 5 13:45:32 mtvs42 kernel: Call Trace: Jun 5 13:45:32 mtvs42 kernel: [pg0+946298889/1069155328] ib_at_paths_by_route+0xa9/0xd0 [ib_at] Jun 5 13:45:32 mtvs42 kernel: [] ib_at_paths_by_route+0xa9/0xd0 [ib_at] Jun 5 13:45:32 mtvs42 kernel: [pg0+963317968/1069155328] dapl_rt_comp_handler+0x135/0x190 [ib_dat_provider] Jun 5 13:45:32 mtvs42 kernel: [] dapl_rt_comp_handler+0x135/0x190 [ib_dat_provider] Jun 5 13:45:32 mtvs42 kernel: [pg0+963319747/1069155328] dapl_ib_connect+0x698/0x6ae [ib_dat_provider] Jun 5 13:45:32 mtvs42 kernel: [] dapl_ib_connect+0x698/0x6ae [ib_dat_provider] Jun 5 13:45:32 mtvs42 kernel: [pg0+963336373/1069155328] dapl_ep_connect+0x73f/0x7b3 [ib_dat_provider] Jun 5 13:45:32 mtvs42 kernel: [] dapl_ep_connect+0x73f/0x7b3 [ib_dat_provider] Jun 5 13:45:32 mtvs42 kernel: [avc_has_perm_noaudit+92/256] avc_has_perm_noaudit+0x5c/0x100 Jun 5 13:45:32 mtvs42 kernel: [] avc_has_perm_noaudit+0x5c/0x100 Jun 5 13:45:33 mtvs42 kernel: [__change_page_attr+33/400] __change_page_attr+0x21/0x190 Jun 5 13:45:33 mtvs42 kernel: [] __change_page_attr+0x21/0x190 Jun 5 13:45:33 mtvs42 kernel: [scheduler_tick+193/720] scheduler_tick+0xc1/0x2d0 Jun 5 13:45:33 mtvs42 kernel: [] scheduler_tick+0xc1/0x2d0 Jun 5 13:45:33 mtvs42 kernel: [__change_page_attr+33/400] __change_page_attr+0x21/0x190 Jun 5 13:45:33 mtvs42 kernel: [] __change_page_attr+0x21/0x190 Jun 5 13:45:33 mtvs42 kernel: [change_page_attr+102/112] change_page_attr+0x66/0x70 Jun 5 13:45:33 mtvs42 kernel: [] change_page_attr+0x66/0x70 Jun 5 13:45:33 mtvs42 kernel: [kernel_map_pages+30/80] kernel_map_pages+0x1e/0x50 Jun 5 13:45:33 mtvs42 kernel: [] kernel_map_pages+0x1e/0x50 Jun 5 13:45:33 mtvs42 kernel: [cache_free_debugcheck+581/592] cache_free_debugcheck+0x245/0x250 Jun 5 13:45:33 mtvs42 kernel: [] cache_free_debugcheck+0x245/0x250 Jun 5 13:45:33 mtvs42 kernel: [__kfree_skb+166/320] __kfree_skb+0xa6/0x140 Jun 5 13:45:33 mtvs42 kernel: [] __kfree_skb+0xa6/0x140 Jun 5 13:45:33 mtvs42 kernel: [__kfree_skb+166/320] __kfree_skb+0xa6/0x140 Jun 5 13:45:33 mtvs42 kernel: [] __kfree_skb+0xa6/0x140 Jun 5 13:45:33 mtvs42 kernel: [tcp_rcv_established+468/2128] tcp_rcv_established+0x1d4/0x850 Jun 5 13:45:33 mtvs42 kernel: [] tcp_rcv_established+0x1d4/0x850 Jun 5 13:45:33 mtvs42 kernel: [tcp_v4_do_rcv+238/272] tcp_v4_do_rcv+0xee/0x110 Jun 5 13:45:33 mtvs42 kernel: [] tcp_v4_do_rcv+0xee/0x110 Jun 5 13:45:33 mtvs42 kernel: [tcp_v4_rcv+2047/2336] tcp_v4_rcv+0x7ff/0x920 Jun 5 13:45:33 mtvs42 kernel: [] tcp_v4_rcv+0x7ff/0x920 Jun 5 13:45:33 mtvs42 kernel: [__change_page_attr+33/400] __change_page_attr+0x21/0x190 Jun 5 13:45:33 mtvs42 kernel: [] __change_page_attr+0x21/0x190 Jun 5 13:45:33 mtvs42 kernel: [handle_IRQ_event+50/112] handle_IRQ_event+0x32/0x70 Jun 5 13:45:33 mtvs42 kernel: [] handle_IRQ_event+0x32/0x70 Jun 5 13:45:33 mtvs42 kernel: [__do_IRQ+257/320] __do_IRQ+0x101/0x140 Jun 5 13:45:33 mtvs42 kernel: [] __do_IRQ+0x101/0x140 Jun 5 13:45:33 mtvs42 kernel: [poison_obj+26/64] poison_obj+0x1a/0x40 Jun 5 13:45:33 mtvs42 kernel: [] poison_obj+0x1a/0x40 Jun 5 13:45:33 mtvs42 kernel: [cache_free_debugcheck+270/592] cache_free_debugcheck+0x10e/0x250 Jun 5 13:45:33 mtvs42 kernel: [] cache_free_debugcheck+0x10e/0x250 Jun 5 13:45:33 mtvs42 kernel: [avc_node_free+13/48] avc_node_free+0xd/0x30 Jun 5 13:45:33 mtvs42 kernel: [] avc_node_free+0xd/0x30 Jun 5 13:45:33 mtvs42 kernel: [avc_node_free+13/48] avc_node_free+0xd/0x30 Jun 5 13:45:33 mtvs42 kernel: [] avc_node_free+0xd/0x30 Jun 5 13:45:33 mtvs42 kernel: [__rcu_process_callbacks+144/192] __rcu_process_callbacks+0x90/0xc0 Jun 5 13:45:33 mtvs42 kernel: [] __rcu_process_callbacks+0x90/0xc0 Jun 5 13:45:33 mtvs42 kernel: [tasklet_action+85/192] tasklet_action+0x55/0xc0 Jun 5 13:45:33 mtvs42 kernel: [] tasklet_action+0x55/0xc0 Jun 5 13:45:33 mtvs42 kernel: [__do_softirq+100/224] __do_softirq+0x64/0xe0 Jun 5 13:45:33 mtvs42 kernel: [] __do_softirq+0x64/0xe0 Jun 5 13:45:33 mtvs42 kernel: [do_IRQ+61/96] do_IRQ+0x3d/0x60 Jun 5 13:45:33 mtvs42 kernel: [] do_IRQ+0x3d/0x60 Jun 5 13:45:33 mtvs42 kernel: [pg0+963435311/1069155328] DT_Transaction_Main+0xd9f/0x1bb0 [kdapltest] Jun 5 13:45:33 mtvs42 kernel: [] DT_Transaction_Main+0xd9f/0x1bb0 [kdapltest] Jun 5 13:45:33 mtvs42 kernel: [__change_page_attr+33/400] __change_page_attr+0x21/0x190 Jun 5 13:45:33 mtvs42 kernel: [] __change_page_attr+0x21/0x190 Jun 5 13:45:33 mtvs42 kernel: [change_page_attr+102/112] change_page_attr+0x66/0x70 Jun 5 13:45:33 mtvs42 kernel: [] change_page_attr+0x66/0x70 Jun 5 13:45:33 mtvs42 kernel: [pg0+963477858/1069155328] DT_Mdep_Thread_Start_Routine+0x12/0x20 [kdapltest] Jun 5 13:45:33 mtvs42 kernel: [] DT_Mdep_Thread_Start_Routine+0x12/0x20 [kdapltest] Jun 5 13:45:33 mtvs42 kernel: [pg0+963477840/1069155328] DT_Mdep_Thread_Start_Routine+0x0/0x20 [kdapltest] Jun 5 13:45:33 mtvs42 kernel: [] DT_Mdep_Thread_Start_Routine+0x0/0x20 [kdapltest] Jun 5 13:45:33 mtvs42 kernel: [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10 Jun 5 13:45:33 mtvs42 kernel: [] kernel_thread_helper+0x5/0x10 Jun 5 13:45:33 mtvs42 kernel: Code: 31 d2 8a 53 24 51 53 68 50 39 ad f8 68 d0 00 00 00 8b bb b0 00 00 00 89 e9 57 68 00 00 30 0c 6a 00 e8 50 48 01 00 83 c4 1c 85 c0 <89> 83 b8 00 00 00 78 18 8b b3 b0 00 00 00 31 c0 01 f6 89 b3 b0 -- Itamar From itamar at mellanox.co.il Sun Jun 5 06:03:51 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Sun, 5 Jun 2005 16:03:51 +0300 Subject: [openib-general] [ib_mad] oops when running kdapltest -T S (server side of test) Message-ID: <91DB792C7985D411BEC300B40080D29CC35C2A@mtvex01.mtv.mtl.com> Hi Hal, I sent you an oops in ib_mad last week . I still see this oops once in a while. it is allways in the server side of the kdapltest this oops is from today (svn rev 2546) my system is : dual x86 xeon 2G with 500MB memory + suse9.1 + k2.6.11 + svn rev 2546 + kdapl Jun 5 10:38:51 mtvs42 kernel: Kdapltest Module Loaded - major 254 Jun 5 10:43:49 mtvs42 kernel: ib_mthca 0000:04:00.0: Registering memory at 28cb8000 (iova 28cb8000) in PD 5; shift 12, npages 6. Jun 5 10:44:04 mtvs42 kernel: Unable to handle kernel paging request at virtual address e 7a9ff00 Jun 5 10:44:04 mtvs42 kernel: printing eip: Jun 5 10:44:04 mtvs42 kernel: f889563f Jun 5 10:44:04 mtvs42 kernel: *pde = 0046a067 Jun 5 10:44:04 mtvs42 kernel: Oops: 0000 [#1] Jun 5 10:44:04 mtvs42 kernel: SMP DEBUG_PAGEALLOC Jun 5 10:44:04 mtvs42 kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at da t ib_umad ib_ipoib ib_sa edd joydev st sd_mod sr_mod scsi_mod ide_cd cdrom nvram usbserial parport_pc lp parport autofs thermal processor fan button battery ac ipv6 e100 mii i2c_i8 01 ib_mthca ib_mad i2c_core ib_core uhci_hcd hw_random usbcore evdev dm_mod ext3 jbd Jun 5 10:44:04 mtvs42 kernel: CPU: 0 Jun 5 10:44:04 mtvs42 kernel: EIP: 0060:[pg0+943945279/1069155328] Not tainted VLI Jun 5 10:44:04 mtvs42 kernel: EIP: 0060:[] Not tainted VLI Jun 5 10:44:04 mtvs42 kernel: EFLAGS: 00010292 (2.6.11-gen2) Jun 5 10:44:05 mtvs42 kernel: EIP is at ib_mad_send_done_handler+0xf/0x120 [ib_mad] Jun 5 10:44:05 mtvs42 kernel: eax: f0aa5df8 ebx: e7a9fef8 ecx: f657332c edx: efcb7f 24 Jun 5 10:44:05 mtvs42 kernel: esi: f0aa5df8 edi: efcb7f24 ebp: e7a9fef8 esp: efcb7f 04 Jun 5 10:44:05 mtvs42 kernel: ds: 007b es: 007b ss: 0068 Jun 5 10:44:05 mtvs42 kernel: Process ib_mad1 (pid: 2794, threadinfo=efcb6000 task=f0b5fa d0) Jun 5 10:44:05 mtvs42 kernel: Stack: 00000001 efcb7f24 f0b54df8 f029a7f8 f0aa5df8 efcb7f2 4 f0aa5df8 f88958d9 Jun 5 10:44:05 mtvs42 kernel: e7a9fef8 00000000 00000000 00000000 efcb7f58 c011bc6 9 00000000 00000402 Jun 5 10:44:05 mtvs42 kernel: 00000006 00000000 00000071 efe747f8 f0aa5e68 efe747f 8 f0aa5e6c c012eea8 Jun 5 10:44:05 mtvs42 kernel: Call Trace: Jun 5 10:44:05 mtvs42 kernel: [pg0+943945945/1069155328] ib_mad_completion_handler+0x89/ 0xa0 [ib_mad] Jun 5 10:44:05 mtvs42 kernel: [] ib_mad_completion_handler+0x89/0xa0 [ib_mad] Jun 5 10:44:05 mtvs42 kernel: [__wake_up+41/64] __wake_up+0x29/0x40 Jun 5 10:44:05 mtvs42 kernel: [] __wake_up+0x29/0x40 Jun 5 10:44:05 mtvs42 kernel: [worker_thread+424/560] worker_thread+0x1a8/0x230 Jun 5 10:44:05 mtvs42 kernel: [] worker_thread+0x1a8/0x230 Jun 5 10:44:05 mtvs42 kernel: [pg0+943945808/1069155328] ib_mad_completion_handler+0x0/0 xa0 [ib_mad] Jun 5 10:44:05 mtvs42 kernel: [] ib_mad_completion_handler+0x0/0xa0 [ib_mad] Jun 5 10:44:05 mtvs42 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x1 0 Jun 5 10:44:05 mtvs42 kernel: [] default_wake_function+0x0/0x10 Jun 5 10:44:05 mtvs42 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x1 0 Jun 5 10:44:05 mtvs42 kernel: [] default_wake_function+0x0/0x10 Jun 5 10:44:05 mtvs42 kernel: [worker_thread+0/560] worker_thread+0x0/0x230 Jun 5 10:44:05 mtvs42 kernel: [] worker_thread+0x0/0x230 Jun 5 10:44:05 mtvs42 kernel: [kthread+136/176] kthread+0x88/0xb0 Jun 5 10:44:05 mtvs42 kernel: [] kthread+0x88/0xb0 Jun 5 10:44:05 mtvs42 kernel: [kthread+0/176] kthread+0x0/0xb0 Jun 5 10:44:05 mtvs42 kernel: [] kthread+0x0/0xb0 Jun 5 10:44:05 mtvs42 kernel: [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10 Jun 5 10:44:05 mtvs42 kernel: [] kernel_thread_helper+0x5/0x10 Jun 5 10:44:05 mtvs42 kernel: Code: 03 00 00 00 6a 00 e8 21 66 88 c7 5d eb 98 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 57 56 53 83 ec 0c 89 54 24 04 8b 1a 89 dd <8b> 73 08 8b 46 18 89 04 24 eb 50 8d b6 00 00 00 00 8b 54 24 04 Itamar Rabenstein Mellanox Technologies Ltd mailto : itamar at mellanox.co.il phone: 03-6259506 -------------- next part -------------- An HTML attachment was scrubbed... URL: From itamar at mellanox.co.il Sun Jun 5 07:32:46 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Sun, 5 Jun 2005 17:32:46 +0300 Subject: [openib-general] [ib_cm] oops when running kdapltest Message-ID: <91DB792C7985D411BEC300B40080D29CC35C2E@mtvex01.mtv.mtl.com> Hi Hal, Sean, I got this oops when running kdapltest -T T ... -t 2 -w 8 client SR server SR this is from the client side. kdapl svn rev 2546 Jun 5 17:00:02 mtvs42 kernel: kDAPL: dapl_ep_disconnect () returns 0x0 Jun 5 17:00:02 mtvs42 kernel: kDAPL: dapl_ep_free (ecfcfbf8) Jun 5 17:00:02 mtvs42 kernel: kDAPL: dapl_ep_disconnect (ecfcfbf8, 0) Jun 5 17:00:02 mtvs42 kernel: kDAPL: dapl_ep_disconnect () returns 0x0 Jun 5 17:00:02 mtvs42 kernel: kDAPL: dapl_ep_free: state b, ep ecfcfbf8 qp_state 1 qp eb60eef8 Jun 5 17:00:02 mtvs42 kernel: kDAPL: dapl_lmr_free (eb208c5c) Jun 5 17:00:02 mtvs42 kernel: kDAPL: dapl_lmr_free (eb208914) Jun 5 17:00:02 mtvs42 kernel: Unable to handle kernel paging request at virtual address ecfcfcb8 Jun 5 17:00:02 mtvs42 kernel: printing eip: Jun 5 17:00:02 mtvs42 kernel: faf978ef Jun 5 17:00:02 mtvs42 kernel: *pde = 0047f067 Jun 5 17:00:02 mtvs42 kernel: Oops: 0002 [#1] Jun 5 17:00:02 mtvs42 kernel: kDAPL: dapl_lmr_free (eb230658) Jun 5 17:00:02 mtvs42 kernel: SMP DEBUG_PAGEALLOC Jun 5 17:00:02 mtvs42 kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at dat ib_umad ib_ipoib ib_sa edd joydev st sd_mod sr_mod scsi_mod ide_cd cdrom nvram usbserial parport_pc lp par port autofs thermal processor fan ipv6 button battery ac e100 mii i2c_i801 i2c_core ib_mthca ib_mad ib_core hw_random uhci_hcd usbcore evdev dm_mod ext3 jbd Jun 5 17:00:02 mtvs42 kernel: CPU: 0 Jun 5 17:00:02 mtvs42 kernel: EIP: 0060:[pg0+984848623/1069155328] Not tainted VLI Jun 5 17:00:02 mtvs42 kernel: EIP: 0060:[] Not tainted VLI Jun 5 17:00:02 mtvs42 kernel: EFLAGS: 00010282 (2.6.11-gen2) Jun 5 17:00:02 mtvs42 kernel: EIP is at dapl_cm_active_cb_handler+0x1a8/0x1cc [ib_dat_provider] Jun 5 17:00:02 mtvs42 kernel: eax: ecfcfbf8 ebx: eb20adf8 ecx: eb01bfb4 edx: 00000200 Jun 5 17:00:02 mtvs42 kernel: esi: eb20adf8 edi: eb03aef8 ebp: eb20ae40 esp: ece07f04 Jun 5 17:00:02 mtvs42 kernel: ds: 007b es: 007b ss: 0068 Jun 5 17:00:02 mtvs42 kernel: Process ib_cm/0 (pid: 9643, threadinfo=ece06000 task=f17faad0) Jun 5 17:00:02 mtvs42 kernel: Stack: eb03af4c f1802ad0 00000296 00000001 eb01bef8 eb03af4c eb20adf 8 eb20adf8 Jun 5 17:00:02 mtvs42 kernel: fa9f76bd eb20adf8 00000001 eb03aef8 eb20ae40 fa9f8d83 0000028 2 00249f00 Jun 5 17:00:02 mtvs42 kernel: 00000000 eb03aef8 ec7617f8 eb03aefc eb03aef8 fa9fa5b7 eb03aef 8 c012eea8 Jun 5 17:00:02 mtvs42 kernel: Call Trace: Jun 5 17:00:02 mtvs42 kernel: [pg0+978949821/1069155328] cm_process_work+0xd/0xc0 [ib_cm] Jun 5 17:00:02 mtvs42 kernel: [] cm_process_work+0xd/0xc0 [ib_cm] Jun 5 17:00:02 mtvs42 kernel: [pg0+978955651/1069155328] cm_dreq_handler+0x193/0x1b0 [ib_cm] Jun 5 17:00:02 mtvs42 kernel: [] cm_dreq_handler+0x193/0x1b0 [ib_cm] Jun 5 17:00:02 mtvs42 kernel: [pg0+978961847/1069155328] cm_work_handler+0x77/0xa0 [ib_cm] Jun 5 17:00:02 mtvs42 kernel: [] cm_work_handler+0x77/0xa0 [ib_cm] Jun 5 17:00:02 mtvs42 kernel: [worker_thread+424/560] worker_thread+0x1a8/0x230 Jun 5 17:00:02 mtvs42 kernel: [] worker_thread+0x1a8/0x230 Jun 5 17:00:02 mtvs42 kernel: [pg0+978961728/1069155328] cm_work_handler+0x0/0xa0 [ib_cm] Jun 5 17:00:02 mtvs42 kernel: [] cm_work_handler+0x0/0xa0 [ib_cm] Jun 5 17:00:02 mtvs42 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 Jun 5 17:00:02 mtvs42 kernel: [] default_wake_function+0x0/0x10 Jun 5 17:00:02 mtvs42 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 Jun 5 17:00:02 mtvs42 kernel: [] default_wake_function+0x0/0x10 Jun 5 17:00:02 mtvs42 kernel: [worker_thread+0/560] worker_thread+0x0/0x230 Jun 5 17:00:02 mtvs42 kernel: [] worker_thread+0x0/0x230 Jun 5 17:00:02 mtvs42 kernel: [kthread+136/176] kthread+0x88/0xb0 Jun 5 17:00:02 mtvs42 kernel: [] kthread+0x88/0xb0 Jun 5 17:00:02 mtvs42 kernel: [kthread+0/176] kthread+0x0/0xb0 Jun 5 17:00:02 mtvs42 kernel: [] kthread+0x0/0xb0 Jun 5 17:00:02 mtvs42 kernel: [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10 Jun 5 17:00:03 mtvs42 kernel: [] kernel_thread_helper+0x5/0x10 Jun 5 17:00:03 mtvs42 kernel: Code: 44 24 10 05 bc 00 00 00 e8 3f 8f 37 c5 83 7c 24 0c 00 74 2a 8b 44 24 10 83 b8 b0 00 00 00 00 74 14 8b 44 24 10 8b 80 b0 00 00 00 80 c0 00 00 00 00 00 00 00 8b 44 24 10 e8 4e ed 1a c5 8b 44 Jun 5 17:00:03 mtvs42 kernel: <1>kDAPL: dapl_ep_disconnect (eb04ebf8, 0) Jun 5 17:00:03 mtvs42 kernel: kDAPL: >>> dapl_ib_disconnect: EP: eb04ebf8 conn eb021ef8 close fl ags 0 Jun 5 17:00:03 mtvs42 kernel: kDAPL: >>> dapl_destroy_cm_id: conn eb021ef8 CM ID eb206df8 Jun 5 17:00:03 mtvs42 kernel: kDAPL: dapl_evd_connection_callback: cm_handle: 00000000 event: 4005 ep_ptr: eb04ebf8 Jun 5 17:00:03 mtvs42 kernel: kDAPL: >>> dapl_ib_disconnect_clean: EP: eb04ebf8 active 1 Jun 5 17:00:03 mtvs42 kernel: kDAPL: >>> dapl_ib_disconnect: EP: eb04ebf8 conn 00000000 close fl ags 0 Jun 5 17:00:03 mtvs42 kernel: kDAPL: dapl_evd_post_event: Called with event # 4005 Itamar Rabenstein Mellanox Technologies Ltd mailto : itamar at mellanox.co.il phone: 03-6259506 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bohra at cs.rutgers.edu Sun Jun 5 09:45:10 2005 From: bohra at cs.rutgers.edu (Aniruddha Bohra) Date: Sun, 05 Jun 2005 12:45:10 -0400 Subject: [openib-general] Using opensm on Fedora Core 3 Message-ID: <42A32C16.7070403@cs.rutgers.edu> Hi, I tried installing the OpenIB stack on Linux 2.6 kernel with the gen2 userland. How can I get the thing to work correctly. Any pointers are appreciated. Thanks Aniruddha I have the following modules loaded : ib_umad 17688 0 ipv6 262560 18 autofs4 23172 0 sunrpc 140868 1 dm_mod 61956 0 video 20100 0 button 10896 0 battery 13700 0 ac 9092 0 uhci_hcd 35088 0 hw_random 9876 0 ib_mthca 85660 0 ib_mad 33568 2 ib_umad,ib_mthca ib_core 44800 3 ib_umad,ib_mthca,ib_mad e1000 87476 0 floppy 62992 0 pci 9216 0 chipreg 7168 1 pci mtdcore 11336 1 pci mptscsih 38432 0 mptbase 46304 1 mptscsih sd_mod 22144 3 scsi_mod 130376 2 mptscsih,sd_mod However, running opensm leads to the following errors : ------------------------------------------------- OpenSM Rev:openib-1.0.0 Command Line Arguments: Log File: /var/log/osm.log ------------------------------------------------- warn: [5164] umad_init: wrong ABI version: /sys/class/infiniband_mad/abi_version is 2 but library ABI is 3 using default guid 0x2c901081e4b51 Error from osm_opensm_bind (0x2A) The log file shows the following : Jun 05 12:17:33 [B7FD86C0] -> osm_vendor_bind: [ Jun 05 12:17:33 [B7FD86C0] -> osm_vendor_bind: Binding to port 0x2c901081e4b51. Jun 05 12:17:33 [B7FD86C0] -> osm_vendor_open_port: [ Jun 05 12:17:33 [B7FD86C0] -> osm_vendor_open_port: ] Jun 05 12:17:33 [B7FD86C0] -> osm_vendor_bind: Unable to Open Port 0x2c901081e4b51. Jun 05 12:17:33 [B7FD86C0] -> osm_vendor_bind: ] Jun 05 12:17:33 [B7FD86C0] -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind() failed. Jun 05 12:17:33 [B7FD86C0] -> osm_sm_mad_ctrl_bind: ] Jun 05 12:17:33 [B7FD86C0] -> osm_sm_bind: ERR 2E10: SM MAD Controller bind() failed (IB_ERROR). Jun 05 12:17:33 [B7FD86C0] -> osm_sm_bind: ] Jun 05 12:20:46 [B7FD86C0] -> osm_vendor_init: Error opening UMAD Jun 05 12:20:46 [B7FD86C0] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0xfe80000000000000,0x0000000000000000 Jun 05 12:20:46 [B7FD86C0] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0xfe80000000000000,0x0000000000000000 Jun 05 12:20:46 [B7FD86C0] -> osm_vendor_get_all_port_attr: assign CA mthca0 port 1 guid (0x2c901081e4b51) as the default port. Jun 05 12:20:46 [B7FD86C0] -> osm_vendor_bind: Binding to port 0x2c901081e4b51. Jun 05 12:20:46 [B7FD86C0] -> osm_vendor_bind: Unable to Open Port 0x2c901081e4b51. Jun 05 12:20:46 [B7FD86C0] -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind() failed. Jun 05 12:20:46 [B7FD86C0] -> osm_sm_bind: ERR 2E10: SM MAD Controller bind() failed (IB_ERROR). From tduffy at sun.com Sun Jun 5 10:47:43 2005 From: tduffy at sun.com (Tom Duffy) Date: Sun, 05 Jun 2005 10:47:43 -0700 Subject: [openib-general] Using opensm on Fedora Core 3 In-Reply-To: <42A32C16.7070403@cs.rutgers.edu> References: <42A32C16.7070403@cs.rutgers.edu> Message-ID: <1117993663.4722.0.camel@duffman> On Sun, 2005-06-05 at 12:45 -0400, Aniruddha Bohra wrote: > /sys/class/infiniband_mad/abi_version is 2 but library ABI is 3 There is your problem. You will need to run a version of userland that matches your kernel. Update both to the latest and you should be off to the races. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From bohra at cs.rutgers.edu Sun Jun 5 11:42:40 2005 From: bohra at cs.rutgers.edu (Aniruddha Bohra) Date: Sun, 05 Jun 2005 14:42:40 -0400 Subject: [openib-general] Using opensm on Fedora Core 3 In-Reply-To: <1117993663.4722.0.camel@duffman> References: <42A32C16.7070403@cs.rutgers.edu> <1117993663.4722.0.camel@duffman> Message-ID: <42A347A0.4040802@cs.rutgers.edu> Hello, Yes I updated that. Also, added the device rule in udev. Now everything works. Thanks Aniruddha Tom Duffy wrote: >On Sun, 2005-06-05 at 12:45 -0400, Aniruddha Bohra wrote: > > >>/sys/class/infiniband_mad/abi_version is 2 but library ABI is 3 >> >> > >There is your problem. You will need to run a version of userland that >matches your kernel. Update both to the latest and you should be off to >the races. > >-tduffy > > From kjreilly at us.ibm.com Sun Jun 5 18:27:52 2005 From: kjreilly at us.ibm.com (Kevin Reilly) Date: Sun, 5 Jun 2005 21:27:52 -0400 Subject: [openib-general] What is the correct method of determining the number of ports Message-ID: I'm been hunting around for the proper method for determining the number of ports a device/hca has by using the gen2 user access library. I can't figure out how except by loop through port numbers calling ibv_query_port(). I figure maybe i'm missing something? Can somebody steer me in the right direction? Kevin J. Reilly STSM, HPC Architecture -Federation/HPS Chief Engineer -HPC interconnect architect (office) 845-433-7976 (tieline) 8-293-7976 From surs at cse.ohio-state.edu Sun Jun 5 19:23:57 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Sun, 5 Jun 2005 22:23:57 -0400 Subject: [openib-general] error messages for debugging Message-ID: <20050606022357.GA28138@cse.ohio-state.edu> Hi, Is there any way of knowing the reason ibv_blah_blah() failed? All I get on return is a -1/NULL. VAPI provides quite helpful error codes on return, with even string equivalents with VAPI_strerror(). I find that method a very easy way to debug. Thanks, Sayantan. -- http://www.cse.ohio-state.edu/~surs From surs at cse.ohio-state.edu Sun Jun 5 19:52:38 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Sun, 5 Jun 2005 22:52:38 -0400 Subject: [openib-general] error messages for debugging In-Reply-To: <20050606022357.GA28138@cse.ohio-state.edu> References: <20050606022357.GA28138@cse.ohio-state.edu> Message-ID: <20050606025235.GB28138@cse.ohio-state.edu> Hi, Upon more debugging, I seem to be hitting this error condition in file libmthca/src:qp.c [SVN revsision: 2455] 592 if (wr->opcode >= sizeof mthca_opcode / sizeof mthca_opcode[0]) { 593 ret = -1; 594 *bad_wr = wr; 595 goto out; 596 } After the post, a dump of the descriptor looks like this: (gdb) print *bad_wr $2 = {next = 0x0, wr_id = 69288064, sg_list = 0x4217050, num_sge = 1, opcode = IBV_WR_RDMA_WRITE, send_flags = 10, imm_data = 0, wr = {rdma = { remote_addr = 69681408, rkey = 2046832896}, atomic = { remote_addr = 69681408, compare_add = 2046832896, swap = 0, rkey = 0}, ud = {ah = 0x4274100, remote_qpn = 2046832896, remote_qkey = 0}}} This doesn't seem to be an error with remote address or keys, since this error is generated locally just after a post operation. Any clues to why I am hitting this condition will be appreciated!! Thanks, Sayantan. * On Jun,1 Sayantan Sur wrote : > Hi, > > Is there any way of knowing the reason ibv_blah_blah() failed? All I get > on return is a -1/NULL. VAPI provides quite helpful error codes on > return, with even string equivalents with VAPI_strerror(). I find that > method a very easy way to debug. > > Thanks, > Sayantan. > > > -- > http://www.cse.ohio-state.edu/~surs > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- http://www.cse.ohio-state.edu/~surs From xma at us.ibm.com Sun Jun 5 21:09:26 2005 From: xma at us.ibm.com (Shirley Ma) Date: Sun, 5 Jun 2005 21:09:26 -0700 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: Message-ID: ibv_get_devices() could help to get the number of ports for each device. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Sun Jun 5 22:07:24 2005 From: xma at us.ibm.com (Shirley Ma) Date: Sun, 5 Jun 2005 22:07:24 -0700 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: Message-ID: I was thinking to use ibv_get_devices() to get the list of device names, then use umad_get_ca() to get ca info. (it actually gets the info from directory /sys/class/infiniband.) Which might not what you want. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From addy at quadrics.com Mon Jun 6 01:34:45 2005 From: addy at quadrics.com (David Addison) Date: Mon, 06 Jun 2005 09:34:45 +0100 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e005060416332c9f4f9b@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> <469958e00506030430687795a8@mail.gmail.com> <20050603172727.GA3888@lst.de> <469958e0050603115887db8fa@mail.gmail.com> <20050604103058.GB17004@lst.de> <469958e005060416332c9f4f9b@mail.gmail.com> Message-ID: <42A40AA5.5020307@quadrics.com> Caitlin Bestler wrote: > On 6/4/05, Christoph Hellwig wrote: > > The Quadrics patch informed zero copy devices > of new mappings, but does so when the device > is not actively doing work. That latter piece of > information is not available for an RDMA device. > For normal devices the OS has a reasonable > expectation on when the device is performing > DMA, because it is doing so in response to a > request that the OS knows about. For an RDMA > device a DMA transfer can be in progress as > as a result of an RDMA Read -- something the > host processor knows nothing about. The Quadrics devices can handle changes in page mappings whilst RDMA operations are in progress. If say a page is being evicted due to swap pressure, then the Quadrics NIC is first informed by an IOPROC invalidate call. This will cause any future device accesses to that page to fault and the network operation to stall. The Quadrics NIC will continue the RDMA once a new valid translation has been loaded by the OS. Cheers Addy. From tcpgate at didamail.com Mon Jun 6 07:07:47 2005 From: tcpgate at didamail.com (Freda Montano) Date: Mon, 06 Jun 2005 15:07:47 +0100 Subject: [openib-general] Your account #194Z7621 Message-ID: <905c.fsf@calle92.net> Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $400,000 for as little as $400 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.t0wers.net/signs.asp Best Regards, Trenton Ortiz to be remov(ed: http://www.t0wers.net/deletion.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From halr at voltaire.com Mon Jun 6 07:32:43 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 Jun 2005 10:32:43 -0400 Subject: [openib-general] [PATCH] user_mad: support retries in ib_umad_write Message-ID: <1118068362.1622.304.camel@hal.voltaire.com> user_mad: Support retries in ib_umad_write This is another ABI version change. There will be parallel changes in userspace/management to go along with this change. Signed-off-by: Hal Rosenstock Index: include/ib_user_mad.h =================================================================== --- include/ib_user_mad.h (revision 2520) +++ include/ib_user_mad.h (working copy) @@ -43,7 +43,7 @@ * Increment this value if any changes that break userspace ABI * compatibility are made. */ -#define IB_USER_MAD_ABI_VERSION 3 +#define IB_USER_MAD_ABI_VERSION 4 /* * Make sure that all structs defined in this file remain laid out so @@ -58,6 +58,7 @@ * received (transaction ID in data[] will be set to TID of original * request) (ignored on send) * @timeout_ms - Milliseconds to wait for response (unset on receive) + * @retries - Number of automatic retries to attempt * @qpn - Remote QP number received from/to be sent to * @qkey - Remote Q_Key to be sent with (unset on receive) * @lid - Remote lid received from/to be sent to @@ -76,6 +77,7 @@ __u32 id; __u32 status; __u32 timeout_ms; + __u32 retries; __u32 qpn; __u32 qkey; __u16 lid; Index: core/user_mad.c =================================================================== --- core/user_mad.c (revision 2533) +++ core/user_mad.c (working copy) @@ -348,6 +348,8 @@ } packet->msg->send_wr.wr.ud.timeout_ms = packet->mad.hdr.timeout_ms; + packet->msg->send_wr.wr.ud.retries = packet->mad.hdr.retries; + /* Override send WR WRID initialized in ib_create_send_mad */ packet->msg->send_wr.wr_id = (unsigned long) packet; From simon at compucall.co.il Mon Jun 6 08:50:43 2005 From: simon at compucall.co.il (Simon Gelfand) Date: Mon, 6 Jun 2005 17:50:43 +0200 Subject: [openib-general] Sanrad iSCSI solutions Message-ID: <6e82e951a1f3dafcecfd5b224d83ae@home> Dear openib.org Manager, I visited your web site and found some great information regarding iSCSI solutions. I thought you might be interested to know that we have a web site dedicated to iSCSI and IP SAN solutions. We were hoping that you might consider exchanging links with us and invite you to review our site at your convenience. If you determine that a link to our site is appropriate, please add it at your discretion, or might we suggest the following link and description: Title: iSCSI - Sanrad Description: iSCSI protocol enables high performance SANs over standard IP networks. Find out more about SANRAD iSCSI SAN solutions. URL: http://www.sanrad.com Or by pasting the below code: iSCSI
iSCSI protocol enables high performance SANs over standard IP networks. Find out more about SANRAD iSCSI SAN solutions.
If you'd like to discuss this further, please feel free to contact us for more details. You can read more on the subject at http://www.sanrad.com/iscsi.asp?IP-SAN=348. I'll be waiting for your kind answer and confirmation. Best wishes, Simon Gelfand Research & Information Management Compucall Web Marketing Ltd. On behalf of Sanrad iSCSI & IP SAN Solutions http://www.sanrad.com ______________________ Tel: +972-3-644-7172 From roland at topspin.com Mon Jun 6 08:18:17 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 08:18:17 -0700 Subject: [openib-general] What is the correct method of determining the number of ports References: Message-ID: <52oeajpl8m.fsf@topspin.com> Kevin> I'm been hunting around for the proper method for Kevin> determining the number of ports a device/hca has by using Kevin> the gen2 user access library. I can't figure out how except Kevin> by loop through port numbers calling ibv_query_port(). I Kevin> figure maybe i'm missing something? Can somebody steer me Kevin> in the right direction? Unfortunately there's not a good way at the moment. I still need to implement ibv_query_device() for userspace. Thanks for reminding me -- I should be able to get to it this week. - R. From roland at topspin.com Mon Jun 6 08:18:18 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 08:18:18 -0700 Subject: [openib-general] low number of CQ entries allowed References: <20050603214400.GA28357@cse.ohio-state.edu> <52y89rroz8.fsf@topspin.com> <20050604191517.GA9257@cse.ohio-state.edu> <527jh9sv78.fsf@topspin.com> <20050604212830.GA9858@cse.ohio-state.edu> Message-ID: <52is0rpl8l.fsf@topspin.com> Sayantan> Is there any documentation for OpenIB developers where Sayantan> such quirks are noted down? Not really -- I'll be starting a README for libibverbs soon but there's nothing written yet. If you want to add content to the OpenIB Wiki at https://openib.org/tiki/tiki-index.php, that would be great. - R. From roland at topspin.com Mon Jun 6 08:33:17 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 08:33:17 -0700 Subject: [openib-general] error messages for debugging References: <20050606022357.GA28138@cse.ohio-state.edu> <20050606025235.GB28138@cse.ohio-state.edu> Message-ID: <52d5qzpkjm.fsf@topspin.com> Sayantan> Upon more debugging, I seem to be hitting this error Sayantan> condition in file libmthca/src:qp.c > 592 if (wr->opcode >= sizeof mthca_opcode / sizeof mthca_opcode[0]) { If you're hitting that, then wr->opcode is out of range. Can you add a print and dump the value of wr->opcode and (sizeof mthca_opcode / sizeof mthca_opcode[0]) so we can see if something is bogus? This doesn't fit with your dump of the bad_wr, which shows the opcode is IBV_WR_RDMA_WRITE, which should be fine. Something doesn't add up here... - R. From surs at cse.ohio-state.edu Mon Jun 6 09:02:23 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Mon, 6 Jun 2005 12:02:23 -0400 Subject: [openib-general] error messages for debugging In-Reply-To: <52d5qzpkjm.fsf@topspin.com> References: <20050606022357.GA28138@cse.ohio-state.edu> <20050606025235.GB28138@cse.ohio-state.edu> <52d5qzpkjm.fsf@topspin.com> Message-ID: <20050606160222.GA3849@cse.ohio-state.edu> * On Jun,3 Roland Dreier wrote : > Sayantan> Upon more debugging, I seem to be hitting this error > Sayantan> condition in file libmthca/src:qp.c > > > 592 if (wr->opcode >= sizeof mthca_opcode / sizeof mthca_opcode[0]) { > > If you're hitting that, then wr->opcode is out of range. Can you add > a print and dump the value of wr->opcode and (sizeof mthca_opcode / > sizeof mthca_opcode[0]) so we can see if something is bogus? > > This doesn't fit with your dump of the bad_wr, which shows the opcode > is IBV_WR_RDMA_WRITE, which should be fine. > > Something doesn't add up here... I seem to have gotten around this issue. When I was creating the QP, I didn't specify any `max_inline_data' (i.e. left attr.qp.max_inline_data completely untouched). After I specify the `max_inline_data' and use IBV_SEND_INLINE in the `send_flags', it works. Funny that today my debugger says that the error condition is actually caught in qp.c (when I try to reproduce that condition): 562 s += sge->length; (gdb) 564 if (s > max_size) { (gdb) 594 *bad_wr = wr; This is clearly a problem with wrong inline size. Anyways, if I hit that problem again, I'll post the gdb output. Thanks, Sayantan. > > - R. -- http://www.cse.ohio-state.edu/~surs From mshefty at ichips.intel.com Mon Jun 6 09:19:38 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 06 Jun 2005 09:19:38 -0700 Subject: [openib-general] [ib_mad] oops when running kdapltest -T S (server side of test) In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C2A@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C2A@mtvex01.mtv.mtl.com> Message-ID: <42A4779A.2070601@ichips.intel.com> Itamar Rabenstein wrote: > Hi Hal, > > I sent you an oops in ib_mad last week . > I still see this oops once in a while. > it is allways in the server side of the kdapltest > this oops is from today (svn rev 2546) > my system is : Thanks for the info. I'm in the process of trying to track down the cause of this crash. From what I've seen so far, I believe that the cause is in a different portion of the code, possibly in another thread. - Sean From iod00d at hp.com Mon Jun 6 09:41:38 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 6 Jun 2005 09:41:38 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e005060416332c9f4f9b@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> <469958e00506030430687795a8@mail.gmail.com> <20050603172727.GA3888@lst.de> <469958e0050603115887db8fa@mail.gmail.com> <20050604103058.GB17004@lst.de> <469958e005060416332c9f4f9b@mail.gmail.com> Message-ID: <20050606164138.GA11120@esmail.cup.hp.com> On Sat, Jun 04, 2005 at 04:33:12PM -0700, Caitlin Bestler wrote: ... > The Quadrics patch informed zero copy devices > of new mappings, but does so when the device > is not actively doing work. That latter piece of > information is not available for an RDMA device. It certainly is. Anyway, I agree with Christoph. > For normal devices the OS has a reasonable > expectation on when the device is performing > DMA, because it is doing so in response to a > request that the OS knows about. For an RDMA > device a DMA transfer can be in progress as > as a result of an RDMA Read -- something the > host processor knows nothing about. That's definitely not true. Host maps all DMA. No one can DMA to/from the host without registering the memory for a DMA mapping. The host does NOT know when any given DMA mapping is active - but that's true for FC/SCSI devices today too. The device driver must take reasonable measures to quiesce that DMA mapping. Maybe send a signal to the task that owns the DMA mapping? Ie warn the task, wait a bit, then kill the task with extreme prejudice if the resources it's sitting on aren't free yet. I don't expect Hotplug support to be simple to implement. But it's definitely possible. grant From millere at emailaccount.com Mon Jun 6 11:16:28 2005 From: millere at emailaccount.com (Mitzi Jorgensen) Date: Mon, 06 Jun 2005 17:16:28 -0100 Subject: [openib-general] Your account #751Z7158 Message-ID: <372c.fsf@calle53.net> Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $400,000 for as little as $400 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.t0wers.net/signs.asp Best Regards, Ramona Fuller to be remov(ed: http://www.t0wers.net/deletion.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From kjreilly at us.ibm.com Mon Jun 6 11:21:45 2005 From: kjreilly at us.ibm.com (Kevin Reilly) Date: Mon, 6 Jun 2005 14:21:45 -0400 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: <52oeajpl8m.fsf@topspin.com> Message-ID: Thanks you Roland and Shirley for your responses!! Roland, Do you know other things that are missing that you can rattle off the top of your head? Else we keep looking for what we need then ask questions if we don't find them. Kevin J. Reilly STSM, HPC Architecture -Federation/HPS Chief Engineer -HPC interconnect architect (office) 845-433-7976 (tieline) 8-293-7976 Roland Dreier To Kevin Reilly/Poughkeepsie/IBM at IBMUS 06/06/2005 11:18 cc AM openib-general at openib.org Subject Re: [openib-general] What is the correct method of determining the number of ports Kevin> I'm been hunting around for the proper method for Kevin> determining the number of ports a device/hca has by using Kevin> the gen2 user access library. I can't figure out how except Kevin> by loop through port numbers calling ibv_query_port(). I Kevin> figure maybe i'm missing something? Can somebody steer me Kevin> in the right direction? Unfortunately there's not a good way at the moment. I still need to implement ibv_query_device() for userspace. Thanks for reminding me -- I should be able to get to it this week. - R. From tduffy at sun.com Mon Jun 6 11:48:43 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 06 Jun 2005 11:48:43 -0700 Subject: [openib-general] Re: [openib-commits] r2551 - in gen2: branches/xen-mlx-branch/src/linux-kernel/infiniband/core trunk/src/linux-kernel/infiniband trunk/src/linux-kernel/infiniband/core In-Reply-To: <20050606104132.842AA2283D4@openib.ca.sandia.gov> References: <20050606104132.842AA2283D4@openib.ca.sandia.gov> Message-ID: <1118083723.6375.0.camel@duffman> On Mon, 2005-06-06 at 03:41 -0700, gdror at openib.org wrote: > Modified: > gen2/trunk/src/linux-kernel/infiniband/Kconfig > gen2/trunk/src/linux-kernel/infiniband/Makefile > gen2/trunk/src/linux-kernel/infiniband/core/Makefile > Log: > initial checkin for gverbs support. Most of files just copied > from main trunk for now. Um...you really didn't mean to do this, did you? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From sean.hefty at intel.com Mon Jun 6 12:06:04 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 6 Jun 2005 12:06:04 -0700 Subject: [openib-general] [PATCH] [MAD] fix locking and cancel MAD issues Message-ID: The following patch fixes a locking issue handling errors posting a send MAD work request. It also corrects an issue canceling a MAD that may have an active send work request at the time that it is canceled. This should fix some CM related issues that are visible, and _may_ fix some of the MAD issues that people are seeing, since it changes locking around the QP. (I believe that there is likely another issue remaining somewhere else.) Signed-off-by: Sean Hefty Index: mad.c =================================================================== --- mad.c (revision 2519) +++ mad.c (working copy) @@ -858,6 +858,7 @@ int ib_send_mad(struct ib_mad_send_wr_pr { struct ib_mad_qp_info *qp_info; struct ib_send_wr *bad_send_wr; + struct list_head *list; unsigned long flags; int ret; @@ -867,22 +868,20 @@ int ib_send_mad(struct ib_mad_send_wr_pr mad_send_wr->mad_list.mad_queue = &qp_info->send_queue; spin_lock_irqsave(&qp_info->send_queue.lock, flags); - if (qp_info->send_queue.count++ < qp_info->send_queue.max_active) { - list_add_tail(&mad_send_wr->mad_list.list, - &qp_info->send_queue.list); - spin_unlock_irqrestore(&qp_info->send_queue.lock, flags); + if (qp_info->send_queue.count < qp_info->send_queue.max_active) { ret = ib_post_send(mad_send_wr->mad_agent_priv->agent.qp, &mad_send_wr->send_wr, &bad_send_wr); - if (ret) { - printk(KERN_ERR PFX "ib_post_send failed: %d\n", ret); - dequeue_mad(&mad_send_wr->mad_list); - } + list = &qp_info->send_queue.list; } else { - list_add_tail(&mad_send_wr->mad_list.list, - &qp_info->overflow_list); - spin_unlock_irqrestore(&qp_info->send_queue.lock, flags); ret = 0; + list = &qp_info->overflow_list; } + + if (!ret) { + qp_info->send_queue.count++; + list_add_tail(&mad_send_wr->mad_list.list, list); + } + spin_unlock_irqrestore(&qp_info->send_queue.lock, flags); return ret; } @@ -2099,6 +2098,7 @@ int ib_modify_mad(struct ib_mad_agent *m struct ib_mad_agent_private *mad_agent_priv; struct ib_mad_send_wr_private *mad_send_wr; unsigned long flags; + int active; mad_agent_priv = container_of(mad_agent, struct ib_mad_agent_private, agent); @@ -2109,13 +2109,14 @@ int ib_modify_mad(struct ib_mad_agent *m return -EINVAL; } + active = (!mad_send_wr->timeout || mad_send_wr->refcount > 1); if (!timeout_ms) { mad_send_wr->status = IB_WC_WR_FLUSH_ERR; mad_send_wr->refcount -= (mad_send_wr->timeout > 0); } mad_send_wr->send_wr.wr.ud.timeout_ms = timeout_ms; - if (!mad_send_wr->timeout || mad_send_wr->refcount > 1) + if (active) mad_send_wr->timeout = msecs_to_jiffies(timeout_ms); else ib_reset_mad_timeout(mad_send_wr, timeout_ms); From mshefty at ichips.intel.com Mon Jun 6 12:09:44 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 06 Jun 2005 12:09:44 -0700 Subject: [openib-general] Re: [ib_cm] oops when running kdapltest In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C2E@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C2E@mtvex01.mtv.mtl.com> Message-ID: <42A49F77.5040208@ichips.intel.com> Itamar Rabenstein wrote: > Hi Hal, Sean, > > I got this oops when running kdapltest -T T ... -t 2 -w 8 client SR server > SR > this is from the client side. > kdapl svn rev 2546 Can you please try re-running with 2558 and let me know if you still see this problem? Thanks, Sean From caitlin.bestler at gmail.com Mon Jun 6 12:15:53 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Mon, 6 Jun 2005 12:15:53 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <20050606164138.GA11120@esmail.cup.hp.com> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> <469958e00506030430687795a8@mail.gmail.com> <20050603172727.GA3888@lst.de> <469958e0050603115887db8fa@mail.gmail.com> <20050604103058.GB17004@lst.de> <469958e005060416332c9f4f9b@mail.gmail.com> <20050606164138.GA11120@esmail.cup.hp.com> Message-ID: <469958e0050606121567925a45@mail.gmail.com> On 6/6/05, Grant Grundler wrote: > On Sat, Jun 04, 2005 at 04:33:12PM -0700, Caitlin Bestler wrote: > ... > > The Quadrics patch informed zero copy devices > > of new mappings, but does so when the device > > is not actively doing work. That latter piece of > > information is not available for an RDMA device. > > It certainly is. > As your reply indicates below, the OS does *not* know *when* the device is active. The memory is registered as being device accessible for an indefinite period of time (must likely until the machine reboots). The information on when an RDMA device is *active* is NOT available to the OS. > Anyway, I agree with Christoph. > > > For normal devices the OS has a reasonable > > expectation on when the device is performing > > DMA, because it is doing so in response to a > > request that the OS knows about. For an RDMA > > device a DMA transfer can be in progress as > > as a result of an RDMA Read -- something the > > host processor knows nothing about. > > That's definitely not true. Host maps all DMA. > No one can DMA to/from the host without registering the > memory for a DMA mapping. The host does NOT know when > any given DMA mapping is active - but that's true for > FC/SCSI devices today too. > > The device driver must take reasonable measures > to quiesce that DMA mapping. Maybe send a signal > to the task that owns the DMA mapping? > Ie warn the task, wait a bit, then kill the task with > extreme prejudice if the resources it's sitting on > aren't free yet. I don't expect Hotplug support > to be simple to implement. But it's definitely possible. > Killing the task will not promptly cause RDMA activity to cease. The memory region must be deallocated or the RDMA device handle must be released. So you're back to how the kernel verbs are told of this. In any event, what you describe would only work for shutdown. It would not work to migrate pages. The same interface can be used for both purposes. From xma at us.ibm.com Mon Jun 6 12:20:34 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 6 Jun 2005 12:20:34 -0700 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: Message-ID: > Roland, Do you know other things that are missing that you can rattle off > the top of your head? A REDME file for TODO list would be helpful. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdror at mellanox.co.il Mon Jun 6 12:44:22 2005 From: gdror at mellanox.co.il (Dror Goldenberg) Date: Mon, 6 Jun 2005 22:44:22 +0300 Subject: [openib-general] Re: [openib-commits] r2551 - in gen2: branch es/xen-mlx-branch/src/linux-kernel/infiniband/core trunk/src/linux-kernel /infiniband trunk/src/linux-kernel/infiniband/core Message-ID: <506C3D7B14CDD411A52C00025558DED608001BB0@mtlex01.yok.mtl.com> > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Monday, June 06, 2005 9:49 PM > >On Mon, 2005-06-06 at 03:41 -0700, gdror at openib.org wrote: >> Modified: >> gen2/trunk/src/linux-kernel/infiniband/Kconfig >> gen2/trunk/src/linux-kernel/infiniband/Makefile >> gen2/trunk/src/linux-kernel/infiniband/core/Makefile >> Log: >> initial checkin for gverbs support. Most of files just copied >> from main trunk for now. > >Um...you really didn't mean to do this, did you? > >-tduffy I'm sorry about that. It was supposed to be on a private branch, but for some reason svn ci went wrong. I reverted those files on the main trunk. You can checkout revision 2561. -Dror -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Mon Jun 6 13:00:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 Jun 2005 16:00:17 -0400 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: References: Message-ID: <1118088017.4493.2.camel@hal.voltaire.com> On Mon, 2005-06-06 at 01:07, Shirley Ma wrote: > I was thinking to use ibv_get_devices() to get the list of device > names, then use umad_get_ca() to get ca info. (it actually gets the > info from directory /sys/class/infiniband.) Which might not what you > want. umad was done long before uverbs was available and using /sys/class/infiniband was the only way to obtain the needed information in userspace. If the community thinks uverbs should be used to obtain this information, this can be converted over. Patches accepted :-) -- Hal From itamar at mellanox.co.il Mon Jun 6 13:37:03 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Mon, 6 Jun 2005 23:37:03 +0300 Subject: [openib-general] RE: [ib_cm] oops when running kdapltest Message-ID: <91DB792C7985D411BEC300B40080D29CC35C39@mtvex01.mtv.mtl.com> Good Work (;-) I am running svn rev 2561 and i am running ./kdapltest -T T -s 11.1.4.43 -D mthca0a -d -t 4 -w 8 -i 10 client SR server SR (4 Threads 8 qps in each thread) i have run this test for about 20 times and i dont see the ib_mad and ib_cm oops's any more. i still got one ib_at oops on connect Itamar > -----Original Message----- > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > Sent: Monday, June 06, 2005 10:10 PM > To: Itamar Rabenstein > Cc: Hal Rosenstock; openib-general at openib.org > Subject: Re: [ib_cm] oops when running kdapltest > > > Itamar Rabenstein wrote: > > Hi Hal, Sean, > > > > I got this oops when running kdapltest -T T ... -t 2 -w 8 > client SR server > > SR > > this is from the client side. > > kdapl svn rev 2546 > > Can you please try re-running with 2558 and let me know if > you still see > this problem? > > Thanks, > Sean > From halr at voltaire.com Mon Jun 6 13:17:35 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 Jun 2005 16:17:35 -0400 Subject: [openib-general] RE: [ib_cm] oops when running kdapltest In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C39@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C39@mtvex01.mtv.mtl.com> Message-ID: <1118089055.4493.14.camel@hal.voltaire.com> On Mon, 2005-06-06 at 16:37, Itamar Rabenstein wrote: > i still got one ib_at oops on connect Same as before or a different one ? -- Hal From mst at mellanox.co.il Mon Jun 6 13:46:52 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 6 Jun 2005 23:46:52 +0300 Subject: [openib-general] Re: What is the correct method of determining the number of ports In-Reply-To: <1118088017.4493.2.camel@hal.voltaire.com> References: <1118088017.4493.2.camel@hal.voltaire.com> Message-ID: <20050606204652.GB30710@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: Re: What is the correct method of determining?the?number of ports > > On Mon, 2005-06-06 at 01:07, Shirley Ma wrote: > > I was thinking to use ibv_get_devices() to get the list of device > > names, then use umad_get_ca() to get ca info. (it actually gets the > > info from directory /sys/class/infiniband.) Which might not what you > > want. > > umad was done long before uverbs was available and using > /sys/class/infiniband was the only way to obtain the needed information > in userspace. If the community thinks uverbs should be used to obtain > this information, this can be converted over. Patches accepted :-) > > -- Hal Off the top of my head, the right approach seems to be to use /dev/infiniband/umad%d for umad. Right? -- MST From jlentini at netapp.com Mon Jun 6 13:45:20 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 6 Jun 2005 16:45:20 -0400 (EDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050531155812.B12767@topspin.com> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> <20050531155812.B12767@topspin.com> Message-ID: I did some research on the first topic. The unified event model was designed based on user feedback. User's of previous RDMA APIs found multiple event streams difficult to manage. They wanted the ability to have a single event stream that maintained ordering guarantees. DAPL allows users to have separate event dispatchers (EVDs) for different event types. So if you wanted to have connection events seperate from data transfer event, you can do that. Here's a little background on your second issue for those who aren't familiar with DAPL: To translate IP addresses to GIDs, the original reference implementation of DAPL looked up these mappings in the SA database. This implementation was dubbed ATS (Address Translation Service). ATS is only one possible way to implement this. Other methods could be used. On Tue, 31 May 2005, Libor Michalek wrote: > On Tue, May 31, 2005 at 02:03:06PM -0700, Tom Duffy wrote: >> On Sat, 2005-05-28 at 09:13 +0200, Christoph Hellwig wrote: >>> On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: >>>> kDAPL is intended as a kernel-level API >>>> for RDMA enabled fabrics. As it was initially written, >>>> it does not meet the Linux coding style and that is why >>>> it is being totally reworked as we speak to meet that goal. >>> >>> The codingstyle alone isn't the problem. The whole design philosophy >>> is rather odd. >> >> As one of the people trying to clean up kDAPL, I would like to know what >> you think, from a design philosophy, is wrong with it. We *can* correct >> any daim bramaged parts. > > Well, from a kernel API design philosophy the evd is somewhat odd. > The whole idea behind the event model seems a bit convoluted. First > multiplex a wide variety of events from the provider into a single event > queue, and then have an API so the consumer can tell what type of event > they actually have and can still receive the event notification in the > provider's context. > > This seems to be a lot of work to first hide useful information, but > also not loose the information in case the consumer really does want it. > It appears to be a case of a decent userspace idea that doesn't make > much sense in the kernel. Why is it there? I imagine it's to abstract a > variety of OS kernels, which was one of the goals of the design. > > Also, I realize it's just an implementation detail, but I've got a > number of issues with ATS. > > > -Libor > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From halr at voltaire.com Mon Jun 6 13:39:38 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 Jun 2005 16:39:38 -0400 Subject: [openib-general] Re: user space verbs examples In-Reply-To: <52fyw14pp0.fsf@topspin.com> References: <1117667806.4490.160.camel@hal.voltaire.com> <52fyw14pp0.fsf@topspin.com> Message-ID: <1118090378.4493.28.camel@hal.voltaire.com> On Wed, 2005-06-01 at 19:34, Roland Dreier wrote: > Hal> ibv_asyncwatch Couldn't get context for mthca0 > > Hal> dmesg shows: Jun 1 19:01:00 hal kernel: UCM: Created struct > Hal> Jun 1 19:01:00 hal kernel: UCM: Deleted struct > > Looks like it's coming from UCM, not any of these tests... That message is coming from a global constructor for user CM. > Hal> Any idea on what is wrong ? > > Nope. Are all your userspace and kernel versions in sync? Yes. > If so, then send the output of 'strace -ewrite=all -etrace=write ibv_asyncwatch' Not very informative :-( strace -etrace=write -ewrite=all ./examples/ibv_asyncwatch --- SIGCHLD (Child exited) @ 0 (0) --- --- SIGCHLD (Child exited) @ 0 (0) --- --- SIGCHLD (Child exited) @ 0 (0) --- --- SIGCHLD (Child exited) @ 0 (0) --- --- SIGCHLD (Child exited) @ 0 (0) --- write(2, "Couldn\'t get context for mthca0\n", 32Couldn't get context for mthca0 ) = 32 | 00000 43 6f 75 6c 64 6e 27 74 20 67 65 74 20 63 6f 6e Couldn't get con | | 00010 74 65 78 74 20 66 6f 72 20 6d 74 68 63 61 30 0a text for mthca0. | Same for all examples except ibv_devices. -- Hal From halr at voltaire.com Mon Jun 6 13:53:55 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 Jun 2005 16:53:55 -0400 Subject: [Fwd: [openib-general] [PATCH] [TRIVIAL] kdapl: Eliminate some dapl_evd_connection_callb.c compile warnings] Message-ID: <1118091183.4493.42.camel@hal.voltaire.com> It looks like with your latest change (r2563) this patch is no longer needed (and this file no longer exists :-) -- Hal -----Forwarded Message----- From: Hal Rosenstock To: James Lentini Cc: openib-general at openib.org Subject: [openib-general] [PATCH] [TRIVIAL] kdapl: Eliminate some dapl_evd_connection_callb.c compile warnings Date: 04 Jun 2005 19:28:47 -0400 kdapl: Eliminate some dapl_evd_connection_callb.c compile warnings Index: dapl_evd_connection_callb.c =================================================================== --- dapl_evd_connection_callb.c (revision 2546) +++ dapl_evd_connection_callb.c (working copy) @@ -31,6 +31,7 @@ #include "dapl.h" #include "dapl_evd_util.h" +#include "dapl_openib_util.h" /* * Connection callback function for ACTIVE connection requests; callbacks _______________________________________________ openib-general mailing list openib-general at openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From halr at voltaire.com Mon Jun 6 14:03:38 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 Jun 2005 17:03:38 -0400 Subject: [openib-general] Re: What is the correct method of determining the number of ports In-Reply-To: <20050606204652.GB30710@mellanox.co.il> References: <1118088017.4493.2.camel@hal.voltaire.com> <20050606204652.GB30710@mellanox.co.il> Message-ID: <1118091818.4493.55.camel@hal.voltaire.com> On Mon, 2005-06-06 at 16:46, Michael S. Tsirkin wrote: > Quoting r. Hal Rosenstock : > > Subject: Re: What is the correct method of determining?the?number of ports > > > > On Mon, 2005-06-06 at 01:07, Shirley Ma wrote: > > > I was thinking to use ibv_get_devices() to get the list of device > > > names, then use umad_get_ca() to get ca info. (it actually gets the > > > info from directory /sys/class/infiniband.) Which might not what you > > > want. > > > > umad was done long before uverbs was available and using > > /sys/class/infiniband was the only way to obtain the needed information > > in userspace. If the community thinks uverbs should be used to obtain > > this information, this can be converted over. Patches accepted :-) > > > > -- Hal > > Off the top of my head, the right approach seems to be to use > /dev/infiniband/umad%d for umad. > Right? It does that... The info it gets from /sys/class/infiniband/ pertains to a variety of driver (verbs) queries. -- Hal From itamar at mellanox.co.il Mon Jun 6 14:44:30 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 7 Jun 2005 00:44:30 +0300 Subject: [openib-general] RE: [ib_cm] oops when running kdapltest Message-ID: <91DB792C7985D411BEC300B40080D29CC35C3A@mtvex01.mtv.mtl.com> same one ! -Itamar > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Monday, June 06, 2005 11:18 PM > To: Itamar Rabenstein > Cc: Sean Hefty; openib-general at openib.org > Subject: RE: [ib_cm] oops when running kdapltest > > > On Mon, 2005-06-06 at 16:37, Itamar Rabenstein wrote: > > i still got one ib_at oops on connect > Same as before or a different one ? > > -- Hal > From tduffy at sun.com Mon Jun 6 14:34:54 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 06 Jun 2005 14:34:54 -0700 Subject: [openib-general] [PATCHv2] kDAPL: remove use of HANDLE's Message-ID: <1118093694.5735.13.camel@duffman> Updated patch now that dapl_evd is consolidated. Quit test verified. Please apply. Signed-off-by: Tom Duffy dat-provider/dapl.h | 258 ++++------- dat-provider/dapl_cr_accept.c | 106 +--- dat-provider/dapl_cr_callback.c | 147 ++---- dat-provider/dapl_cr_handoff.c | 2 dat-provider/dapl_cr_query.c | 10 dat-provider/dapl_cr_reject.c | 28 - dat-provider/dapl_cr_util.c | 21 dat-provider/dapl_ep.c | 607 +++++++++++--------------- dat-provider/dapl_evd.c | 175 +++---- dat-provider/dapl_evd.h | 6 dat-provider/dapl_get_consumer_context.c | 80 --- dat-provider/dapl_get_handle_type.c | 74 --- dat-provider/dapl_ia_close.c | 19 dat-provider/dapl_ia_memtype_hint.c | 10 dat-provider/dapl_ia_open.c | 62 +- dat-provider/dapl_ia_query.c | 20 dat-provider/dapl_ia_util.c | 180 +++---- dat-provider/dapl_lmr.c | 221 ++++----- dat-provider/dapl_mr_util.c | 2 dat-provider/dapl_openib_cm.c | 64 +- dat-provider/dapl_openib_qp.c | 10 dat-provider/dapl_openib_util.c | 12 dat-provider/dapl_openib_util.h | 14 dat-provider/dapl_provider.c | 5 dat-provider/dapl_psp.c | 105 ++-- dat-provider/dapl_pz.c | 79 +-- dat-provider/dapl_rmr.c | 78 +-- dat-provider/dapl_rsp.c | 87 +-- dat-provider/dapl_set_consumer_context.c | 74 --- dat-provider/dapl_sp_util.c | 46 - dat-provider/dapl_srq.c | 117 ++--- dat/api.c | 16 dat/dat.h | 469 ++++++++++---------- patches/alt_dat_provider_makefile | 3 test/dapltest/include/dapl_bpool.h | 10 test/dapltest/include/dapl_fft_util.h | 12 test/dapltest/include/dapl_performance_test.h | 14 test/dapltest/include/dapl_proto.h | 84 +-- test/dapltest/include/dapl_tdep.h | 12 test/dapltest/include/dapl_test_data.h | 20 test/dapltest/include/dapl_transaction_test.h | 18 test/dapltest/kdapl/kdapl_tdep_evd.c | 34 - test/dapltest/test/dapl_bpool.c | 38 - test/dapltest/test/dapl_client.c | 94 ++-- test/dapltest/test/dapl_cnxn.c | 4 test/dapltest/test/dapl_fft_connmgt.c | 4 test/dapltest/test/dapl_fft_dataxfer.c | 4 test/dapltest/test/dapl_fft_dataxfer_client.c | 12 test/dapltest/test/dapl_fft_endpoint.c | 118 ++--- test/dapltest/test/dapl_fft_hwconn.c | 22 test/dapltest/test/dapl_fft_mem.c | 42 - test/dapltest/test/dapl_fft_pz.c | 90 +-- test/dapltest/test/dapl_fft_queryinfo.c | 190 ++++---- test/dapltest/test/dapl_fft_util.c | 102 ++-- test/dapltest/test/dapl_limit.c | 204 ++++---- test/dapltest/test/dapl_performance_client.c | 24 - test/dapltest/test/dapl_performance_server.c | 68 +- test/dapltest/test/dapl_performance_util.c | 88 +-- test/dapltest/test/dapl_server.c | 102 ++-- test/dapltest/test/dapl_test_util.c | 108 ++-- test/dapltest/test/dapl_transaction_test.c | 164 +++---- test/dapltest/test/dapl_transaction_util.c | 50 +- 62 files changed, 2228 insertions(+), 2711 deletions(-) Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -33,7 +33,7 @@ boolean_t DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -61,10 +61,10 @@ DT_Performance_Test_Create ( test_ptr->remote_ia_addr = remote_ia_addr; test_ptr->is_remote_little_endian = is_remote_little_endian; test_ptr->base_port = (DAT_CONN_QUAL) pt_ptr->Server_Info.first_port_number; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; - ret = dat_ia_query (test_ptr->ia_handle, + ret = dat_ia_query (test_ptr->ia, NULL, &test_ptr->ia_attr, NULL); @@ -95,17 +95,17 @@ DT_Performance_Test_Create ( test_ptr->creq_evd_length = DT_PERF_DFLT_EVD_LENGTH; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; return FALSE; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->recv_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG, @@ -114,11 +114,11 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; return FALSE; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->reqt_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -127,7 +127,7 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: 0x%x dat_evd_create (request) error: %s\n", test_ptr->reqt_evd_length,test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; return FALSE; } @@ -135,30 +135,30 @@ DT_Performance_Test_Create ( if ( is_server ) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->creq_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CR_FLAG, &test_ptr->creq_evd_hdl); /* cr */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; return FALSE; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->conn_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CONNECTION_FLAG, &test_ptr->conn_evd_hdl); /* conn */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; return FALSE; } @@ -176,18 +176,18 @@ DT_Performance_Test_Create ( test_ptr->ep_context.ep_attr.max_request_dtos = pipeline_len; /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &test_ptr->ep_context.ep_attr, /* EP attrs */ - &test_ptr->ep_context.ep_handle); + &test_ptr->ep_context.ep); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->ep_context.ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context.ep = NULL; return FALSE; } @@ -197,10 +197,10 @@ DT_Performance_Test_Create ( */ test_ptr->ep_context.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, + NULL, /* rmr */ DT_PERF_SYNC_BUFF_SIZE, 2, /* 2 RMIs */ 255, /* FIXME should query for this */ @@ -226,7 +226,7 @@ DT_Performance_Test_Create ( * Post recv and sync buffers */ if ( !DT_post_recv_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_RECV_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE) ) @@ -269,13 +269,13 @@ DT_Performance_Test_Destroy ( boolean_t is_server) { u32 ret; - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; DT_Tdep_Print_Head *phead; int status = 0; phead = pt_ptr->Params.phead; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ if (test_ptr->ep_context.op.bp) @@ -310,9 +310,9 @@ DT_Performance_Test_Destroy ( * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context.ep_handle) + if (test_ptr->ep_context.ep) { - ret = dat_ep_disconnect (test_ptr->ep_context.ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context.ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -323,7 +323,7 @@ DT_Performance_Test_Destroy ( status = 1; } else if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -331,10 +331,10 @@ DT_Performance_Test_Destroy ( } } - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free error: %s\n", @@ -394,9 +394,9 @@ DT_Performance_Test_Destroy ( } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -417,7 +417,7 @@ DT_Performance_Test_Destroy ( boolean_t DT_performance_post_rdma_op ( Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats) { unsigned int j; @@ -456,7 +456,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -473,7 +473,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -502,17 +502,17 @@ DT_performance_post_rdma_op ( unsigned int DT_performance_reap ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Mode_Type mode, Performance_Stats_t *stats) { if ( BLOCKING_MODE == mode ) { - return DT_performance_wait (phead, evd_handle, stats); + return DT_performance_wait (phead, evd, stats); } else { - return DT_performance_poll (phead, evd_handle, stats); + return DT_performance_poll (phead, evd, stats); } } @@ -520,7 +520,7 @@ DT_performance_reap ( unsigned int DT_performance_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { int i; @@ -537,7 +537,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_wait ( evd_handle, + ret = DT_Tdep_evd_wait ( evd, DAT_TIMEOUT_MAX, &event); @@ -571,7 +571,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); @@ -611,7 +611,7 @@ DT_performance_wait ( unsigned int DT_performance_poll ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { u32 ret; @@ -626,7 +626,7 @@ DT_performance_poll ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: linux-kernel/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_client.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_performance_client.c (working copy) @@ -34,7 +34,7 @@ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr) { Performance_Test_t *test_ptr = NULL; @@ -47,7 +47,7 @@ DT_Performance_Test_Client ( DT_Tdep_PT_Debug (1,(phead,"Client: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - ia_handle, + ia, remote_ia_addr, FALSE, pt_ptr->Server_Info.is_little_endian, @@ -102,7 +102,7 @@ DT_Performance_Test_Client_Connect ( test_ptr->base_port, test_ptr->ep_context.port)); retry: - ret = dat_ep_connect (test_ptr->ep_context.ep_handle, + ret = dat_ep_connect (test_ptr->ep_context.ep, test_ptr->remote_ia_addr, test_ptr->ep_context.port, DAT_TIMEOUT_MAX, @@ -119,7 +119,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -297,7 +297,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -308,7 +308,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -375,9 +375,9 @@ DT_Performance_Test_Client_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -406,7 +406,7 @@ DT_Performance_Test_Client_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) @@ -461,7 +461,7 @@ DT_Performance_Test_Client_Exchange ( DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Sending Sync Msg\n", test_ptr->base_port)); if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -478,7 +478,7 @@ DT_Performance_Test_Client_Exchange ( if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Client_Sync_Send")) Index: linux-kernel/test/dapltest/test/dapl_cnxn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_cnxn.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_cnxn.c (working copy) @@ -29,7 +29,7 @@ /****************************************************************************/ int -get_ep_connection_state (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle) +get_ep_connection_state (DT_Tdep_Print_Head *phead, struct dat_ep *ep) { enum dat_ep_state ep_state; boolean_t in_dto_idle; @@ -39,7 +39,7 @@ get_ep_connection_state (DT_Tdep_Print_H char *req_status = "Idle"; - ret = dat_ep_get_status (ep_handle, &ep_state, &in_dto_idle, + ret = dat_ep_get_status (ep, &ep_state, &in_dto_idle, &out_dto_idle); if (ret != 0) { Index: linux-kernel/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_server.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_server.c (working copy) @@ -103,7 +103,7 @@ DT_cs_Server (Params_t * params_ptr) ret = dat_ia_open (Server_Cmd->dapl_name, DFLT_QLEN, &ps_ptr->async_evd_hdl, - &ps_ptr->ia_handle); + &ps_ptr->ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -111,28 +111,28 @@ DT_cs_Server (Params_t * params_ptr) module, Server_Cmd->dapl_name, DT_RetToString (ret)); - ps_ptr->ia_handle = DAT_HANDLE_NULL; + ps_ptr->ia = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: IA %s opened\n", module, Server_Cmd->dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ps_ptr->ia_handle, &ps_ptr->pz_handle); + ret = dat_pz_create (ps_ptr->ia, &ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->pz_handle = DAT_HANDLE_NULL; + ps_ptr->pz = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: PZ created\n", module)); /* Create 4 events - recv, request, connection-request, connect */ - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -142,11 +142,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->recv_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -156,11 +156,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->reqt_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CR_FLAG, @@ -170,11 +170,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (cr) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->creq_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -184,44 +184,44 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->conn_evd_hdl = NULL; status = 1; goto server_exit; } /* Create the EP */ - ret = dat_ep_create (ps_ptr->ia_handle, /* IA */ - ps_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (ps_ptr->ia, /* IA */ + ps_ptr->pz, /* PZ */ ps_ptr->recv_evd_hdl, /* recv */ ps_ptr->reqt_evd_hdl, /* request */ ps_ptr->conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ps_ptr->ep_handle); + &ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->ep_handle = DAT_HANDLE_NULL; + ps_ptr->ep = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: EP created\n", module)); /* Create PSP */ - ret = dat_psp_create (ps_ptr->ia_handle, + ret = dat_psp_create (ps_ptr->ia, SERVER_PORT_NUMBER, ps_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &ps_ptr->psp_handle); + &ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->psp_handle = DAT_HANDLE_NULL; + ps_ptr->psp = NULL; status = 1; goto server_exit; } @@ -233,10 +233,10 @@ DT_cs_Server (Params_t * params_ptr) */ ps_ptr->bpool = DT_BpoolAlloc (NULL, phead, - ps_ptr->ia_handle, - ps_ptr->pz_handle, - ps_ptr->ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ps_ptr->ia, + ps_ptr->pz, + ps_ptr->ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ 256, /* FIXME query for this value */ @@ -275,7 +275,7 @@ DT_cs_Server (Params_t * params_ptr) */ for (;/* EVER */;) { - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; @@ -299,7 +299,7 @@ DT_cs_Server (Params_t * params_ptr) /* Server_Info, Client_Info, Params set up below */ /* Gather whatever info we want about defaults */ - if (!DT_query (pt_ptr, ps_ptr->ia_handle, ps_ptr->ep_handle)) + if (!DT_query (pt_ptr, ps_ptr->ia, ps_ptr->ep)) { status = 1; goto server_exit; @@ -308,7 +308,7 @@ DT_cs_Server (Params_t * params_ptr) /* Post recv buffers for ClientInfo and Transaction_Cmd_t */ DT_Tdep_PT_Debug (1,(phead,"%s: Posting 2 recvs\n", module)); if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 0, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0))) @@ -320,7 +320,7 @@ DT_cs_Server (Params_t * params_ptr) goto server_exit; } if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 1, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1))) @@ -343,9 +343,9 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_cr_event_wait (phead, ps_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check (phead, &cr_stat, - ps_ptr->psp_handle, + ps_ptr->psp, SERVER_PORT_NUMBER, - &cr_handle, + &cr, module)) { @@ -357,7 +357,7 @@ DT_cs_Server (Params_t * params_ptr) } DT_Tdep_PT_Debug (1,(phead,"%s: Accepting Connection Request\n", module)); - ret = dat_cr_accept (cr_handle, ps_ptr->ep_handle, 0, (void *)0); + ret = dat_cr_accept (cr, ps_ptr->ep, 0, (void *)0); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -370,7 +370,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Debug (1,(phead,"%s: Awaiting connection ...\n", module)); if (!DT_conn_event_wait (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->conn_evd_hdl, &event_num)) { @@ -384,7 +384,7 @@ DT_cs_Server (Params_t * params_ptr) if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ps_ptr->ep_handle); + get_ep_connection_state (phead, ps_ptr->ep); } /* Wait for Client_Info */ @@ -395,7 +395,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0), dto_cookie, "Client_Info_Recv")) @@ -420,7 +420,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1), dto_cookie, "Client_Cmd_Recv")) @@ -585,7 +585,7 @@ DT_cs_Server (Params_t * params_ptr) /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -601,7 +601,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) @@ -619,7 +619,7 @@ DT_cs_Server (Params_t * params_ptr) /* we passed the pt_ptr to the thread and must now 'forget' it */ pt_ptr = NULL; - ret = dat_ep_disconnect (ps_ptr->ep_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect fails: %s\n", @@ -635,7 +635,7 @@ DT_cs_Server (Params_t * params_ptr) } /* reset the EP to get back into the game */ - dat_ep_reset (ps_ptr->ep_handle); + dat_ep_reset (ps_ptr->ep); DT_Tdep_PT_Debug (1,(phead,"%s: Waiting for another client...\n", module)); } /* end loop accepting connections */ @@ -669,9 +669,9 @@ server_exit: * we also get here on error, hence abrupt closure to * flush any lingering buffers posted. */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_disconnect (ps_ptr->ep_handle, + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -704,9 +704,9 @@ server_exit: } /* Free the PSP */ - if (ps_ptr->psp_handle) + if (ps_ptr->psp) { - ret = dat_psp_free (ps_ptr->psp_handle); + ret = dat_psp_free (ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_free error: %s\n", @@ -718,9 +718,9 @@ server_exit: } /* Free the EP */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_free (ps_ptr->ep_handle); + ret = dat_ep_free (ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free error: %s\n", @@ -781,9 +781,9 @@ server_exit: } /* Free the PZ */ - if (ps_ptr->pz_handle) + if (ps_ptr->pz) { - ret = dat_pz_free (ps_ptr->pz_handle); + ret = dat_pz_free (ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free error: %s\n", @@ -794,17 +794,17 @@ server_exit: } /* Close the IA */ - if (ps_ptr->ia_handle) + if (ps_ptr->ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) error: %s\n", module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -871,7 +871,7 @@ send_control_data ( sizeof (Server_Info_t)); DT_Server_Info_Endian ((Server_Info_t *) buffp); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -886,7 +886,7 @@ send_control_data ( if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) Index: linux-kernel/test/dapltest/test/dapl_fft_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) @@ -55,22 +55,22 @@ void DT_assert_fail (DT_Tdep_Print_Head } /* helper function to open an IA */ -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle) +int DT_ia_open (char *dev_name, struct dat_ia **ia) { - DAT_EVD_HANDLE evd_handle; - evd_handle = DAT_HANDLE_NULL; - return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, ia_handle); + struct dat_evd *evd; + evd = NULL; + return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, ia); } /* helper function to create an endpoint and its associated EVDs */ int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle) + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep) { u32 status; DT_Tdep_Print_Head *phead; @@ -80,7 +80,7 @@ int DT_ep_create (Params_t *params_ptr, *cr_evd = NULL; phead = params_ptr->phead; - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CR_FLAG, cr_evd); if (status != DAT_SUCCESS) { @@ -88,7 +88,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CONNECTION_FLAG, conn_evd); if (status != DAT_SUCCESS) { @@ -96,7 +96,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, send_evd); if (status != DAT_SUCCESS) @@ -105,7 +105,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, recv_evd); if (status != DAT_SUCCESS) { @@ -113,8 +113,8 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = dat_ep_create (ia_handle, pz_handle, *recv_evd, - *send_evd, *conn_evd, NULL, ep_handle); + status = dat_ep_create (ia, pz, *recv_evd, + *send_evd, *conn_evd, NULL, ep); if (status != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "dat_ep_create failed %s\n", DT_RetToString (status)); @@ -125,15 +125,15 @@ int DT_ep_create (Params_t *params_ptr, /* function that initializes the connection struct */ void DT_fft_init_conn_struct (FFT_Connection_t *conn) { - conn->ia_handle = NULL; - conn->pz_handle = NULL; - conn->psp_handle = NULL; - conn->ep_handle = NULL; + conn->ia = NULL; + conn->pz = NULL; + conn->psp = NULL; + conn->ep = NULL; conn->cr_evd = NULL; conn->send_evd = NULL; conn->conn_evd = NULL; conn->recv_evd = NULL; - conn->cr_handle = NULL; + conn->cr = NULL; conn->remote_netaddr = NULL; conn->bpool = NULL; conn->pt_ptr = NULL; @@ -154,27 +154,27 @@ void DT_fft_init_client (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); if (rc != DAT_SUCCESS) { /* make sure the handle has an invalid value */ - conn->ia_handle = NULL; + conn->ia = NULL; } DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* if a server name is given, allocate memory for a net address and set it @@ -198,11 +198,11 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_Print_Head *phead; phead = params_ptr->phead; - if (conn->ep_handle) + if (conn->ep) { if (conn->connected) { - rc = dat_ep_disconnect (conn->ep_handle, DAT_CLOSE_DEFAULT); + rc = dat_ep_disconnect (conn->ep, DAT_CLOSE_DEFAULT); DT_assert_clean (phead, rc == DAT_SUCCESS); if (!DT_disco_event_wait ( phead, conn->cr_evd, NULL )) @@ -211,16 +211,16 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_PT_Printf (phead, "DT_fft_destroy_conn_struct: bad disconnect event\n"); } } - rc = dat_ep_free (conn->ep_handle); + rc = dat_ep_free (conn->ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->bpool) { DT_Bpool_Destroy (NULL, phead, conn->bpool); } - if (conn->psp_handle) + if (conn->psp) { - rc = dat_psp_free (conn->psp_handle); + rc = dat_psp_free (conn->psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->cr_evd) @@ -247,14 +247,14 @@ int DT_fft_destroy_conn_struct (Params_t { DT_Free_Per_Test_Data (conn->pt_ptr); } - if (conn->pz_handle) + if (conn->pz) { - rc = dat_pz_free (conn->pz_handle); + rc = dat_pz_free (conn->pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn->ia_handle) + if (conn->ia) { - rc = dat_ia_close (conn->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (conn->ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return rc; @@ -272,31 +272,31 @@ void DT_fft_init_server (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PSP */ - rc = dat_psp_create (conn->ia_handle, SERVER_PORT_NUMBER, conn->cr_evd, - DAT_PSP_CONSUMER_FLAG, &conn->psp_handle); + rc = dat_psp_create (conn->ia, SERVER_PORT_NUMBER, conn->cr_evd, + DAT_PSP_CONSUMER_FLAG, &conn->psp); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate memory for buffers */ - conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia_handle, conn->pz_handle, + conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia, conn->pz, NULL, NULL, 8192, 2, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, conn->bpool); @@ -316,19 +316,19 @@ void DT_fft_listen (Params_t *params_ptr DT_assert_dat (phead, DT_cr_event_wait (phead, conn->cr_evd, &conn->cr_stat) && DT_cr_check (phead, &conn->cr_stat, - conn->psp_handle, + conn->psp, SERVER_PORT_NUMBER, - &conn->cr_handle, + &conn->cr, "DT_fft_listen")); /* accept the connection */ - rc =dat_cr_accept (conn->cr_handle, conn->ep_handle, 0, (void *)0); + rc =dat_cr_accept (conn->cr, conn->ep, 0, (void *)0); DT_assert_dat (phead, rc == DAT_SUCCESS); /* wait on a conn event via the conn EVD */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); conn->connected = TRUE; @@ -351,7 +351,7 @@ int DT_fft_connect (Params_t *params_ptr DT_Tdep_PT_Printf (phead, "Connection to server, attempt #%d\n", wait_count+1); /* attempt to connect, timeout = 10 secs */ - rc = dat_ep_connect (conn->ep_handle, conn->remote_netaddr, + rc = dat_ep_connect (conn->ep, conn->remote_netaddr, SERVER_PORT_NUMBER, 10*1000000, 0, (void *)0, DAT_QOS_BEST_EFFORT, DAT_CONNECT_DEFAULT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); @@ -359,7 +359,7 @@ int DT_fft_connect (Params_t *params_ptr /* wait on conn event */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); Index: linux-kernel/test/dapltest/test/dapl_fft_mem.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_mem.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_mem.c (working copy) @@ -37,28 +37,28 @@ int DT_mem_generic (Params_t *params_ptr FFT_Connection_t conn; DAT_REGION_DESCRIPTION region; u64 reg_size; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; DAT_LMR_CONTEXT lmr_context; u64 reg_addr; unsigned char *alloc_ptr; int res; u64 buffer_size; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; rc = 0; expect = 0; res = 1; - lmr_handle = NULL; + lmr = NULL; lmr_context = 0; reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; DT_fft_init_client (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); if (flag == 2) { @@ -76,24 +76,24 @@ int DT_mem_generic (Params_t *params_ptr memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - ia_handle = conn.ia_handle; + ia = conn.ia; if (flag != 3) { - pz_handle = conn.pz_handle; + pz = conn.pz; } if (flag != 4) { DT_Tdep_PT_Printf (phead, "Registering memory\n"); - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -110,21 +110,21 @@ int DT_mem_generic (Params_t *params_ptr } if (flag == 1) { - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_dat (phead, rc == DAT_SUCCESS); } - lmr_handle = NULL; + lmr = NULL; - rc = dat_lmr_kcreate (conn.ia_handle, + rc = dat_lmr_kcreate (conn.ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -133,9 +133,9 @@ int DT_mem_generic (Params_t *params_ptr } cleanup: - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (alloc_ptr) @@ -197,7 +197,7 @@ int DT_mem_case4 ( Params_t *params_ptr, DT_Tdep_Print_Head *phead; phead = params_ptr->phead; DT_Tdep_PT_Printf (phead, "\ - Description: Try to deregister memory with null lmr_handle\n"); + Description: Try to deregister memory with null lmr\n"); return DT_mem_generic (params_ptr, cmd, 4); } Index: linux-kernel/test/dapltest/test/dapl_limit.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) @@ -33,13 +33,13 @@ static boolean_t more_handles (DT_Tdep_Print_Head *phead, - DAT_HANDLE **old_ptrptr, /* pointer to current pointer */ + void ***old_ptrptr, /* pointer to current pointer */ unsigned int *old_count, /* number pointed to */ unsigned int size) /* size of one datum */ { unsigned int count = *old_count; - DAT_HANDLE *old_handles = *old_ptrptr; - DAT_HANDLE *handle_tmp = DT_Mdep_Malloc (count * 2 * size); + void * *old_handles = *old_ptrptr; + void * *handle_tmp = DT_Mdep_Malloc (count * 2 * size); if (!handle_tmp) { @@ -74,19 +74,19 @@ limit_test ( DT_Tdep_Print_Head *phead, Limit_Cmd_t *cmd, Limit_Index depth) { - DAT_EVD_HANDLE conn_handle; + struct dat_evd * conn_handle; typedef struct obj_set { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; - DAT_LMR_HANDLE lmr_handle; + struct dat_evd * evd; + struct dat_ep * ep; + struct dat_lmr * lmr; char * lmr_buffer; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; } Obj_Set; @@ -135,19 +135,19 @@ limit_test ( DT_Tdep_Print_Head *phead, for (w = 0; w < cmd->width; w++) { /* Specify that we want to get back an async EVD. */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdl_sets[w].ia_async_handle, - &hdl_sets[w].ia_handle); + &hdl_sets[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", module, cmd->device_name, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; - hdl_sets[w].ia_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; + hdl_sets[w].ia = NULL; goto clean_up_now; } } @@ -159,8 +159,8 @@ limit_test ( DT_Tdep_Print_Head *phead, */ typedef struct _ia { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; } OneOpen; unsigned int count = START_COUNT; @@ -178,7 +178,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -187,11 +187,11 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } /* Specify that we want to get back an async EVD. */ - hdlptr[w].ia_async_handle = DAT_HANDLE_NULL; + hdlptr[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdlptr[w].ia_async_handle, - &hdlptr[w].ia_handle); + &hdlptr[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", @@ -209,14 +209,14 @@ limit_test ( DT_Tdep_Print_Head *phead, for (tmp = 0; tmp < w; tmp++) { DT_Mdep_Schedule(); - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) fails: %s\n", module, DT_RetToString (ret)); retval = FALSE; - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -245,14 +245,14 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_pz_create (hdl_sets[w].ia_handle, - &hdl_sets[w].pz_handle); + ret = dat_pz_create (hdl_sets[w].ia, + &hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].pz_handle = DAT_HANDLE_NULL; + hdl_sets[w].pz = NULL; goto clean_up_now; } } @@ -263,7 +263,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PZs we can create */ unsigned int count = START_COUNT; - DAT_PZ_HANDLE *hdlptr = (DAT_PZ_HANDLE *) + struct dat_pz **hdlptr = (struct dat_pz **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PZ Exhaustion test loop */ @@ -279,7 +279,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -287,7 +287,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_pz_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_pz_create (hdl_sets[w % cmd->width].ia, &hdlptr[w]); if (ret != DAT_SUCCESS) { @@ -335,13 +335,13 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_cno_create (hdl_sets[w].ia_handle, + ret = dat_cno_create (hdl_sets[w].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdl_sets[w].cno_handle); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { DT_Tdep_PT_Printf (phead, "%s: dat_cno_create unimplemented\n", module); - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; /* ignore this error */ break; } @@ -350,7 +350,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_cno_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; goto clean_up_now; } } @@ -377,7 +377,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -385,7 +385,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_cno_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_cno_create (hdl_sets[w % cmd->width].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) @@ -446,7 +446,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -456,22 +456,22 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; goto clean_up_now; } for (w = 0; w < cmd->width; w++) { - ret = DT_Tdep_evd_create (hdl_sets[w].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w].ia, DFLT_QLEN, hdl_sets[w].cno_handle, flags, - &hdl_sets[w].evd_handle); + &hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].evd_handle = DAT_HANDLE_NULL; + hdl_sets[w].evd = NULL; goto clean_up_now; } } @@ -482,7 +482,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EVDs we can create */ unsigned int count = START_COUNT; - DAT_EVD_HANDLE *hdlptr = (DAT_EVD_HANDLE *) + struct dat_evd **hdlptr = (struct dat_evd **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); enum dat_evd_flags flags = ( DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG @@ -500,7 +500,7 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -510,13 +510,13 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->maximum; w++) { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -524,7 +524,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia, DFLT_QLEN, hdl_sets[w % cmd->width].cno_handle, flags, @@ -542,10 +542,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; /* EVD Cleanup loop */ - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (tmp = 0; tmp < w; tmp++) { @@ -578,19 +578,19 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_ep_create (hdl_sets[w].ia_handle, - hdl_sets[w].pz_handle, - hdl_sets[w].evd_handle, /* recv */ - hdl_sets[w].evd_handle, /* request */ + ret = dat_ep_create (hdl_sets[w].ia, + hdl_sets[w].pz, + hdl_sets[w].evd, /* recv */ + hdl_sets[w].evd, /* request */ conn_handle, /* connect */ (struct dat_ep_attr *) NULL, - &hdl_sets[w].ep_handle); + &hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ep_handle = DAT_HANDLE_NULL; + hdl_sets[w].ep = NULL; goto clean_up_now; } } @@ -601,7 +601,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EPs we can create */ unsigned int count = START_COUNT; - DAT_EP_HANDLE *hdlptr = (DAT_EP_HANDLE *) + struct dat_ep **hdlptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* EP Exhaustion test loop */ @@ -615,7 +615,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -623,10 +623,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, /* connect */ (struct dat_ep_attr *) NULL, &hdlptr[w]); @@ -672,9 +672,9 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many RSPs we can create */ unsigned int count = START_COUNT; - DAT_RSP_HANDLE *hdlptr = (DAT_RSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); - DAT_EP_HANDLE *epptr = (DAT_EP_HANDLE *) + struct dat_ep **epptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*epptr)); /* RSP Exhaustion test loop */ @@ -692,7 +692,7 @@ limit_test ( DT_Tdep_Print_Head *phead, unsigned int count1 = count; unsigned int count2 = count; - if (!more_handles (phead, (DAT_HANDLE **) &hdlptr, + if (!more_handles (phead, (void ***) &hdlptr, &count1, sizeof (*hdlptr))) { @@ -700,7 +700,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - if (!more_handles (phead, (DAT_HANDLE **) &epptr, + if (!more_handles (phead, (void ***) &epptr, &count2, sizeof (*epptr))) { @@ -723,10 +723,10 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * Each RSP needs a unique EP, so create one first */ - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, (struct dat_ep_attr *) NULL, &epptr[w]); @@ -738,10 +738,10 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } - ret = dat_rsp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_rsp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, epptr[w], - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { @@ -808,7 +808,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PSPs we can create */ unsigned int count = START_COUNT; - DAT_PSP_HANDLE *hdlptr = (DAT_PSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PSP Exhaustion test loop */ @@ -822,7 +822,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -830,9 +830,9 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_psp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_psp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, DAT_PSP_CONSUMER_FLAG, &hdlptr[w]); if (ret != DAT_SUCCESS) @@ -840,7 +840,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_psp_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); retval = TRUE; - hdlptr[w] = DAT_HANDLE_NULL; + hdlptr[w] = NULL; break; } } @@ -897,14 +897,14 @@ limit_test ( DT_Tdep_Print_Head *phead, memset (®ion, 0, sizeof (region)); region.for_va = hdl_sets[w].lmr_buffer; - ret = dat_lmr_kcreate (hdl_sets[w].ia_handle, + ret = dat_lmr_kcreate (hdl_sets[w].ia, DAT_MEM_TYPE_VIRTUAL, region, DFLT_BUFFSZ, - hdl_sets[w].pz_handle, + hdl_sets[w].pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &hdl_sets[w].lmr_handle, + &hdl_sets[w].lmr, &hdl_sets[w].lmr_context, NULL, /* FIXME */ ®_size, ®_addr); @@ -913,7 +913,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_lmr_kcreate #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].lmr_handle = DAT_HANDLE_NULL; + hdl_sets[w].lmr = NULL; goto clean_up_now; } if ((uintptr_t)reg_addr > (uintptr_t)hdl_sets[w].lmr_buffer @@ -949,7 +949,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -966,10 +966,10 @@ limit_test ( DT_Tdep_Print_Head *phead, */ hdlptr[w] = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].ep_handle, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].ep, + hdl_sets[w % cmd->width].evd, DFLT_BUFFSZ, 1, 256, /* FIXME should query for this */ @@ -1026,7 +1026,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, cmd->width * sizeof (*hdlptr))) { @@ -1054,7 +1054,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_ep_post_recv #%d\n", module, w * cmd->width + i + 1); - ret = dat_ep_post_recv (hdl_sets[i].ep_handle, + ret = dat_ep_post_recv (hdl_sets[i].ep, 1, iovp, cookie, @@ -1091,7 +1091,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * outstanding recv DTOs in error, and otherwise * be a no-op. */ - ret = dat_ep_reset (hdl_sets[i].ep_handle); + ret = dat_ep_reset (hdl_sets[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect (abrupt) fails: %s\n", @@ -1107,7 +1107,7 @@ limit_test ( DT_Tdep_Print_Head *phead, */ do { - ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd_handle, + ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd, &event); } while (ret == DAT_SUCCESS); } @@ -1130,10 +1130,10 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Mdep_Schedule(); test_bpool = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[0].ia_handle, - hdl_sets[0].pz_handle, - hdl_sets[0].ep_handle, - hdl_sets[0].evd_handle, + hdl_sets[0].ia, + hdl_sets[0].pz, + hdl_sets[0].ep, + hdl_sets[0].evd, test_size, 1, 256, /* FIXME should query for this */ @@ -1185,9 +1185,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].lmr_handle) + if (hdl_sets[w].lmr) { - ret = dat_lmr_free (hdl_sets[w].lmr_handle); + ret = dat_lmr_free (hdl_sets[w].lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_lmr_free fails: %s\n", @@ -1220,9 +1220,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ep_handle) + if (hdl_sets[w].ep) { - ret = dat_ep_free (hdl_sets[w].ep_handle); + ret = dat_ep_free (hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free fails: %s\n", @@ -1237,16 +1237,16 @@ clean_up_now: { unsigned int w; - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].evd_handle) + if (hdl_sets[w].evd) { - ret = DT_Tdep_evd_free (hdl_sets[w].evd_handle); + ret = DT_Tdep_evd_free (hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_free fails: %s\n", @@ -1284,9 +1284,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].pz_handle) + if (hdl_sets[w].pz) { - ret = dat_pz_free (hdl_sets[w].pz_handle); + ret = dat_pz_free (hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free fails: %s\n", @@ -1303,10 +1303,10 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ia_handle) + if (hdl_sets[w].ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { @@ -1318,7 +1318,7 @@ clean_up_now: * we may as well try the largest hammer we have. */ retval = FALSE; - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { Index: linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -37,31 +37,31 @@ int DT_queryinfo_basic (Params_t *params u32 result_wanted) { char *dev_name; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; struct dat_ia_attr ia_attributes; struct dat_provider_attr provider_attributes; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE cr_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *cr_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; + struct dat_ep *ep; struct dat_ep_param ep_param; DAT_CNO_HANDLE cno_handle; #ifndef __KDAPLTEST__ DAT_CNO_PARAM cno_param; #endif struct dat_evd_param evd_param; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * psp; struct dat_psp_param psp_param; - DAT_RSP_HANDLE rsp_handle; + struct dat_sp * rsp; struct dat_rsp_param rsp_param; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; struct dat_pz_param pz_param; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; struct dat_lmr_param lmr_param; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; struct dat_rmr_param rmr_param; DAT_REGION_DESCRIPTION region; u64 reg_size; @@ -77,30 +77,30 @@ int DT_queryinfo_basic (Params_t *params reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - lmr_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + lmr = NULL; rmr_handle = NULL; - pz_handle = NULL; - psp_handle = NULL; - rsp_handle = NULL; + pz = NULL; + psp = NULL; + rsp = NULL; cno_handle = NULL; - evd_handle = DAT_HANDLE_NULL; - conn_evd_handle = DAT_HANDLE_NULL; - cr_evd_handle = DAT_HANDLE_NULL; - recv_evd_handle = DAT_HANDLE_NULL; - send_evd_handle = DAT_HANDLE_NULL; + evd = NULL; + conn_evd = NULL; + cr_evd = NULL; + recv_evd = NULL; + send_evd = NULL; dev_name = cmd->device_name; - /* All functions require an ia_handle to be created */ + /* All functions require an ia to be created */ rc = dat_ia_open ((const char *)dev_name, DEFAULT_QUEUE_LEN, - &evd_handle, - &ia_handle); + &evd, + &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - /* These functions require a pz_handle to be created */ + /* These functions require a pz to be created */ if ( (object_to_query == QUERY_EVD) || (object_to_query == QUERY_RMR) || (object_to_query == QUERY_LMR) || @@ -108,43 +108,43 @@ int DT_queryinfo_basic (Params_t *params (object_to_query == QUERY_RSP) || (object_to_query == QUERY_PZ) ) { - rc = dat_pz_create (ia_handle, - &pz_handle); + rc = dat_pz_create (ia, + &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - /* These functions require a ep_handle to be created */ + /* These functions require a ep to be created */ if ( (object_to_query == QUERY_EP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CONNECTION_FLAG, - &conn_evd_handle); + &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, - pz_handle, - recv_evd_handle, - send_evd_handle, - conn_evd_handle, + rc = dat_ep_create (ia, + pz, + recv_evd, + send_evd, + conn_evd, NULL, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -152,11 +152,11 @@ int DT_queryinfo_basic (Params_t *params if ( (object_to_query == QUERY_PSP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CR_FLAG, - &cr_evd_handle); + &cr_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -165,8 +165,8 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -177,15 +177,15 @@ int DT_queryinfo_basic (Params_t *params * NULL out ia_attr and for the DAT_IA_ATTR_MASK to * have values */ - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, NULL, &provider_attributes); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_ia_query (evd_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -197,11 +197,11 @@ int DT_queryinfo_basic (Params_t *params #ifndef __KDAPLTEST__ #if defined(WIN32) - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, NULLPROXY, &cno_handle); #else - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &cno_handle); #endif @@ -220,7 +220,7 @@ int DT_queryinfo_basic (Params_t *params } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_cno_query (ia_handle, + rc = dat_cno_query (ia, &cno_param); } #endif @@ -230,17 +230,17 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, &evd_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_evd_query (ia_handle, + rc = dat_evd_query (evd, &evd_param); } } @@ -248,25 +248,25 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_psp_query function */ else if (object_to_query == QUERY_PSP) { - rc = dat_psp_create (ia_handle, + rc = dat_psp_create (ia, SERVER_PORT_NUMBER, - cr_evd_handle, + cr_evd, DAT_PSP_PROVIDER_FLAG, - &psp_handle); + &psp); DT_assert_dat (phead, rc == DAT_SUCCESS); if (result_wanted == DAT_SUCCESS) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, &psp_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_psp_query (evd_handle, + rc = dat_psp_query (psp, &psp_param); } } @@ -274,13 +274,13 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_rsp_query function */ else if (object_to_query == QUERY_RSP) { - rc = dat_rsp_create (ia_handle, + rc = dat_rsp_create (ia, SERVER_PORT_NUMBER, - ep_handle, - cr_evd_handle, - &rsp_handle); + ep, + cr_evd, + &rsp); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_rsp_query (rsp_handle, + rc = dat_rsp_query (rsp, &rsp_param); } @@ -294,14 +294,14 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_ep_query function */ else if (object_to_query == QUERY_EP) { - rc = dat_ep_query (ep_handle, + rc = dat_ep_query (ep, &ep_param); } /* Test dat_pz_query function */ else if (object_to_query == QUERY_PZ) { - rc = dat_pz_query (pz_handle, + rc = dat_pz_query (pz, &pz_param); } @@ -312,27 +312,27 @@ int DT_queryinfo_basic (Params_t *params DT_assert (phead, alloc_ptr); memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, ®_addr); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_lmr_query (lmr_handle, + rc = dat_lmr_query (lmr, &lmr_param); } /* Test dat_rmr_query function */ else if (object_to_query == QUERY_RMR) { - rc = dat_rmr_create (pz_handle, + rc = dat_rmr_create (pz, &rmr_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_rmr_query (rmr_handle, @@ -342,39 +342,39 @@ int DT_queryinfo_basic (Params_t *params DT_assert_dat (phead, DAT_GET_TYPE (rc) == result_wanted); cleanup: - if (rsp_handle) + if (rsp) { - rc = dat_rsp_free (rsp_handle); + rc = dat_rsp_free (rsp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -391,27 +391,27 @@ cleanup: DT_assert_clean (phead, rc == DAT_SUCCESS); } #endif - if (psp_handle) + if (psp) { - rc = dat_psp_free (psp_handle); + rc = dat_psp_free (psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (cr_evd_handle) + if (cr_evd) { - rc = DT_Tdep_evd_free (cr_evd_handle); + rc = DT_Tdep_evd_free (cr_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -47,7 +47,7 @@ /****************************************************************************/ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr) { Transaction_Cmd_t *cmd = &pt_ptr->Params.u.Transaction_Cmd; @@ -68,7 +68,7 @@ DT_Transaction_Test_Client (Per_Test_Dat DT_Tdep_PT_Debug (1,(phead,"Client: Starting Client side of test\n")); if (!DT_Transaction_Create_Test (pt_ptr, - ia_handle, + ia, FALSE, port_num, pt_ptr->Server_Info.is_little_endian, @@ -110,7 +110,7 @@ DT_Transaction_Test_Server (void *params + i * cmd->eps_per_thread; if (!DT_Transaction_Create_Test (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, TRUE, port_num, pt_ptr->Client_Info.is_little_endian, @@ -156,7 +156,7 @@ DT_Transaction_Test_Server (void *params boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -181,7 +181,7 @@ DT_Transaction_Create_Test (Per_Test_Dat test_ptr->remote_is_little_endian = remote_is_little_endian; test_ptr->is_server = is_server; test_ptr->pt_ptr = pt_ptr; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->base_port = (DAT_CONN_QUAL) port_num; test_ptr->cmd = &pt_ptr->Params.u.Transaction_Cmd; test_ptr->time_out = DFLT_TMO * 1000; /* DFLT_TMO seconds */ @@ -244,18 +244,18 @@ DT_Transaction_Main (void *param) private_data_str = "DAPL and RDMA rule! Test 4321."; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; status = 1; goto test_failure; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG, @@ -264,12 +264,12 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; status = 1; goto test_failure; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -278,7 +278,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (request) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; status = 1; goto test_failure; } @@ -286,7 +286,7 @@ DT_Transaction_Main (void *param) if (pt_ptr->local_is_server) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CR_FLAG, @@ -295,13 +295,13 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; status = 1; goto test_failure; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CONNECTION_FLAG, @@ -310,7 +310,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; status = 1; goto test_failure; } @@ -363,18 +363,18 @@ DT_Transaction_Main (void *param) } /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &ep_attr, /* EP attrs */ - &test_ptr->ep_context[i].ep_handle); + &test_ptr->ep_context[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - test_ptr->ep_context[i].ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context[i].ep = NULL; status = 1; goto test_failure; } @@ -385,10 +385,10 @@ DT_Transaction_Main (void *param) */ test_ptr->ep_context[i].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ buff_size, 4, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -423,7 +423,7 @@ DT_Transaction_Main (void *param) * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -433,7 +433,7 @@ DT_Transaction_Main (void *param) goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -457,11 +457,11 @@ DT_Transaction_Main (void *param) * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->creq_evd_hdl, - &test_ptr->ep_context[i].rsp_handle); + &test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create #%d error: %s\n", @@ -472,11 +472,11 @@ DT_Transaction_Main (void *param) } else { - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &test_ptr->ep_context[i].psp_handle); + &test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create #%d error: %s\n", @@ -519,7 +519,7 @@ DT_Transaction_Main (void *param) RemoteMemoryInfo *RemoteMemInfo; struct dat_dto_completion_event_data dto_stat; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; /* * Establish the connection @@ -538,16 +538,16 @@ DT_Transaction_Main (void *param) &cr_stat) || !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].rsp_handle, + test_ptr->ep_context[i].rsp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error:(%x) %s\n", @@ -571,21 +571,21 @@ DT_Transaction_Main (void *param) } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, + ret = dat_cr_accept (cr, NULL, /* NULL for RSP */ 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -594,7 +594,7 @@ DT_Transaction_Main (void *param) goto test_failure; } /* throw away single-use PSP */ - ret = dat_rsp_free (test_ptr->ep_context[i].rsp_handle); + ret = dat_rsp_free (test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free #%d error: %s\n", @@ -622,16 +622,16 @@ DT_Transaction_Main (void *param) if ( !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].psp_handle, + test_ptr->ep_context[i].psp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error: %s\n", @@ -656,33 +656,33 @@ DT_Transaction_Main (void *param) /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context[i].ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context[i].ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ - (void) dat_psp_free (test_ptr->ep_context[i].psp_handle); + /* cr consumed on failure */ + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { /* error message printed by DT_cr_event_wait */ - (void) dat_psp_free (&test_ptr->ep_context[i].psp_handle); + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* throw away single-use PSP */ - ret = dat_psp_free (test_ptr->ep_context[i].psp_handle); + ret = dat_psp_free (test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free #%d error: %s\n", @@ -709,7 +709,7 @@ DT_Transaction_Main (void *param) test_ptr->ep_context[i].ia_port)); retry: - ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_connect (test_ptr->ep_context[i].ep, test_ptr->remote_ia_addr, test_ptr->ep_context[i].ia_port, DAT_TIMEOUT_MAX, @@ -726,7 +726,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -742,7 +742,7 @@ retry: struct dat_event event; int drained = 0; - dat_ep_reset (test_ptr->ep_context[i].ep_handle); + dat_ep_reset (test_ptr->ep_context[i].ep); do { ret = DT_Tdep_evd_dequeue ( test_ptr->recv_evd_hdl, @@ -756,7 +756,7 @@ retry: * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -766,7 +766,7 @@ retry: goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -835,9 +835,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -871,9 +871,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -906,10 +906,10 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -976,7 +976,7 @@ retry: /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_SEND_BUFFER_ID, buff_size)) @@ -993,7 +993,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Send" @@ -1017,7 +1017,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Recv" @@ -1082,9 +1082,9 @@ test_failure: /* Foreach EP */ for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ for (j = 0; j < test_ptr->cmd->num_ops; j++) @@ -1122,9 +1122,9 @@ test_failure: * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context[i].ep_handle) + if (test_ptr->ep_context[i].ep) { - ret = dat_ep_disconnect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context[i].ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -1148,7 +1148,7 @@ test_failure: { if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -1164,16 +1164,16 @@ test_failure: */ for (j = 0; j < test_ptr->cmd->eps_per_thread; j++) { - if ( test_ptr->ep_context[j].ep_handle == ep_handle ) + if ( test_ptr->ep_context[j].ep == ep ) { - test_ptr->ep_context[j].ep_handle = NULL; + test_ptr->ep_context[j].ep = NULL; } } } } else /* !success - QP may be in error state */ { - ep_handle = test_ptr->ep_context[i].ep_handle; + ep = test_ptr->ep_context[i].ep; } /* @@ -1182,7 +1182,7 @@ test_failure: * disconnected as we are racing with the remote side * disconnects. */ - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { struct dat_event event; /* @@ -1195,7 +1195,7 @@ test_failure: &event); } while (ret == DAT_SUCCESS); /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free #%d error: %s\n", @@ -1259,9 +1259,9 @@ test_failure: } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -1357,7 +1357,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1432,7 +1432,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1923,8 +1923,8 @@ DT_Print_Transaction_Test (DT_Tdep_Print test_ptr->remote_is_little_endian); DT_Tdep_PT_Printf (phead, "TransTest.base_port : " F64x "\n", test_ptr->base_port); - DT_Tdep_PT_Printf (phead, "TransTest.pz_handle : %p\n", - test_ptr->pz_handle); + DT_Tdep_PT_Printf (phead, "TransTest.pz : %p\n", + test_ptr->pz); /* statistics */ DT_Tdep_PT_Printf (phead, "TransTest.bytes_send : %d\n", test_ptr->stats.stat_bytes_send); Index: linux-kernel/test/dapltest/test/dapl_performance_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_server.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_performance_server.c (working copy) @@ -41,7 +41,7 @@ DT_Performance_Test_Server ( DT_Tdep_PT_Debug (1,(phead,"Server: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, (struct sockaddr *) 0, TRUE, pt_ptr->Client_Info.is_little_endian, @@ -97,15 +97,15 @@ DT_Performance_Test_Server_Connect ( { u32 ret; boolean_t status; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; enum dat_event_number event_num; - rsp_handle = DAT_HANDLE_NULL; - psp_handle = DAT_HANDLE_NULL; + rsp = NULL; + psp = NULL; #if 0 /* FIXME */ if (test_ptr->cmd->use_rsp) { @@ -113,11 +113,11 @@ DT_Performance_Test_Server_Connect ( * Server - create a single-use RSP and * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context.port, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->creq_evd_hdl, - &rsp_handle); + &rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create error: %s\n", @@ -132,9 +132,9 @@ DT_Performance_Test_Server_Connect ( /* wait for the connection request */ if (!DT_cr_event_wait (test_ptr->conn_evd_hdl, &cr_stat) || !DT_cr_check ( &cr_stat, - DAT_HANDLE_NULL, + NULL, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -142,20 +142,20 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ - if (!DT_conn_event_wait ( test_ptr->ep_context.ep_handle, + if (!DT_conn_event_wait ( test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -173,17 +173,17 @@ DT_Performance_Test_Server_Connect ( */ status = TRUE; - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context.port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &psp_handle); + &psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); status = FALSE; - psp_handle = DAT_HANDLE_NULL; + psp = NULL; return (status); } @@ -204,9 +204,9 @@ DT_Performance_Test_Server_Connect ( if (!DT_cr_event_wait (phead, test_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check ( phead, &cr_stat, - psp_handle, + psp, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -214,22 +214,22 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num ) ) { @@ -241,10 +241,10 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_PT_Debug (1, (phead,"Server[" F64x "]: Accept on port 0x" F64x "\n", test_ptr->base_port, test_ptr->ep_context.port)); psp_free: - if ( DAT_HANDLE_NULL != psp_handle ) + if ( NULL != psp ) { /* throw away single-use PSP */ - ret = dat_psp_free (psp_handle); + ret = dat_psp_free (psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free error: %s\n", @@ -252,10 +252,10 @@ psp_free: status = FALSE; } } - if ( DAT_HANDLE_NULL != rsp_handle ) + if ( NULL != rsp ) { /* throw away single-use PSP */ - ret = dat_rsp_free (rsp_handle); + ret = dat_rsp_free (rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free error: %s\n", @@ -282,9 +282,9 @@ DT_Performance_Test_Server_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -340,7 +340,7 @@ DT_Performance_Test_Server_Exchange ( /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -358,7 +358,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Send Sync_Msg") ) @@ -380,7 +380,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (working copy) @@ -40,7 +40,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ u32 rc=0; DT_fft_init_client (phead, cmd, &conn); - DT_assert_dat (phead, conn.ia_handle != NULL) + DT_assert_dat (phead, conn.ia != NULL) DT_assert (phead, DT_fft_connect (phead, &conn)); @@ -58,8 +58,8 @@ int DT_dataxfer_client_generic (DT_Tdep_ { conn.bpool = DT_BpoolAlloc (0, phead, - conn.ia_handle, - conn.pz_handle, + conn.ia, + conn.pz, NULL, NULL, 4096, @@ -68,7 +68,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ FALSE, FALSE); DT_assert (phead, conn.bpool != 0); - rc = DT_post_send_buffer (phead, conn.ep_handle, conn.bpool, 0, + rc = DT_post_send_buffer (phead, conn.ep, conn.bpool, 0, DT_Bpool_GetBuffSize (conn.bpool, 0)); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_evd_wait (conn.send_evd, 10*1000000, 1, &conn.event, @@ -80,11 +80,11 @@ int DT_dataxfer_client_generic (DT_Tdep_ /* cleanup */ cleanup: - if (conn.ep_handle) + if (conn.ep) { /* disconnect */ DT_Tdep_PT_Printf (phead, "Disconnect\n"); - rc = dat_ep_disconnect (conn.ep_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ep_disconnect (conn.ep, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } rc = DT_fft_destroy_conn_struct (phead, &conn); Index: linux-kernel/test/dapltest/test/dapl_fft_connmgt.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (working copy) @@ -39,7 +39,7 @@ int DT_connmgt_case0 (Params_t *params_p Description: Ensure time in dat_evd_wait works correctly\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 10000, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); @@ -62,7 +62,7 @@ int DT_connmgt_case1 (Params_t *params_p Description: Attempt to use timeout of 0 in dat_evd_wait\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 0, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -86,10 +86,10 @@ Bpool * DT_BpoolAlloc ( Per_Test_Data_t *pt_ptr, DT_Tdep_Print_Head *phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia * ia, + struct dat_pz * pz, + struct dat_ep * ep, + struct dat_evd * rmr_evd, int seg_size, int num_segs, int alignment, @@ -131,16 +131,16 @@ DT_BpoolAlloc ( bpool_ptr->alloc_ptr = alloc_ptr; bpool_ptr->alloc_size = alloc_size; - bpool_ptr->pz_handle = pz_handle; + bpool_ptr->pz = pz; bpool_ptr->num_segs = num_segs; - bpool_ptr->ep_handle = ep_handle; + bpool_ptr->ep = ep; bpool_ptr->buffer_size = seg_size * num_segs; bpool_ptr->buffer_start = DT_AlignPtr (alloc_ptr, alignment) + GG_ALLOC_OFFSET; bpool_ptr->tripl_start = (struct dat_lmr_triplet *) (bpool_ptr + 1); bpool_ptr->seg_size = seg_size; bpool_ptr->enable_rdma_write = enable_rdma_write; bpool_ptr->enable_rdma_read = enable_rdma_read; - bpool_ptr->rmr_evd_handle = rmr_evd_handle; + bpool_ptr->rmr_evd = rmr_evd; DT_Tdep_PT_Debug (3, (phead, "lmr_create [%p, " F64x "]\n", @@ -160,14 +160,14 @@ DT_BpoolAlloc ( region.for_pa = virt_to_phys(region.for_va); } - ret = dat_lmr_kcreate (ia_handle, + ret = dat_lmr_kcreate (ia, DT_mem_type, region, bp_len, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &bpool_ptr->lmr_handle, + &bpool_ptr->lmr, &bpool_ptr->lmr_context, &bpool_ptr->rmr_context, &bpool_ptr->reg_size, @@ -213,7 +213,7 @@ DT_BpoolAlloc ( struct dat_rmr_bind_completion_event_data rmr_stat; /* create the RMR */ - ret = dat_rmr_create (pz_handle, &bpool_ptr->rmr_handle); + ret = dat_rmr_create (pz, &bpool_ptr->rmr_handle); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_rmr_create failed %s\n", @@ -238,7 +238,7 @@ DT_BpoolAlloc ( ret = dat_rmr_bind ( bpool_ptr->rmr_handle, &iov, mflags, - bpool_ptr->ep_handle, + bpool_ptr->ep, cookie, DAT_COMPLETION_DEFAULT_FLAG, &bpool_ptr->rmr_context); @@ -253,7 +253,7 @@ DT_BpoolAlloc ( /* await the bind result */ if (!DT_rmr_event_wait (phead, - bpool_ptr->rmr_evd_handle, + bpool_ptr->rmr_evd, &rmr_stat) || !DT_rmr_check (phead, &rmr_stat, @@ -290,9 +290,9 @@ err: DT_RetToString (ret)); } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - ret = dat_lmr_free (bpool_ptr->lmr_handle); + ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -357,9 +357,9 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - u32 ret = dat_lmr_free (bpool_ptr->lmr_handle); + u32 ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -433,8 +433,8 @@ DT_Bpool_print (DT_Tdep_Print_Head *phea "BPOOL alloc_size %x\n", (int) bpool_ptr->alloc_size); DT_Tdep_PT_Printf (phead, - "BPOOL pz_handle %p\n", - bpool_ptr->pz_handle); + "BPOOL pz %p\n", + bpool_ptr->pz); DT_Tdep_PT_Printf (phead, "BPOOL num_segs %x\n", (int) bpool_ptr->num_segs); Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -32,11 +32,11 @@ */ boolean_t DT_query ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle) + struct dat_ia * ia, + struct dat_ep * ep) { unsigned char *module = "DT_query"; - DAT_EVD_HANDLE async_evd_hdl; /* not used */ + struct dat_evd * async_evd_hdl; /* not used */ struct dat_ep_param ep_params; u32 ret; DT_Tdep_Print_Head *phead; @@ -44,7 +44,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, phead = pt_ptr->Params.phead; /* Query the IA */ - ret = dat_ia_query (ia_handle, + ret = dat_ia_query (ia, &async_evd_hdl, &pt_ptr->ia_attr, &pt_ptr->provider_attr); @@ -57,7 +57,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, } /* Query the EP */ - ret = dat_ep_query (ep_handle, &ep_params); + ret = dat_ep_query (ep, &ep_params); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_query error: %s\n", @@ -160,7 +160,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, */ boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -184,7 +184,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Recv #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_recv (ep_handle, + ret = dat_ep_post_recv (ep, 1, iov, cookie, @@ -205,7 +205,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head */ boolean_t DT_post_send_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -228,7 +228,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Send #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_send (ep_handle, + ret = dat_ep_post_send (ep, 1, iov, cookie, @@ -249,7 +249,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head */ boolean_t DT_cr_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p) { int err_cnt; @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -306,8 +306,8 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p */ boolean_t DT_conn_event_wait (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep * ep, + struct dat_evd *evd, enum dat_event_number *event_number) { @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -341,7 +341,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * { /* * Could return struct dat_connection_event_data and verify: - * event.event_data.connect_event_data.ep_handle + * event.event_data.connect_event_data.ep * event.event_data.connect_event_data.private_data_size * event.event_data.connect_event_data.private_data */ @@ -361,15 +361,15 @@ DT_conn_event_wait (DT_Tdep_Print_Head * */ boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ) + struct dat_evd *evd, + struct dat_ep * *ep ) { for (;;) { u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -392,9 +392,9 @@ DT_disco_event_wait ( DT_Tdep_Print_Head if (event.event_number == DAT_CONNECTION_EVENT_DISCONNECTED) { - if ( ep_handle != NULL ) + if ( ep != NULL ) { - *ep_handle = event.event_data.connect_event_data.ep_handle; + *ep = event.event_data.connect_event_data.ep; } return (TRUE); } @@ -412,17 +412,17 @@ DT_disco_event_wait ( DT_Tdep_Print_Head */ boolean_t DT_dto_event_reap (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dto_statusp) { if (poll) { - return DT_dto_event_poll (phead, evd_handle, dto_statusp); + return DT_dto_event_poll (phead, evd, dto_statusp); } else { - return DT_dto_event_wait (phead, evd_handle, dto_statusp); + return DT_dto_event_wait (phead, evd, dto_statusp); } } @@ -432,7 +432,7 @@ DT_dto_event_reap (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_poll (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -440,7 +440,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); if (DAT_GET_TYPE (ret) == DAT_QUEUE_EMPTY) @@ -460,7 +460,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -484,7 +484,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -505,7 +505,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -528,7 +528,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p */ boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_statusp) { for (;;) @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", @@ -572,12 +572,12 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p boolean_t DT_dto_check ( DT_Tdep_Print_Head *phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message ) { - if ( ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) || dto_p->transfered_length != len_expected || dto_p->user_cookie.as_64 != cookie_expected.as_64 || dto_p->status != DAT_DTO_SUCCESS ) @@ -588,10 +588,10 @@ DT_dto_check ( DT_Tdep_Print_Head *phead : (dto_p->status == DAT_DTO_ERR_FLUSHED ? "FAILURE" : "LengthError"))); DT_Test_Error (); - if ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) { DT_Tdep_PT_Printf (phead, "\tEndPoint mismatch (got %p wanted %p)\n", - dto_p->ep_handle, + dto_p->ep, ep_expected); } if (dto_p->transfered_length != len_expected) @@ -620,11 +620,11 @@ DT_dto_check ( DT_Tdep_Print_Head *phead boolean_t DT_rmr_check ( DT_Tdep_Print_Head *phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message) { - if ( rmr_p->rmr_handle != rmr_expected + if ( rmr_p->rmr != rmr_expected || rmr_p->user_cookie.as_ptr != cookie_expected || rmr_p->status != DAT_RMR_BIND_SUCCESS ) { @@ -634,10 +634,10 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead (rmr_p->status == DAT_RMR_BIND_SUCCESS ? "OK" : "FAILURE")); DT_Test_Error (); - if (rmr_p->rmr_handle != rmr_expected) + if (rmr_p->rmr != rmr_expected) { DT_Tdep_PT_Printf (phead, "\tRMR handle mismatch (got 0x%p wanted 0x%p)\n", - rmr_p->rmr_handle, + rmr_p->rmr, rmr_expected); } if (rmr_p->user_cookie.as_ptr != cookie_expected) @@ -659,21 +659,21 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead boolean_t DT_cr_check ( DT_Tdep_Print_Head *phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp * psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr * *crp, char *message) { u32 ret; - if (cr_handlep) + if (crp) { - *cr_handlep = (DAT_CR_HANDLE) 0; + *crp = (struct dat_cr *) 0; } if (cr_stat_p->conn_qual != port_expected || - (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected)) + (psp_expected && + cr_stat_p->sp != psp_expected)) { DT_Tdep_PT_Printf (phead, "Test Error: %s CR data problem\n", message); @@ -684,20 +684,20 @@ DT_cr_check ( DT_Tdep_Print_Head *phead " (got 0x" F64x " wanted 0x" F64x ")\n", cr_stat_p->conn_qual, port_expected); } - if (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected) + if (psp_expected && + cr_stat_p->sp != psp_expected) { DT_Tdep_PT_Printf (phead, "\tPSP mismatch (got 0x%p wanted 0x%p)\n", - cr_stat_p->sp_handle.psp_handle, - psp_handle_expected); + cr_stat_p->sp, + psp_expected); } - if (!cr_stat_p->cr_handle) + if (!cr_stat_p->cr) { - DT_Tdep_PT_Printf (phead, "\tGot NULL cr_handle\n"); + DT_Tdep_PT_Printf (phead, "\tGot NULL cr\n"); } else { - ret = dat_cr_reject (cr_stat_p->cr_handle); + ret = dat_cr_reject (cr_stat_p->cr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "\tdat_cr_reject error: %s\n", @@ -707,9 +707,9 @@ DT_cr_check ( DT_Tdep_Print_Head *phead return ( FALSE ); } - if (cr_handlep) + if (crp) { - *cr_handlep = cr_stat_p->cr_handle; + *crp = cr_stat_p->cr; } return ( TRUE ); } Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -41,13 +41,13 @@ DT_cs_Client (Params_t * params_ptr, u32 total_threads) { Per_Test_Data_t *pt_ptr = NULL; - DAT_IA_HANDLE ia_handle = DAT_HANDLE_NULL; - DAT_PZ_HANDLE pz_handle = DAT_HANDLE_NULL; - DAT_EVD_HANDLE recv_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE reqt_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE conn_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE async_evd_hdl = DAT_HANDLE_NULL; - DAT_EP_HANDLE ep_handle = DAT_HANDLE_NULL; + struct dat_ia *ia = NULL; + struct dat_pz *pz = NULL; + struct dat_evd *recv_evd_hdl = NULL; + struct dat_evd *reqt_evd_hdl = NULL; + struct dat_evd *conn_evd_hdl = NULL; + struct dat_evd *async_evd_hdl = NULL; + struct dat_ep *ep = NULL; Server_Info_t *sinfo = NULL; Transaction_Cmd_t *Transaction_Cmd = NULL; Quit_Cmd_t *Quit_Cmd = NULL; @@ -98,7 +98,7 @@ DT_cs_Client (Params_t * params_ptr, ret = dat_ia_open (dapl_name, DFLT_QLEN, &async_evd_hdl, - &ia_handle); + &ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -106,28 +106,28 @@ DT_cs_Client (Params_t * params_ptr, module, dapl_name, DT_RetToString (ret)); - ia_handle = DAT_HANDLE_NULL; + ia = NULL; status = 1; goto client_exit; } DT_Tdep_PT_Debug (1,(phead, "%s: IA %s opened\n", module, dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ia_handle, &pz_handle); + ret = dat_pz_create (ia, &pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - pz_handle = DAT_HANDLE_NULL; + pz = NULL; status = 1; goto client_exit; } /* Create 3 events - recv, request, connect */ - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -138,11 +138,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - recv_evd_hdl = DAT_HANDLE_NULL; + recv_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -153,11 +153,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - reqt_evd_hdl = DAT_HANDLE_NULL; + reqt_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -168,26 +168,26 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - conn_evd_hdl = DAT_HANDLE_NULL; + conn_evd_hdl = NULL; status = 1; goto client_exit; } /* Create an EP */ - ret = dat_ep_create (ia_handle, /* IA */ - pz_handle, /* PZ */ + ret = dat_ep_create (ia, /* IA */ + pz, /* PZ */ recv_evd_hdl, /* recv */ reqt_evd_hdl, /* request */ conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ep_handle); + &ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ep_handle = DAT_HANDLE_NULL; + ep = NULL; status = 1; goto client_exit; } @@ -197,7 +197,7 @@ DT_cs_Client (Params_t * params_ptr, * Gather whatever info we want about defaults, * and check that we can handle the requested parameters. */ - if (!DT_query (pt_ptr, ia_handle, ep_handle) || + if (!DT_query (pt_ptr, ia, ep) || !DT_check_params (pt_ptr, module)) { status = 1; @@ -206,10 +206,10 @@ DT_cs_Client (Params_t * params_ptr, bpool = DT_BpoolAlloc (pt_ptr, phead, - ia_handle, - pz_handle, - ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ia, + pz, + ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ pt_ptr->provider_attr.optimal_buffer_alignment, @@ -238,7 +238,7 @@ DT_cs_Client (Params_t * params_ptr, DT_Tdep_PT_Debug (1,(phead, "%s: Posting 1 recv buffer\n", module)); retry_repost: if (!DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, DT_Bpool_GetBuffSize (bpool, 0))) @@ -253,7 +253,7 @@ retry_repost: DT_Tdep_PT_Debug (1,(phead, "%s: Connect Endpoint\n", module)); try_connect =1; retry: - ret = dat_ep_connect (ep_handle, + ret = dat_ep_connect (ep, server_netaddr, SERVER_PORT_NUMBER, DAT_TIMEOUT_MAX, @@ -271,7 +271,7 @@ retry: } DT_Tdep_PT_Debug (1,(phead, "%s: Await connection ...\n", module)); - if (!DT_conn_event_wait (phead, ep_handle, conn_evd_hdl, &event_num)) + if (!DT_conn_event_wait (phead, ep, conn_evd_hdl, &event_num)) { if ( event_num == DAT_CONNECTION_EVENT_PEER_REJECTED ) { @@ -287,7 +287,7 @@ retry: * See if any buffers were flushed as a result of * the REJECT; clean them up and repost if so */ - dat_ep_reset (ep_handle); + dat_ep_reset (ep); do { @@ -315,7 +315,7 @@ retry: if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ep_handle); + get_ep_connection_state (phead, ep); } /* Send Client_Info (using 2nd buffer in the pool) */ @@ -326,7 +326,7 @@ retry: sizeof (Client_Info_t)); DT_Client_Info_Endian ((Client_Info_t *) buffp); if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 1, DT_Bpool_GetBuffSize (bpool, 1))) @@ -343,7 +343,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 1), dto_cookie, "Client_Info_Send")) @@ -396,7 +396,7 @@ retry: /* Send the Command buffer */ if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 2, DT_Bpool_GetBuffSize (bpool, 2))) @@ -414,7 +414,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 2), dto_cookie, "Client_Cmd_Send")) @@ -430,7 +430,7 @@ retry: if (!DT_dto_event_wait (phead, recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 0), dto_cookie, "Server_Info_Recv")) @@ -476,7 +476,7 @@ retry: DT_Transaction_Cmd_PT_Print (phead, Transaction_Cmd); } status = DT_Transaction_Test_Client (pt_ptr, - ia_handle, + ia, server_netaddr); params_ptr->Client_Stats_T.bytes_rdma_read = pt_ptr->Client_Stats.bytes_rdma_read; @@ -508,7 +508,7 @@ retry: status = DT_Performance_Test_Client (params_ptr, pt_ptr, - ia_handle, + (struct dat_ia *)ia, server_netaddr); break; } @@ -521,13 +521,13 @@ client_exit: DT_Tdep_PT_Debug (1,(phead, "%s: Cleaning Up ...\n", module)); /* Disconnect the EP */ - if (ep_handle && try_connect) + if (ep && try_connect) { /* * graceful attempt might fail because we got here due to * some error above, so we may as well try harder. */ - ret = dat_ep_disconnect (ep_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ep_disconnect (ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -547,7 +547,7 @@ client_exit: DT_Bpool_Destroy (pt_ptr, phead, bpool); /* Free the EP */ - if (ep_handle) + if (ep) { struct dat_event event; /* @@ -560,7 +560,7 @@ client_exit: &event); } while (ret == DAT_SUCCESS); - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -614,9 +614,9 @@ client_exit: } /* Free the PZ */ - if (pz_handle) + if (pz) { - ret = dat_pz_free (pz_handle); + ret = dat_pz_free (pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -629,10 +629,10 @@ client_exit: } /* Close the IA */ - if (ia_handle) + if (ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -640,7 +640,7 @@ client_exit: module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, Index: linux-kernel/test/dapltest/test/dapl_fft_endpoint.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) @@ -36,59 +36,59 @@ int DT_endpoint_generic (Params_t *param boolean_t destroy_pz_early) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; u32 rc, wanted; int res; DT_Tdep_Print_Head *phead; res = 1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - evd_handle = NULL; - conn_evd_handle = NULL; - send_evd_handle = NULL; - recv_evd_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + evd = NULL; + conn_evd = NULL; + send_evd = NULL; + recv_evd = NULL; dev_name = cmd->device_name; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; phead = params_ptr->phead; - rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, &ia_handle); + rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); if (destroy_pz_early) { - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } } - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, - DAT_EVD_CONNECTION_FLAG, &conn_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, + DAT_EVD_CONNECTION_FLAG, &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, pz_handle, recv_evd_handle, send_evd_handle, - conn_evd_handle, NULL, &ep_handle); + rc = dat_ep_create (ia, pz, recv_evd, send_evd, + conn_evd, NULL, &ep); if (destroy_pz_early) { wanted = DAT_INVALID_HANDLE; @@ -100,39 +100,39 @@ int DT_endpoint_generic (Params_t *param DT_assert_dat (phead, DAT_GET_TYPE (rc) == wanted); cleanup: - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (!destroy_pz_early && pz_handle) + if (!destroy_pz_early && pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; @@ -163,10 +163,10 @@ int DT_endpoint_case1 (Params_t *params_ int DT_endpoint_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE send_evd, conn_evd, recv_evd, cr_evd; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_ep *ep; + struct dat_evd *send_evd, *conn_evd, *recv_evd, *cr_evd; + struct dat_pz *pz; struct dat_event event; Bpool *bpool; int res; @@ -177,39 +177,39 @@ int DT_endpoint_case2 (Params_t *params_ Description: try to destroy ep with descriptor still in working queue\n"); res = 1; bpool = NULL; - pz_handle = NULL; - ia_handle = NULL; - ep_handle = NULL; + pz = NULL; + ia = NULL; + ep = NULL; send_evd = NULL; conn_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name = cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, 4096, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, 4096, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); DT_assert (phead, DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, 4096) == TRUE); - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -229,14 +229,14 @@ cleanup: rc = DT_Bpool_Destroy (NULL, phead, bpool); DT_assert_clean (phead, rc != FALSE); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_util.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c (working copy) @@ -59,7 +59,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the recv */ - ret = dat_ep_post_recv ( ep_context[i].ep_handle, + ret = dat_ep_post_recv ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -85,7 +85,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H boolean_t DT_handle_send_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll) @@ -122,7 +122,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the send */ - ret = dat_ep_post_send ( ep_context[i].ep_handle, + ret = dat_ep_post_send ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -173,7 +173,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -188,7 +188,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Send")) @@ -201,7 +201,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -215,7 +215,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -235,8 +235,8 @@ DT_handle_send_op (DT_Tdep_Print_Head *p boolean_t DT_handle_recv_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -283,7 +283,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Receive: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -298,7 +298,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Recv")) @@ -315,7 +315,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -353,7 +353,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: "F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -374,11 +374,11 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p /* * If we have multiple EPs we can't guarantee the order of - * completions, so disable ep_handle check + * completions, so disable ep check */ if (!DT_dto_check (phead, &dto_stat, - num_eps == 1?ep_context[i].ep_handle: NULL, + num_eps == 1?ep_context[i].ep: NULL, op->num_segs * op->seg_size, dto_cookie, "Send-reaped-on-recv")) @@ -395,7 +395,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -411,7 +411,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -428,7 +428,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -463,7 +463,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -516,7 +516,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (opcode == RDMA_WRITE) { - ret = dat_ep_post_rdma_write (ep_context[i].ep_handle, + ret = dat_ep_post_rdma_write (ep_context[i].ep, op->num_segs, iov, cookie, @@ -527,7 +527,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p else /* opcode == RDMA_READ */ { - ret = dat_ep_post_rdma_read ( ep_context[i].ep_handle, + ret = dat_ep_post_rdma_read ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -574,7 +574,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: %s: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -588,7 +588,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, (opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD"))) @@ -601,7 +601,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: Secondary completion seen for endpoint 0x%p (%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -619,7 +619,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: No completion seen for endpoint 0x%p (#%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); Index: linux-kernel/test/dapltest/test/dapl_fft_pz.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) @@ -35,9 +35,9 @@ int DT_pz_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_evd *evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -47,25 +47,25 @@ int DT_pz_case0 ( Params_t *params_ptr, Description: Test if we can normally create pz and destroy it.\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + ia = NULL; + pz = NULL; + evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); cleanup: - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); } return res; @@ -75,10 +75,10 @@ cleanup: int DT_pz_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE conn_evd, send_evd, recv_evd, cr_evd; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *conn_evd, *send_evd, *recv_evd, *cr_evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -88,42 +88,42 @@ int DT_pz_case1 (Params_t *params_ptr, F Description: try to destroy pz with vi still associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; conn_evd = NULL; send_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } cleanup: /* corrrect order */ - if (ep_handle) + if (ep) { - rc=dat_ep_free (ep_handle); + rc=dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn_evd) @@ -141,15 +141,15 @@ cleanup: rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -160,8 +160,8 @@ cleanup: int DT_pz_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz *pz; Bpool *bpool; u32 rc; int res; @@ -174,26 +174,26 @@ int DT_pz_case2 (Params_t *params_ptr, F associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; bpool = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate and register bpool */ - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, BUFFSIZE, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } @@ -205,15 +205,15 @@ cleanup: DT_Tdep_PT_Printf (phead, "Warning: Destroy bpool fails, reboot for cleanup\n"); return 0; } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: linux-kernel/test/dapltest/test/dapl_fft_hwconn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (working copy) @@ -31,8 +31,8 @@ int DT_hwconn_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE nic_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * nic_handle; + struct dat_evd *evd; u32 rc; int res = 1; DT_Tdep_Print_Head *phead; @@ -43,9 +43,9 @@ int DT_hwconn_case0 ( Params_t *params_p dev_name= cmd->device_name; nic_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); @@ -58,9 +58,9 @@ cleanup: /*--------------------------------------------------------*/ int DT_hwconn_case1 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; char dev_name[100]; int i; DT_Tdep_Print_Head *phead; @@ -94,8 +94,8 @@ int DT_hwconn_case1 ( Params_t *params_p sprintf (dev_name, "%s", "34df"); /* number_letter */ } - evd_handle = DAT_HANDLE_NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + evd = NULL; + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); if (DAT_GET_TYPE (rc) != DAT_PROVIDER_NOT_FOUND) { const char *major_msg, *minor_msg; @@ -120,7 +120,7 @@ int DT_hwconn_case1 ( Params_t *params_p /*--------------------------------------------------------*/ int DT_hwconn_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; int res=1; DT_Tdep_Print_Head *phead; @@ -156,9 +156,9 @@ int DT_hwconn_case3 (Params_t *params_pt DT_fft_init_client (params_ptr, cmd, &conn); /* try to close nic when vi have not destroyed */ - if (conn.ia_handle) + if (conn.ia) { - rc= dat_ia_close (conn.ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc= dat_ia_close (conn.ia, DAT_CLOSE_ABRUPT_FLAG); if (rc !=DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Warning: dat_ia_close fails %s, reboot for cleanup\n", Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (revision 2563) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (working copy) @@ -34,7 +34,7 @@ int DT_dataxfer_generic ( DT_Tdep_Print_ u32 rc=0; int res=1; DT_fft_init_server (phead, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); DT_fft_listen (phead, &conn); @@ -81,7 +81,7 @@ cleanup: int DT_dataxfer_case0 ( DT_Tdep_Print_Head *phead, FFT_Cmd_t *cmd) { DT_Tdep_PT_Printf (phead, "\ - Description: Call dat_ep_post_send with null ep_handle.\n"); + Description: Call dat_ep_post_send with null ep.\n"); return DT_dataxfer_generic (phead, cmd, 0); } Index: linux-kernel/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_tdep.h (revision 2563) +++ linux-kernel/test/dapltest/include/dapl_tdep.h (working copy) @@ -31,7 +31,7 @@ #include "dapl_proto.h" #ifdef __KDAPL__ -typedef DAT_HANDLE DAT_CNO_HANDLE; +typedef void * DAT_CNO_HANDLE; #endif /* function prototypes */ @@ -45,21 +45,21 @@ int DT_Tdep_Execute_Test ( Params_t *params_ptr ) ; u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr); + struct dat_evd * *evd_ptr); u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle); +DT_Tdep_evd_free (struct dat_evd * evd); u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd * evd, unsigned long timeout, struct dat_event *event); u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd * evd, struct dat_event *event); #endif Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2563) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -54,7 +54,7 @@ typedef struct { typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL port; int pipeline_len; @@ -70,17 +70,17 @@ typedef struct boolean_t is_remote_little_endian; DAT_CONN_QUAL base_port; struct dat_ia_attr ia_attr; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; int reqt_evd_length; - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ int recv_evd_length; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ + struct dat_evd * recv_evd_hdl; /* receive */ int conn_evd_length; - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ + struct dat_evd * conn_evd_hdl; /* connect */ int creq_evd_length; - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_evd * creq_evd_hdl; /* "" request */ Performance_Ep_Context_t ep_context; } Performance_Test_t; Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2563) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -81,10 +81,10 @@ extern int g_status; /* dapl_bpool.c */ Bpool * DT_BpoolAlloc (Per_Test_Data_t * pt_ptr, DT_Tdep_Print_Head* phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_ep *ep, + struct dat_evd *rmr_evd, int seg_size, int num_segs, int alignment, @@ -106,7 +106,7 @@ void DT_Bpool_print (DT_Tdep_ /* dapl_cnxn.c */ int get_ep_connection_state (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle); + struct dat_ep *ep); /* dapl_client.c */ int DT_cs_Client (Params_t * params_ptr, @@ -236,7 +236,7 @@ void DT_Performance_Cmd_Endia /* dapl_performance_client.c */ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, struct sockaddr *remote); boolean_t DT_Performance_Test_Client_Connect ( @@ -261,7 +261,7 @@ boolean_t DT_Performance_Test /* dapl_performance_util.c */ boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -272,20 +272,20 @@ int DT_Performance_Test_Dest boolean_t is_server); boolean_t DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats); unsigned int DT_performance_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Mode_Type mode, Performance_Stats_t *stats); unsigned int DT_performance_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); unsigned int DT_performance_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); /* dapl_performance_stats.c */ @@ -346,69 +346,69 @@ void DT_Free_Per_Test_Data (Per_Test_D /* dapl_test_util.c */ boolean_t DT_query (Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle); + struct dat_ia * ia, + struct dat_ep * ep); boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_post_send_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_conn_event_wait (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep *ep, + struct dat_evd *evd, enum dat_event_number *event_number); boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ); + struct dat_evd *evd, + struct dat_ep * *ep ); boolean_t DT_cr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p); boolean_t DT_dto_event_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_ptr); boolean_t DT_dto_check ( DT_Tdep_Print_Head* phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message); boolean_t DT_rmr_check ( DT_Tdep_Print_Head* phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message); boolean_t DT_cr_check (DT_Tdep_Print_Head* phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp *psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr **crp, char *message); /* dapl_thread.c */ @@ -460,13 +460,13 @@ void DT_Transaction_Cmd_Endia boolean_t to_wire); /* dapl_transaction_test.c */ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote); void DT_Transaction_Test_Server (void *params); boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -491,15 +491,15 @@ boolean_t DT_handle_post_recv boolean_t DT_handle_send_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll); boolean_t DT_handle_recv_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -507,7 +507,7 @@ boolean_t DT_handle_recv_op ( boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -600,15 +600,15 @@ void DT_assert_fail (DT_Tdep_Print_H char *file, char *baseFile, int line); -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle); +int DT_ia_open (char *dev_name, struct dat_ia **ia); int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle); + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep); void DT_fft_init_conn_struct (FFT_Connection_t *conn); void DT_fft_init_client (Params_t *params_ptr, FFT_Cmd_t *cmd, Index: linux-kernel/test/dapltest/include/dapl_test_data.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_test_data.h (revision 2563) +++ linux-kernel/test/dapltest/include/dapl_test_data.h (working copy) @@ -46,16 +46,16 @@ typedef struct int NextPortNumber; int num_clients; spinlock_t num_clients_lock; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; - DAT_EVD_HANDLE reqt_evd_hdl; - DAT_EVD_HANDLE conn_evd_hdl; - DAT_EVD_HANDLE creq_evd_hdl; - DAT_EVD_HANDLE async_evd_hdl; - DAT_EVD_HANDLE rmr_evd_hdl; - DAT_EP_HANDLE ep_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_evd *recv_evd_hdl; + struct dat_evd *reqt_evd_hdl; + struct dat_evd *conn_evd_hdl; + struct dat_evd *creq_evd_hdl; + struct dat_evd *async_evd_hdl; + struct dat_evd *rmr_evd_hdl; + struct dat_ep *ep; + struct dat_sp *psp; Bpool *bpool; } Per_Server_Data_t; Index: linux-kernel/test/dapltest/include/dapl_bpool.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2563) +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) @@ -38,21 +38,21 @@ typedef struct Bpool_tag Bpool; struct Bpool_tag { unsigned char *alloc_ptr; uint32_t alloc_size; - DAT_PZ_HANDLE pz_handle; + struct dat_pz * pz; int seg_size; int num_segs; /* num segments */ unsigned char *buffer_start; /* Start of buffer area */ u64 buffer_size; /* Size of data buffer (rounded) */ u64 reg_addr; /* start of registered area */ u64 reg_size; /* size of registered area */ - DAT_EP_HANDLE ep_handle; /* EP area is registered to */ - DAT_LMR_HANDLE lmr_handle; /* local access */ + struct dat_ep * ep; /* EP area is registered to */ + struct dat_lmr * lmr; /* local access */ DAT_LMR_CONTEXT lmr_context; struct dat_lmr_triplet *tripl_start; /* local IOV */ boolean_t enable_rdma_write; /* remote access */ boolean_t enable_rdma_read; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; - DAT_EVD_HANDLE rmr_evd_handle; + struct dat_evd * rmr_evd; }; #endif Index: linux-kernel/test/dapltest/include/dapl_fft_util.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2563) +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) @@ -55,14 +55,14 @@ typedef struct { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_PSP_HANDLE psp_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE cr_evd, conn_evd, send_evd, recv_evd; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_sp *psp; + struct dat_ep *ep; + struct dat_evd *cr_evd, *conn_evd, *send_evd, *recv_evd; struct dat_event event; int count; - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; Bpool *bpool; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2563) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -49,13 +49,13 @@ typedef struct typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL ia_port; Bpool *bp; Transaction_Test_Op_t op[ MAX_OPS ]; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; } Ep_Context_t; @@ -75,7 +75,7 @@ typedef struct boolean_t is_server; boolean_t remote_is_little_endian; Per_Test_Data_t *pt_ptr; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; @@ -84,11 +84,11 @@ typedef struct Thread *thread; /* This group set up by each thread in DT_Transaction_Main() */ - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_pz * pz; + struct dat_evd * recv_evd_hdl; /* receive */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ + struct dat_evd * conn_evd_hdl; /* connect */ + struct dat_evd * creq_evd_hdl; /* "" request */ Ep_Context_t *ep_context; /* Statistics set by DT_Transaction_Run() */ Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2563) +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -39,7 +39,7 @@ typedef struct Tdep_Evd_Tag struct Tdep_Evd_Tag *evd_next; struct Tdep_Event_Tag *event_next; struct Tdep_Event_Tag *event_last; - DAT_EVD_HANDLE evd_handle; + struct dat_evd * evd; DT_WAIT_OBJECT wait_object; } Tdep_Evd; @@ -94,11 +94,11 @@ KDT_Evd_Destroy(void) } u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr) + struct dat_evd **evd_ptr_ptr) { u32 dat_status; @@ -120,12 +120,12 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha upcall.upcall_func = DT_Tdep_Event_Callback; upcall.instance_data = evd_ptr; - dat_status = dat_evd_kcreate (ia_handle, + dat_status = dat_evd_kcreate (ia, evd_min_qlen, DAT_UPCALL_SINGLE_INSTANCE, &upcall, evd_flags, - evd_handle_ptr); + evd_ptr_ptr); if (dat_status != DAT_SUCCESS) { goto error; @@ -134,7 +134,7 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha /* initialize evd_ptr */ evd_ptr->event_next = NULL; evd_ptr->event_last = NULL; - evd_ptr->evd_handle = *evd_handle_ptr; + evd_ptr->evd = *evd_ptr_ptr; DT_Mdep_wait_object_init (&evd_ptr->wait_object); /* add evd_ptr to front of evd list */ @@ -154,7 +154,7 @@ error: return (dat_status); } u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd *evd, struct dat_event *dat_event) { u32 dat_status; @@ -163,11 +163,11 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ unsigned long flags; dat_status = DAT_SUCCESS; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -181,7 +181,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } /* Get event */ spin_lock_irqsave (&DT_Evd_Lock, flags); - event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd_handle */ + event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd */ if (event) { evd_ptr->event_next = event->event_next; /* point to next event */ @@ -212,7 +212,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd *evd, unsigned long timeout, struct dat_event *dat_event) { @@ -223,11 +223,11 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han dat_status = DAT_CLASS_ERROR | DAT_INTERNAL_ERROR; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -290,7 +290,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han } u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle) +DT_Tdep_evd_free (struct dat_evd *evd) { Tdep_Evd *next; Tdep_Evd *last; @@ -298,13 +298,13 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han last = NULL; spin_lock_irq (&DT_Evd_Lock); next = DT_Evd_List; - if (next->evd_handle == evd_handle) + if (next->evd == evd) { DT_Evd_List = next->evd_next; } else { - while (next && next->evd_handle != evd_handle) + while (next && next->evd != evd) { last = next; next = next->evd_next; @@ -317,7 +317,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han spin_unlock_irq (&DT_Evd_Lock); DT_Mdep_Free (next); - return dat_evd_free(evd_handle); + return dat_evd_free(evd); } static void DT_Tdep_Event_Callback ( Index: linux-kernel/dat-provider/dapl_cr_reject.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_reject.c (revision 2563) +++ linux-kernel/dat-provider/dapl_cr_reject.c (working copy) @@ -40,36 +40,36 @@ * Reject a connection request from the active remote side requesting * an Endpoint. */ -u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) +u32 dapl_cr_reject(struct dat_cr *cr) { struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; struct dapl_sp *sp_ptr; u32 dat_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* * Clean up provider created EP if there is one: only if * DAT_PSP_PROVIDER_FLAG was set on the PSP */ - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - entry_ep_handle = cr_ptr->param.local_ep_handle; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; + entry_ep = cr_ptr->param.local_ep; entry_ep_state = 0; if (ep_ptr) { entry_ep_state = ep_ptr->param.ep_state; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - cr_ptr->param.local_ep_handle = NULL; + cr_ptr->param.local_ep = NULL; } dat_status = dapl_ib_reject_connection(cr_ptr->ib_cm_handle); @@ -77,8 +77,8 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand if (ep_ptr != NULL) { /* Revert our state to the beginning */ ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; - cr_ptr->param.local_ep_handle = (DAT_EP_HANDLE) ep_ptr; + cr_ptr->param.local_ep = entry_ep; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; } } else { /* @@ -88,13 +88,13 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand */ sp_ptr = cr_ptr->sp_ptr; if (ep_ptr && sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) - (void)dapl_ep_free(ep_ptr); + (void)dapl_ep_free((struct dat_ep *)ep_ptr); /* Remove the CR from the queue, then free it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); } Index: linux-kernel/dat-provider/dapl_srq.c =================================================================== --- linux-kernel/dat-provider/dapl_srq.c (revision 2563) +++ linux-kernel/dat-provider/dapl_srq.c (working copy) @@ -51,13 +51,9 @@ */ static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) { - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); - - /* reset magic to prevent reuse */ - srq_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); + dapl_ia_unlink_srq(srq_ptr->common.owner_ia, srq_ptr); dapl_cb_free(&srq_ptr->recv_buffer); - /* no need to destroy srq_ptr->header.lock */ + /* no need to destroy srq_ptr->common.lock */ kfree(srq_ptr); } @@ -91,19 +87,14 @@ static struct dapl_srq *dapl_srq_alloc(s /* zero the structure */ memset(srq_ptr, 0, sizeof *srq_ptr); - /* - * initialize the header - */ - srq_ptr->header.provider = ia_ptr->header.provider; - srq_ptr->header.magic = DAPL_MAGIC_SRQ; - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; - srq_ptr->header.owner_ia = ia_ptr; - srq_ptr->header.user_context.as_64 = 0; - srq_ptr->header.user_context.as_ptr = NULL; + srq_ptr->srq.provider = ia_ptr->ia.provider; + srq_ptr->common.owner_ia = ia_ptr; + srq_ptr->common.user_context.as_64 = 0; + srq_ptr->common.user_context.as_ptr = NULL; atomic_set(&srq_ptr->srq_ref_count, 0); - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); - spin_lock_init(&srq_ptr->header.lock); + dapl_llist_init_entry(&srq_ptr->common.ia_list_entry); + spin_lock_init(&srq_ptr->common.lock); /* * Initialize the body. @@ -130,15 +121,15 @@ bail: * dapl_srq_create * * Create an instance of a Shared Receive Queue that is provided to the - * consumer at srq_handle. + * consumer at srq. * * Input: - * ia_handle - * pz_handle + * ia + * pz * srq_attr * * Output: - * srq_handle + * srq * * Returns: * DAT_SUCCESS @@ -148,8 +139,8 @@ bail: * ?DAT_INVALID_ATTRIBUTE?? * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) +u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_srq_attr *srq_attr, struct dat_srq **srq) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; @@ -157,14 +148,14 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_create (%p, %p, %p, %p)\n", - ia_handle, pz_handle, srq_attr, srq_handle); + ia, pz, srq_attr, srq); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -175,12 +166,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - - if (srq_handle == NULL) { + if (srq == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -197,9 +183,9 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han goto bail; } - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; + srq_ptr->param.ia = (struct dat_ia *) ia_ptr; srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; - srq_ptr->param.pz_handle = pz_handle; + srq_ptr->param.pz = pz; /* * XXX Allocate provider resource here!!! @@ -211,7 +197,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han /* Link it onto the IA */ dapl_ia_link_srq(ia_ptr, srq_ptr); - *srq_handle = srq_ptr; + *srq = (struct dat_srq *)srq_ptr; bail: return status; @@ -223,7 +209,7 @@ bail: * Destroy an instance of an SRQ * * Input: - * srq_handle + * srq * * Output: * none @@ -233,22 +219,22 @@ bail: * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) +u32 dapl_srq_free(struct dat_srq *srq) { struct dapl_srq *srq_ptr; struct dapl_ia *ia_ptr; struct dat_srq_param *param; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq); - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; param = &srq_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { + if (!srq_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -263,17 +249,17 @@ u32 dapl_srq_free(DAT_SRQ_HANDLE srq_han goto bail; } - ia_ptr = srq_ptr->header.owner_ia; + ia_ptr = srq_ptr->common.owner_ia; /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_lock_irqsave(&srq_ptr->common.lock, srq_ptr->common.flags); /* Remove the SRQ from the IA */ dapl_ia_unlink_srq(ia_ptr, srq_ptr); - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_unlock_irqrestore(&srq_ptr->common.lock, srq_ptr->common.flags); /* * Finish tearing everything down. @@ -301,7 +287,7 @@ bail: * over a connection of any ep handle into local_iov * * Input: - * srq_handle + * srq * num_segments * local_iov * user_cookie @@ -318,7 +304,7 @@ bail: * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie) { @@ -328,14 +314,14 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_post_recv (%p, %d, %p, %P)\n", - srq_handle, num_segments, local_iov, user_cookie.as_64); + srq, num_segments, local_iov, user_cookie.as_64); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * Synchronization ok since this buffer is only used for receive @@ -369,16 +355,15 @@ bail: return status; } -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) +u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_srq_query (%p, %x, %p)\n", - srq_handle, srq_param); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", srq, + srq_param); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -387,7 +372,7 @@ u32 dapl_srq_query(DAT_SRQ_HANDLE srq_ha goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * XXX Need to calculate available_dto_count and outstanding_dto_count @@ -407,7 +392,7 @@ bail: * Modify the size fo the event queue of a Shared Recieve Queue * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -421,22 +406,22 @@ bail: * DAT_INVALID_STATE */ -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) +u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", - srq_handle, srq_max_recv_dto); + srq, srq_max_recv_dto); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; - ia_ptr = srq_ptr->header.owner_ia; + srq_ptr = (struct dapl_srq *)srq; + ia_ptr = srq_ptr->common.owner_ia; /* * Check for nonsense requests per the spec @@ -463,7 +448,7 @@ bail: * event if it is reached. * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -476,20 +461,20 @@ bail: * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) +u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", - srq_handle, low_watermark); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq, + low_watermark); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* XXX Put implementation here XXX */ Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2563) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -171,7 +171,7 @@ u32 dapl_ib_close_hca(struct dapl_hca *h * Alloc a CQ * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * cqlen minimum QLen * @@ -227,7 +227,7 @@ u32 dapl_ib_cq_resize(struct dapl_ia *ia * Set up CQ completion notifications * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * * Output: @@ -253,7 +253,7 @@ u32 dapl_set_cq_notify(struct dapl_ia *i * Register a virtual memory region * * Input: - * ia_handle IA handle + * ia IA handle * lmr pointer to dapl_lmr struct * virt_addr virtual address of beginning of mem region * length length of memory region @@ -300,7 +300,7 @@ u32 dapl_ib_mr_register_physical(struct iova = buf_list[0].addr; acl = dapl_ib_convert_mem_privileges(privileges); acl |= IB_ACCESS_MW_BIND; - mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz_handle)->pd, + mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz)->pd, buf_list, length, acl, &iova); kfree(buf_list); if (IS_ERR(mr)) { @@ -393,7 +393,7 @@ u32 dapl_ib_mw_alloc(struct dapl_rmr *rm int ib_status; struct ib_mw *mw; - mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz_handle)->pd); + mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz)->pd); if (IS_ERR(mw)) { ib_status = PTR_ERR(mw); dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -540,7 +540,7 @@ u32 dapl_ib_mw_unbind(struct dapl_rmr *r * Set up an asynchronous callbacks of various kinds * * Input: - * ia_handle IA handle + * ia IA handle * handler_type type of handler to set up * callback_handle handle param for completion callbacks * callback callback routine pointer Index: linux-kernel/dat-provider/dapl_ia_memtype_hint.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_memtype_hint.c (revision 2563) +++ linux-kernel/dat-provider/dapl_ia_memtype_hint.c (working copy) @@ -37,7 +37,7 @@ * Provide a hint about optimal memory alignment and size * * Input: - * ia_handle, + * ia, * mem_type, * length, * mem_optimization, @@ -53,7 +53,7 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, @@ -63,11 +63,11 @@ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE i u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", - ia_handle); + ia); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; Index: linux-kernel/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2563) +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) @@ -74,17 +74,17 @@ u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ attr = &ep_ptr->param.ep_attr; - dapl_os_assert(ep_ptr->param.pz_handle != NULL); - ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz_handle)->pd; + dapl_os_assert(ep_ptr->param.pz != NULL); + ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz)->pd; ib_hca_handle = ia_ptr->hca_ptr->ib_hca_handle; cq_recv = NULL; cq_send = NULL; cq_recv = dapl_get_dto_cq(ia_ptr, - (struct dapl_evd *)ep_ptr->param.recv_evd_handle); + (struct dapl_evd *)ep_ptr->param.recv_evd); cq_send = dapl_get_dto_cq(ia_ptr, (struct dapl_evd *)ep_ptr->param. - request_evd_handle); + request_evd); qp_attr.send_cq = cq_send; qp_attr.recv_cq = cq_recv; @@ -160,7 +160,7 @@ struct ib_cq *dapl_get_dto_cq(struct dap int cq_entries; int status; - if (evd_ptr != DAT_HANDLE_NULL) + if (evd_ptr != NULL) cq = evd_ptr->cq; else if (ia_ptr->hca_ptr->null_cq != NULL) cq = ia_ptr->hca_ptr->null_cq; Index: linux-kernel/dat-provider/dapl_openib_util.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.h (revision 2563) +++ linux-kernel/dat-provider/dapl_openib_util.h (working copy) @@ -59,7 +59,7 @@ u32 dapl_ib_qp_free(struct dapl_ep *ep_p u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, struct dat_ep_attr *ep_attr); -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data); @@ -70,12 +70,12 @@ u32 dapl_ib_setup_conn_listener(struct d u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr); -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, + struct dat_ep *ep, int private_data_size, const void *private_data); -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle); +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm); u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, DAPL_ASYNC_HANDLER_TYPE handler_type, @@ -126,14 +126,14 @@ u32 dapl_ib_completion_notify(struct dap enum dat_dto_completion_status dapl_ib_get_dto_status(struct ib_wc *cqe_ptr); -void dapl_ib_reinit_ep(struct dapl_ep * ep_ptr); +void dapl_ib_reinit_ep(struct dapl_ep *ep_ptr); -void dapl_ib_disconnect_clean(struct dapl_ep * ep_ptr, boolean_t passive); +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t passive); u32 dapl_ib_get_async_event(struct ib_event *cause_ptr, enum dat_event_number *async_event); -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address); static inline u32 dapl_ib_status_convert(int32_t ib_status) Index: linux-kernel/dat-provider/dapl_ia_close.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_close.c (revision 2563) +++ linux-kernel/dat-provider/dapl_ia_close.c (working copy) @@ -40,7 +40,7 @@ * Close a provider, clean up resources, etc. * * Input: - * ia_handle + * ia * * Output: * none @@ -50,30 +50,29 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags ia_flags) +u32 dapl_ia_close(struct dat_ia *ia, enum dat_close_flags ia_flags) { struct dapl_ia *ia_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); + "dapl_ia_close (%p, %d)\n", ia, ia_flags); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) { + if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) dat_status = dapl_ia_abrupt_close(ia_ptr); - } else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) { + else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) dat_status = dapl_ia_graceful_close(ia_ptr); - } else { + else dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_cr_callback.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_callback.c (revision 2563) +++ linux-kernel/dat-provider/dapl_cr_callback.c (working copy) @@ -50,9 +50,9 @@ static u32 dapl_connection_request(struc struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; - cr_ptr = dapl_cr_alloc(sp_ptr->header.owner_ia); + cr_ptr = dapl_cr_alloc(sp_ptr->common.owner_ia); if (cr_ptr == NULL) { /* Invoking function will call dapl_ib_cm_reject() */ return DAT_INSUFFICIENT_RESOURCES; @@ -79,7 +79,7 @@ static u32 dapl_connection_request(struc } /* EP will be NULL unless RSP service point */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { /* @@ -89,7 +89,7 @@ static u32 dapl_connection_request(struc * EP we are out of resources and need to tell the * requestor that we cant help them. */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; ep_ptr = dapl_ep_alloc(ia_ptr, NULL); if (ep_ptr == NULL) { dapl_cr_free(cr_ptr); @@ -100,7 +100,7 @@ static u32 dapl_connection_request(struc dapl_ia_link_ep(ia_ptr, ep_ptr); } - cr_ptr->param.local_ep_handle = ep_ptr; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; if (ep_ptr != NULL) { /* Assign valid EP fields: RSP and PSP_PROVIDER_FLAG only */ @@ -109,12 +109,11 @@ static u32 dapl_connection_request(struc DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING; } else { /* RSP */ - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == DAT_SP_TYPE_RSP); ep_ptr->param.ep_state = DAT_EP_STATE_PASSIVE_CONNECTION_PENDING; } - ep_ptr->cm_handle = ib_cm_handle; + ep_ptr->cm = ib_cm_handle; } /* link the CR onto the SP so we can pick it up later */ @@ -122,16 +121,16 @@ static u32 dapl_connection_request(struc /* Post the event. */ /* assign sp_ptr to union to avoid typecast errors from some compilers */ - sp_handle.psp_handle = (DAT_PSP_HANDLE) sp_ptr; - dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp_handle, - (struct sockaddr *) &sp_ptr->header.owner_ia->hca_ptr->hca_address, - sp_ptr->conn_qual, (DAT_CR_HANDLE) cr_ptr); + sp = (struct dat_sp *)sp_ptr; + dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp, + (struct sockaddr *)&sp_ptr->common.owner_ia->hca_ptr->hca_address, + sp_ptr->conn_qual, (struct dat_cr *)cr_ptr); if (dat_status != DAT_SUCCESS) { /* Take the CR off the list, we can't use it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); (void)dapl_ib_reject_connection(ib_cm_handle); @@ -157,13 +156,7 @@ struct dapl_ep *dapl_get_sp_ep(struct da /* * acquire the lock, we may be racing with other threads here */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); - /* Verify under lock that the SP is still valid */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - return NULL; - } + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * There are potentially multiple connections in progress. Need to * go through the list and find the one we are interested @@ -172,16 +165,12 @@ struct dapl_ep *dapl_get_sp_ep(struct da */ cr_ptr = dapl_sp_search_cr(sp_ptr, ib_cm_handle); if (cr_ptr == NULL) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); return NULL; } - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - - /* Quick check to ensure our EP is still valid */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) - ep_ptr = NULL; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; /* The CR record is discarded in all except for the CONNECTED case, * as it will have no further relevance. @@ -211,31 +200,31 @@ struct dapl_ep *dapl_get_sp_ep(struct da "dapl_get_sp_ep disconnect dump sp: %p \n", sp_ptr); /* Decrement the ref count on the EVD */ - if (sp_ptr->evd_handle) { + if (sp_ptr->evd) { atomic_dec(&((struct dapl_evd *)sp_ptr-> - evd_handle)->evd_ref_count); - sp_ptr->evd_handle = NULL; + evd)->evd_ref_count); + sp_ptr->evd = NULL; } sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - (void)dapl_ib_remove_conn_listener(sp_ptr->header. + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + (void)dapl_ib_remove_conn_listener(sp_ptr->common. owner_ia, sp_ptr); dapl_ia_unlink_sp((struct dapl_ia *) - sp_ptr->header.owner_ia, + sp_ptr->common.owner_ia, sp_ptr); dapl_sp_dealloc(sp_ptr); dapl_cr_free(cr_ptr); goto skip_unlock; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); } else - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); skip_unlock: return ep_ptr; @@ -255,16 +244,6 @@ void dapl_cr_callback(struct dapl_cm_id ib_cm_handle, event, sp_ptr); /* - * The SP pointer could have been cleaned up in a racing - * CM callback, check to see if we should just exit here - */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - return; - } - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); - - /* * CONNECT_REQUEST events create an event on the PSP * EVD, which will trigger connection processing. The * sequence is: @@ -277,7 +256,7 @@ void dapl_cr_callback(struct dapl_cm_id */ if (event == DAT_CONNECTION_REQUEST_EVENT) { ep_ptr = NULL; - evd_ptr = sp_ptr->evd_handle; + evd_ptr = (struct dapl_evd *)sp_ptr->evd; } else { /* see if there is an EP connected with this CM handle */ ep_ptr = dapl_get_sp_ep(ib_cm_handle, sp_ptr, event); @@ -286,7 +265,7 @@ void dapl_cr_callback(struct dapl_cm_id if (ep_ptr == NULL) return; - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; /* if something has happened to our EVD, bail. */ if (evd_ptr == NULL) return; @@ -296,30 +275,29 @@ void dapl_cr_callback(struct dapl_cm_id switch (event) { case DAT_CONNECTION_REQUEST_EVENT: - { /* * Requests arriving on a disabled SP are immediately rejected */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); if (sp_ptr->listening == FALSE) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, "---> dapl_cr_callback: conn event on down SP\n"); (void)dapl_ib_reject_connection(ib_cm_handle); return; } - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_RSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_RSP) { /* * RSP connections only allow a single connection. Close * it down NOW so we reject any further connections. */ sp_ptr->listening = FALSE; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* * Only occurs on the passive side of a connection @@ -332,24 +310,20 @@ void dapl_cr_callback(struct dapl_cm_id evd_ptr = NULL; break; - } case DAT_CONNECTION_EVENT_ESTABLISHED: - { /* This is just a notification the connection is now * established, there isn't any private data to deal with. * * Update the EP state and cache a copy of the cm handle, * then let the user know we are ready to go. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->header.magic != DAPL_MAGIC_EP || - ep_ptr->param.ep_state != - DAT_EP_STATE_COMPLETION_PENDING) { + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING) { /* If someone pulled the plug on the EP or connection, * just exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; /* Set evd_ptr = NULL so we don't generate an event below */ evd_ptr = NULL; @@ -358,72 +332,63 @@ void dapl_cr_callback(struct dapl_cm_id } ep_ptr->param.ep_state = DAT_EP_STATE_CONNECTED; - ep_ptr->cm_handle = ib_cm_handle; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + ep_ptr->cm = ib_cm_handle; + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_DISCONNECTED: - { /* * EP is now fully disconnected; initiate any post processing * to reset the underlying QP and get the EP ready for * another connection */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { /* The disconnect has already occurred, we are now * cleaned up and ready to exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); return; } ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: - { /* * After posting an accept the requesting node has * stopped talking. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_BROKEN: - { - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } default: - { evd_ptr = NULL; dapl_os_assert(0); /* shouldn't happen */ break; } - } if (evd_ptr != NULL) dat_status = dapl_evd_post_connection_event(evd_ptr, event, - (DAT_HANDLE) ep_ptr, + (struct dat_ep *)ep_ptr, 0, NULL); if (dat_status != DAT_SUCCESS) { Index: linux-kernel/dat-provider/dapl_cr_accept.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_accept.c (revision 2563) +++ linux-kernel/dat-provider/dapl_cr_accept.c (working copy) @@ -39,42 +39,39 @@ * Establish a connection between active side (remote endpoint) * and passive side (local endpoint). */ -u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - int private_data_size, const void *private_data) +u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, + const void *private_data) { struct dapl_ep *ep_ptr; u32 dat_status; struct dapl_cr *cr_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_cr_accept (%p, %p, %d, %p)\n", - cr_handle, ep_handle, private_data_size, private_data); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_accept (%p, %p, %d, %p)\n", + cr, ep, private_data_size, private_data); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* - * Return an error if we have an ep_handle and the CR already has an + * Return an error if we have an ep and the CR already has an * EP, indicating this is an RSP connection or PSP_PROVIDER_FLAG was * specified. */ - if (ep_handle != NULL && - (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP) || - cr_ptr->param.local_ep_handle != NULL)) { + if (ep != NULL && cr_ptr->param.local_ep != NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* Make sure we have an EP handle in one place or another */ - if (ep_handle == NULL && cr_ptr->param.local_ep_handle == NULL) { + if (ep == NULL && cr_ptr->param.local_ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -86,84 +83,71 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand } /* - * ep_handle is NULL if the user specified DAT_PSP_PROVIDER_FLAG + * ep is NULL if the user specified DAT_PSP_PROVIDER_FLAG * OR this is an RSP connection; retrieve it from the cr. */ - if (ep_handle == NULL) { - ep_handle = cr_ptr->param.local_ep_handle; - if ((((struct dapl_ep *)ep_handle)->param.ep_state != + if (ep == NULL) { + ep = cr_ptr->param.local_ep; + if ((((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) - && (((struct dapl_ep *)ep_handle)->param.ep_state != + && (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING)) { return DAT_INVALID_STATE; } } else { /* ensure this EP isn't connected or in use */ - if (((struct dapl_ep *)ep_handle)->param.ep_state != + if (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_UNCONNECTED) { return DAT_INVALID_STATE; } } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required * We do all verification and state change under lock, at which * point the EP state should protect us from most races. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* The qp must be attached by this point! */ - if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) - { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } entry_ep_state = ep_ptr->param.ep_state; - entry_ep_handle = cr_ptr->param.local_ep_handle; + entry_ep = cr_ptr->param.local_ep; ep_ptr->param.ep_state = DAT_EP_STATE_COMPLETION_PENDING; - ep_ptr->cm_handle = cr_ptr->ib_cm_handle; + ep_ptr->cm = cr_ptr->ib_cm_handle; ep_ptr->cr_ptr = cr_ptr; ep_ptr->param.remote_ia_address_ptr = cr_ptr->param.remote_ia_address_ptr; - cr_ptr->param.local_ep_handle = ep_handle; + cr_ptr->param.local_ep = ep; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); - dat_status = dapl_ib_accept_connection(cr_handle, - ep_handle, + dat_status = dapl_ib_accept_connection(cr, + ep, private_data_size, private_data); /* @@ -176,16 +160,14 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand * system error, app termination, perhaps other reasons. */ dat_status = - dapl_evd_post_connection_event(ep_ptr->param. - request_evd_handle, + dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. + request_evd, DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR, - (DAT_HANDLE) ep_ptr, + (void *) ep_ptr, 0, NULL); - - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } else { ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; + cr_ptr->param.local_ep = entry_ep; ep_ptr->cr_ptr = NULL; ep_ptr->param.remote_ia_address_ptr = NULL; } @@ -199,14 +181,8 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); } - } else { - /* Make this CR invalid. We need to hang on to it until - * the connection terminates, but it's destroyed from - * the app point of view. - */ - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_pz.c =================================================================== --- linux-kernel/dat-provider/dapl_pz.c (revision 2563) +++ linux-kernel/dat-provider/dapl_pz.c (working copy) @@ -44,18 +44,13 @@ static struct dapl_pz *dapl_pz_alloc(str /* zero the structure */ memset(pz, 0, sizeof *pz); - /* - * initialize the header - */ - pz->header.provider = ia->header.provider; - pz->header.magic = DAPL_MAGIC_PZ; - pz->header.handle_type = DAT_HANDLE_TYPE_PZ; - pz->header.owner_ia = ia; - pz->header.user_context.as_64 = 0; - pz->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&pz->header.ia_list_entry); + pz->pz.provider = ia->ia.provider; + pz->common.owner_ia = ia; + pz->common.user_context.as_64 = 0; + pz->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&pz->common.ia_list_entry); dapl_ia_link_pz(ia, pz); - spin_lock_init(&pz->header.lock); + spin_lock_init(&pz->common.lock); /* * initialize the body @@ -67,83 +62,81 @@ static struct dapl_pz *dapl_pz_alloc(str static void dapl_pz_dealloc(struct dapl_pz *pz) { - /* reset magic to prevent reuse */ - pz->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_pz(pz->header.owner_ia, pz); + dapl_ia_unlink_pz(pz->common.owner_ia, pz); kfree(pz); } -u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) +u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; int ib_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", - ia_handle, pz_handle); + ia, pz); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error1; } - if (NULL == pz_handle) { + if (NULL == pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto error1; } - ia = (struct dapl_ia *)ia_handle; + dapl_ia = (struct dapl_ia *)ia; - pz = dapl_pz_alloc(ia); - if (NULL == pz) { + dapl_pz = dapl_pz_alloc(dapl_ia); + if (!dapl_pz) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - pz->pd = ib_alloc_pd(ia->hca_ptr->ib_hca_handle); - if (IS_ERR(pz->pd)) { - ib_status = PTR_ERR(pz->pd); + dapl_pz->pd = ib_alloc_pd(dapl_ia->hca_ptr->ib_hca_handle); + if (IS_ERR(dapl_pz->pd)) { + ib_status = PTR_ERR(dapl_pz->pd); dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_alloc_pd failed: %X\n", ib_status); status = dapl_ib_status_convert(ib_status); goto error2; } - *pz_handle = pz; + *pz = (struct dat_pz *)dapl_pz; return DAT_SUCCESS; error2: - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); error1: - *pz_handle = NULL; + *pz = NULL; return status; } -u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) +u32 dapl_pz_free(struct dat_pz *pz) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; int ib_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - if (0 != atomic_read(&pz->pz_ref_count)) { + if (0 != atomic_read(&dapl_pz->pz_ref_count)) { status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_PZ_IN_USE); goto error; } - ib_status = ib_dealloc_pd(pz->pd); + ib_status = ib_dealloc_pd(dapl_pz->pd); if (ib_status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_dealloc_pd failed: %X\n", ib_status); @@ -151,22 +144,22 @@ u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle goto error; } - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); return DAT_SUCCESS; error: return status; } -u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) +u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_param) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query(%p, %x, %p)\n", - pz_handle, pz_param); + pz, pz_param); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } @@ -175,8 +168,8 @@ u32 dapl_pz_query(DAT_PZ_HANDLE pz_handl goto error; } - pz = (struct dapl_pz *)pz_handle; - pz_param->ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; + dapl_pz = (struct dapl_pz *)pz; + pz_param->ia = (struct dat_ia *)dapl_pz->common.owner_ia; return DAT_SUCCESS; error: Index: linux-kernel/dat-provider/dapl_cr_handoff.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_handoff.c (revision 2563) +++ linux-kernel/dat-provider/dapl_cr_handoff.c (working copy) @@ -37,7 +37,7 @@ * Hand the connection request to another Sevice pont specified by the * Connectin Qualifier. */ -u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) +u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL cr_handoff) { /* handoff */ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); } Index: linux-kernel/dat-provider/dapl_lmr.c =================================================================== --- linux-kernel/dat-provider/dapl_lmr.c (revision 2563) +++ linux-kernel/dat-provider/dapl_lmr.c (working copy) @@ -36,7 +36,7 @@ static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_desc, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags mem_priv) { struct dapl_lmr *lmr; @@ -49,27 +49,22 @@ static struct dapl_lmr *dapl_lmr_alloc(s /* zero the structure */ memset(lmr, 0, sizeof *lmr); - /* - * initialize the header - */ - lmr->header.provider = ia->header.provider; - lmr->header.magic = DAPL_MAGIC_LMR; - lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; - lmr->header.owner_ia = ia; - lmr->header.user_context.as_64 = 0; - lmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&lmr->header.ia_list_entry); + lmr->lmr.provider = ia->ia.provider; + lmr->common.owner_ia = ia; + lmr->common.user_context.as_64 = 0; + lmr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&lmr->common.ia_list_entry); dapl_ia_link_lmr(ia, lmr); - spin_lock_init(&lmr->header.lock); + spin_lock_init(&lmr->common.lock); /* * initialize the body */ - lmr->param.ia_handle = (DAT_IA_HANDLE) ia; + lmr->param.ia = (struct dat_ia *)ia; lmr->param.mem_type = mem_type; lmr->param.region_desc = region_desc; lmr->param.length = length; - lmr->param.pz_handle = pz_handle; + lmr->param.pz = pz; lmr->param.mem_priv = mem_priv; atomic_set(&lmr->lmr_ref_count, 0); @@ -78,9 +73,7 @@ static struct dapl_lmr *dapl_lmr_alloc(s static void dapl_lmr_dealloc(struct dapl_lmr *lmr) { - /* reset magic to prevent reuse */ - lmr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); + dapl_ia_unlink_lmr(lmr->common.owner_ia, lmr); kfree(lmr); } @@ -88,49 +81,50 @@ static void dapl_lmr_dealloc(struct dapl static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, u64 length, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; reg_desc.for_va = virt_addr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, + (struct dat_pz *)pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register(ia, lmr, virt_addr, length, privileges); + status = dapl_ib_mr_register(ia, new_lmr, virt_addr, length, + privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64)(unsigned long) virt_addr; if (registered_length) @@ -139,9 +133,9 @@ static inline u32 dapl_lmr_create_virtua return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; @@ -151,26 +145,26 @@ static inline u32 dapl_lmr_create_physic DAT_REGION_DESCRIPTION phys_addr, u64 page_count, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; u64 *array = phys_addr.for_array; u32 status; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, page_count, - (DAT_PZ_HANDLE) pz, privileges); + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, + page_count, (struct dat_pz *) pz, privileges); - if (NULL == lmr) { + if (NULL == new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, + status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr.for_array, page_count, privileges); if (DAT_SUCCESS != status) @@ -178,23 +172,23 @@ static inline u32 dapl_lmr_create_physic /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = array[0]; if (registered_length) @@ -203,9 +197,9 @@ static inline u32 dapl_lmr_create_physic return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } @@ -214,13 +208,13 @@ static inline u32 dapl_lmr_create_lmr(st struct dapl_lmr *original_lmr, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; @@ -230,39 +224,39 @@ static inline u32 dapl_lmr_create_lmr(st if (status != DAT_SUCCESS) goto error1; - reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; + reg_desc.for_lmr = (struct dat_lmr *) original_lmr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, + (struct dat_pz *) pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_shared(ia, lmr, privileges); + status = dapl_ib_mr_register_shared(ia, new_lmr, privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64) (unsigned long) original_lmr->param.region_desc.for_va; @@ -272,73 +266,72 @@ static inline u32 dapl_lmr_create_lmr(st return DAT_SUCCESS; error3: - dapl_ib_mr_deregister(lmr); + dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } -u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags privileges, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - u64 *registered_length, u64 *registered_address) + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, + u64 *registered_address) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", - ia_handle, mem_type); + ia, mem_type); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - ia = (struct dapl_ia *)ia_handle; - pz = (struct dapl_pz *)pz_handle; + dapl_ia = (struct dapl_ia *)ia; + dapl_pz = (struct dapl_pz *)pz; switch (mem_type) { case DAT_MEM_TYPE_VIRTUAL: - status = dapl_lmr_create_virtual(ia, region_description.for_va, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_virtual(dapl_ia, + region_description.for_va, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_PHYSICAL: - status = dapl_lmr_create_physical(ia, region_description, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_physical(dapl_ia, region_description, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_LMR: { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; - if (DAPL_BAD_HANDLE(region_description.for_lmr_handle, - DAPL_MAGIC_LMR)) { + if (!region_description.for_lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto bail; } - lmr = (struct dapl_lmr *)region_description.for_lmr_handle; + dapl_lmr = (struct dapl_lmr *)region_description.for_lmr; - status = dapl_lmr_create_lmr(ia, lmr, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_lmr(dapl_ia, dapl_lmr, dapl_pz, + privileges, lmr, lmr_context, rmr_context, registered_length, registered_address); break; @@ -357,45 +350,45 @@ bail: return status; } -u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) +u32 dapl_lmr_free(struct dat_lmr *lmr) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } - lmr = (struct dapl_lmr *)lmr_handle; + dapl_lmr = (struct dapl_lmr *)lmr; - switch (lmr->param.mem_type) { + switch (dapl_lmr->param.mem_type) { case DAT_MEM_TYPE_PHYSICAL: case DAT_MEM_TYPE_VIRTUAL: case DAT_MEM_TYPE_LMR: { struct dapl_pz *pz; - if (0 != atomic_read(&lmr->lmr_ref_count)) + if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) return DAT_INVALID_STATE; status = dapl_hash_remove( - lmr->header.owner_ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + dapl_lmr->common.owner_ia->hca_ptr->lmr_hash_table, + dapl_lmr->param.lmr_context, NULL); if (status != DAT_SUCCESS) goto error; - status = dapl_ib_mr_deregister(lmr); + status = dapl_ib_mr_deregister(dapl_lmr); if (status == DAT_SUCCESS) { - pz = (struct dapl_pz *)lmr->param.pz_handle; + pz = (struct dapl_pz *)dapl_lmr->param.pz; atomic_dec(&pz->pz_ref_count); - dapl_lmr_dealloc(lmr); - } else /* failure; put lmr back in hash table */ - dapl_hash_insert(lmr->header.owner_ia-> + dapl_lmr_dealloc(dapl_lmr); + } else /* failure; put dapl_lmr back in hash table */ + dapl_hash_insert(dapl_lmr->common.owner_ia-> hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + dapl_lmr->param.lmr_context, dapl_lmr); break; } @@ -412,15 +405,15 @@ error: return status; } -u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) +u32 dapl_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *lmr_param) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", - lmr_handle, lmr_param); + lmr, lmr_param); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } @@ -429,8 +422,8 @@ u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_ha goto error; } - lmr = (struct dapl_lmr *)lmr_handle; - memcpy(lmr_param, &lmr->param, sizeof *lmr_param); + dapl_lmr = (struct dapl_lmr *)lmr; + memcpy(lmr_param, &dapl_lmr->param, sizeof *lmr_param); status = DAT_SUCCESS; error: return status; @@ -441,7 +434,7 @@ error: * will sync the cache with memory. */ -u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -449,11 +442,11 @@ u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDL u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_read(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } @@ -462,7 +455,7 @@ error: return status; } -u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -470,11 +463,11 @@ u32 dapl_lmr_sync_rdma_write(DAT_IA_HAND u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_write(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } Index: linux-kernel/dat-provider/dapl_get_handle_type.c =================================================================== --- linux-kernel/dat-provider/dapl_get_handle_type.c (revision 2563) +++ linux-kernel/dat-provider/dapl_get_handle_type.c (working copy) @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_get_handle_type - * - * Gets the handle type for the given dat_handle - * - * Input: - * dat_handle - * - * Output: - * handle_type - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ - -u32 dapl_get_handle_type(DAT_HANDLE dat_handle, enum dat_handle_type *type) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - *type = header->handle_type; - -bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_get_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_get_consumer_context.c (revision 2563) +++ linux-kernel/dat-provider/dapl_get_consumer_context.c (working copy) @@ -1,80 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_get_consumer_context - * - * DAPL Requirements Version xxx, 6.2.2.2 - * - * Gets the consumer context from the specified dat_object - * - * Input: - * dat_handle - * - * Output: - * context - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ -u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, union dat_context *context) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - if (context == NULL || ((unsigned long)(header) & 3)) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - *context = header->user_context; - - bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_rmr.c =================================================================== --- linux-kernel/dat-provider/dapl_rmr.c (revision 2563) +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) @@ -61,24 +61,19 @@ static struct dapl_rmr *dapl_rmr_alloc(s /* zero the structure */ memset(rmr, 0, sizeof *rmr); - /* - * initialize the header - */ - rmr->header.provider = pz->header.provider; - rmr->header.magic = DAPL_MAGIC_RMR; - rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; - rmr->header.owner_ia = pz->header.owner_ia; - rmr->header.user_context.as_64 = 0; - rmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&rmr->header.ia_list_entry); - dapl_ia_link_rmr(rmr->header.owner_ia, rmr); - spin_lock_init(&rmr->header.lock); + rmr->rmr.provider = pz->pz.provider; + rmr->common.owner_ia = pz->common.owner_ia; + rmr->common.user_context.as_64 = 0; + rmr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&rmr->common.ia_list_entry); + dapl_ia_link_rmr(rmr->common.owner_ia, rmr); + spin_lock_init(&rmr->common.lock); /* * initialize the body */ - rmr->param.ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; - rmr->param.pz_handle = (DAT_PZ_HANDLE)pz; + rmr->param.ia = (struct dat_ia *)pz->common.owner_ia; + rmr->param.pz = (struct dat_pz *)pz; rmr->param.lmr_triplet.lmr_context = 0; rmr->param.lmr_triplet.pad = 0; rmr->param.lmr_triplet.virtual_address = 0; @@ -93,11 +88,8 @@ static struct dapl_rmr *dapl_rmr_alloc(s static void dapl_rmr_dealloc(struct dapl_rmr *rmr) { - /* reset magic to prevent reuse */ - rmr->header.magic = DAPL_MAGIC_INVALID; - - dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); - /* no need to destroy rmr->header.lock */ + dapl_ia_unlink_rmr(rmr->common.owner_ia, rmr); + /* no need to destroy rmr->common.lock */ kfree(rmr); } @@ -115,7 +107,7 @@ static inline u32 dapl_rmr_bind_fuse(str u32 status; boolean_t is_signaled; - status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, + status = dapl_hash_search(rmr->common.owner_ia->hca_ptr->lmr_hash_table, lmr_triplet->lmr_context, (DAPL_HASH_DATA *) &lmr); if (DAT_SUCCESS != status) { @@ -150,8 +142,8 @@ static inline u32 dapl_rmr_bind_fuse(str } /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ - if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || - (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { + if ((ep_ptr->param.pz != lmr->param.pz) || + (ep_ptr->param.pz != rmr->param.pz)) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -240,7 +232,7 @@ static inline u32 dapl_rmr_bind_unfuse(s } /* If the RMR and EP are not in the same PZ, there is an error */ - if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { + if (ep_ptr->param.pz != rmr->param.pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail1; } @@ -306,9 +298,9 @@ bail1: * Input: * Output: */ -u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, const struct dat_lmr_triplet *lmr_triplet, - enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, + enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) @@ -316,14 +308,14 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han struct dapl_rmr *rmr; struct dapl_ep *ep_ptr; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) + if (!rmr_handle) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) + if (!ep) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); rmr = (struct dapl_rmr *)rmr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* if the rmr should be bound */ if (0 != lmr_triplet->segment_length) @@ -341,7 +333,7 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * Create a remote memory region for the specified protection zone * * Input: - * pz_handle + * pz * * Output: * rmr_handle @@ -351,20 +343,20 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) +u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr) { - struct dapl_pz *pz; - struct dapl_rmr *rmr; + struct dapl_pz *dapl_pz; + struct dapl_rmr *dapl_rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - rmr = dapl_rmr_alloc(pz); + dapl_rmr = dapl_rmr_alloc(dapl_pz); if (rmr == NULL) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, @@ -372,18 +364,18 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han goto bail; } - status = dapl_ib_mw_alloc(rmr); + status = dapl_ib_mw_alloc(dapl_rmr); if (status != DAT_SUCCESS) { - dapl_rmr_dealloc(rmr); + dapl_rmr_dealloc(dapl_rmr); status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY_REGION); goto bail; } - atomic_inc(&pz->pz_ref_count); + atomic_inc(&dapl_pz->pz_ref_count); - *rmr_handle = rmr; + *rmr = (struct dat_rmr *)dapl_rmr; bail: return status; @@ -404,12 +396,12 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) +u32 dapl_rmr_free(struct dat_rmr *rmr_handle) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } @@ -438,12 +430,12 @@ bail: return status; } -u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) +u32 dapl_rmr_query(struct dat_rmr *rmr_handle, struct dat_rmr_param *rmr_param) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } Index: linux-kernel/dat-provider/dapl_cr_util.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_util.c (revision 2563) +++ linux-kernel/dat-provider/dapl_cr_util.c (working copy) @@ -49,17 +49,12 @@ struct dapl_cr *dapl_cr_alloc(struct dap /* zero the structure */ memset(cr_ptr, 0, sizeof *cr_ptr); - /* - * initialize the header - */ - cr_ptr->header.provider = ia_ptr->header.provider; - cr_ptr->header.magic = DAPL_MAGIC_CR; - cr_ptr->header.handle_type = DAT_HANDLE_TYPE_CR; - cr_ptr->header.owner_ia = ia_ptr; - cr_ptr->header.user_context.as_64 = 0; - cr_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&cr_ptr->header.ia_list_entry); - spin_lock_init(&cr_ptr->header.lock); + cr_ptr->cr.provider = ia_ptr->ia.provider; + cr_ptr->common.owner_ia = ia_ptr; + cr_ptr->common.user_context.as_64 = 0; + cr_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&cr_ptr->common.ia_list_entry); + spin_lock_init(&cr_ptr->common.lock); return cr_ptr; } @@ -69,9 +64,5 @@ struct dapl_cr *dapl_cr_alloc(struct dap */ void dapl_cr_free(struct dapl_cr *cr_ptr) { - dapl_os_assert(cr_ptr->header.magic == DAPL_MAGIC_CR || - cr_ptr->header.magic == DAPL_MAGIC_CR_DESTROYED); - /* reset magic to prevent reuse */ - cr_ptr->header.magic = DAPL_MAGIC_INVALID; kfree(cr_ptr); } Index: linux-kernel/dat-provider/dapl_psp.c =================================================================== --- linux-kernel/dat-provider/dapl_psp.c (revision 2563) +++ linux-kernel/dat-provider/dapl_psp.c (working copy) @@ -45,13 +45,13 @@ * the user. * * Input: - * ia_handle - * evd_handle + * ia + * evd * psp_flags * * Output: * conn_qual - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -61,9 +61,9 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ struct dapl_ia *ia_ptr; @@ -72,19 +72,19 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia u32 status = DAT_SUCCESS; int i; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } @@ -93,7 +93,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -117,10 +117,10 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; - sp_ptr->evd_handle = evd_handle; + sp_ptr->ia = ia; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -153,7 +153,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia if (status != DAT_SUCCESS) { atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -168,7 +168,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia * Return handle to the user */ *conn_qual = sp_ptr->conn_qual; - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -183,13 +183,13 @@ bail: * in a notification event. * * Input: - * ia_handle + * ia * conn_qual - * evd_handle + * evd * psp_flags * * Output: - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -198,9 +198,9 @@ bail: * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -208,24 +208,24 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -261,11 +261,11 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -298,7 +298,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han * wrong! Decrements the EVD refcount & release it. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -313,7 +313,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -325,7 +325,7 @@ bail: * Destroy a specific instance of a Service Point. * * Input: - * psp_handle + * psp * * Output: * none @@ -334,26 +334,25 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) +u32 dapl_psp_free(struct dat_sp *psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; DAPL_SP_STATE save_state; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if it has been established * and there are no current connections in progress. @@ -361,18 +360,18 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * container until the last connection is disconnected, after * which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); sp_ptr->listening = FALSE; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -389,8 +388,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han sp_ptr->cr_list_count == 0) { save_state = sp_ptr->state; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -407,7 +406,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free: PSP PENDING\n"); } @@ -416,13 +416,12 @@ bail: return status; } -u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) +u32 dapl_psp_query(struct dat_sp *psp, struct dat_psp_param *psp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || - ((struct dapl_sp *)psp_handle)->listening != TRUE) { + if (!psp || ((struct dapl_sp *)psp)->listening != TRUE) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } @@ -432,11 +431,11 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha goto bail; } - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; - psp_param->ia_handle = sp_ptr->ia_handle; + psp_param->ia = sp_ptr->ia; psp_param->conn_qual = sp_ptr->conn_qual; - psp_param->evd_handle = sp_ptr->evd_handle; + psp_param->evd = sp_ptr->evd; psp_param->psp_flags = sp_ptr->psp_flags; status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_rsp.c =================================================================== --- linux-kernel/dat-provider/dapl_rsp.c (revision 2563) +++ linux-kernel/dat-provider/dapl_rsp.c (working copy) @@ -44,13 +44,13 @@ * event * * Input: - * ia_handle + * ia * conn_qual - * ep_handle - * evd_handle + * ep + * evd * * Output: - * rsp_handle + * rsp * * Returns: * DAT_SUCCESS @@ -59,9 +59,9 @@ * DAT_INVALID_STATE * DAT_CONN_QUAL_IN_USE */ -u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle) +u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -70,39 +70,39 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free conn_qual: %x EP: %p\n", - conn_qual, ep_handle); + conn_qual, ep); - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (rsp_handle == NULL) { + if (rsp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -126,11 +126,11 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Fill out the RSP args */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = 0; - sp_ptr->ep_handle = ep_handle; + sp_ptr->ep = ep; /* * Take a reference on the EVD handle @@ -162,7 +162,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han * the state to FREE, so we know the call failed. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; sp_ptr->state = DAPL_SP_STATE_FREE; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -178,7 +178,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *rsp_handle = (DAT_RSP_HANDLE)sp_ptr; + *rsp = (struct dat_sp *)sp_ptr; bail: return status; @@ -190,7 +190,7 @@ bail: * Destroy a specific instance of a Reserved Service Point. * * Input: - * rsp_handle + * rsp * * Output: * none @@ -199,50 +199,49 @@ bail: * DAT_SUCCESS * DAT_INVALID_HANDLE */ -u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) +u32 dapl_rsp_free(struct dat_sp *rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp_handle); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_RSP)) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp); + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if there are no connections. If * we defer removing the sp it becomes something of a zombie * container until disconnection, after which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * Make sure we don't leave a dangling EP. If the state is still * RESERVED then the RSP still owns it. */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -258,8 +257,8 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han /* This RSP has never been used. Clean it up */ sp_ptr->listening = FALSE; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -275,20 +274,20 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); } bail: return status; } -u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) +u32 dapl_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *rsp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { + if (!rsp) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } @@ -298,15 +297,15 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha goto bail; } - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Fill in the RSP params */ - rsp_param->ia_handle = sp_ptr->ia_handle; + rsp_param->ia = sp_ptr->ia; rsp_param->conn_qual = sp_ptr->conn_qual; - rsp_param->evd_handle = sp_ptr->evd_handle; - rsp_param->ep_handle = sp_ptr->ep_handle; + rsp_param->evd = sp_ptr->evd; + rsp_param->ep = sp_ptr->ep; status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2563) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -43,11 +43,6 @@ static void dapl_ep_dealloc(struct dapl_ep *ep_ptr) { - dapl_os_assert(ep_ptr->header.magic == DAPL_MAGIC_EP); - - /* reset magic to prevent reuse */ - ep_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_cb_free(&ep_ptr->req_buffer); dapl_cb_free(&ep_ptr->recv_buffer); @@ -73,18 +68,12 @@ struct dapl_ep *dapl_ep_alloc(struct dap /* zero the structure */ memset(ep_ptr, 0, sizeof *ep_ptr + sizeof (struct sockaddr)); - /* - * initialize the header - */ - ep_ptr->header.provider = ia_ptr->header.provider; - ep_ptr->header.magic = DAPL_MAGIC_EP; - ep_ptr->header.handle_type = DAT_HANDLE_TYPE_EP; - ep_ptr->header.owner_ia = ia_ptr; - ep_ptr->header.user_context.as_64 = 0; - ep_ptr->header.user_context.as_ptr = NULL; - - dapl_llist_init_entry(&ep_ptr->header.ia_list_entry); - spin_lock_init(&ep_ptr->header.lock); + ep_ptr->ep.provider = ia_ptr->ia.provider; + ep_ptr->common.owner_ia = ia_ptr; + ep_ptr->common.user_context.as_64 = 0; + ep_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&ep_ptr->common.ia_list_entry); + spin_lock_init(&ep_ptr->common.lock); /* * Initialize the body @@ -105,7 +94,7 @@ struct dapl_ep *dapl_ep_alloc(struct dap ep_ptr->qp = NULL; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - ep_ptr->cm_handle = NULL; + ep_ptr->cm = NULL; atomic_set(&ep_ptr->req_count, 0); atomic_set(&ep_ptr->recv_count, 0); @@ -165,7 +154,7 @@ static u32 dapl_ep_check_recv_completion return DAT_SUCCESS; } -static u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, +static u32 dapl_ep_post_send_req(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -177,13 +166,13 @@ static u32 dapl_ep_post_send_req(DAT_EP_ struct dapl_cookie *cookie; u32 dat_status; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Synchronization ok since this buffer is only used for send @@ -281,11 +270,11 @@ enum dat_return_subtype dapl_ep_state_su return dat_status; } -u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) +u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + const struct dat_ep_attr *ep_attr, struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -294,18 +283,18 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create (%p, %p, %p, %p, %p, %p, %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -317,44 +306,34 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != DAT_HANDLE_NULL && - DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -370,12 +349,12 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -413,25 +392,25 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (recv_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_recv_dtos = 0; - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (request_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_request_dtos = 0; /* @@ -440,9 +419,9 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -450,7 +429,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -460,36 +439,36 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free - * evd_handles: + * pz: dat_pz_free + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = (struct dat_ep *)ep_ptr; bail: return dat_status; } -u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +u32 dapl_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *ep_attr, - DAT_EP_HANDLE *ep_handle) + struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -498,18 +477,18 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create_with_srq (%p, %p, %p, %p, %p, %p, %p %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, srq_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, srq, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -521,35 +500,27 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)-> + evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); @@ -560,13 +531,13 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * Verify the SRQ handle. It is an error to invoke this call with * a NULL handle */ - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -582,12 +553,12 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -636,26 +607,26 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (recv_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_recv_dtos = 0; } - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (request_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_request_dtos = 0; } @@ -665,9 +636,9 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -675,7 +646,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -685,24 +656,24 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free, - * evd_handles: + * pz: dat_pz_free, + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = ep_ptr; #endif /* notdef */ @@ -710,7 +681,7 @@ bail: return dat_status; } -u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_free(struct dat_ep *ep) { struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; @@ -718,15 +689,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle enum ib_qp_state save_qp_state; u32 dat_status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; param = &ep_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -744,7 +715,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle goto bail; } - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* If we are connected, issue a disconnect. If we are in the * disconnect_pending state, disconnect with the ABRUPT flag @@ -754,14 +725,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle /* * Invoke ep_disconnect to clean up outstanding connections */ - (void)dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + (void)dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); dapl_os_assert(ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_UNCONNECTED); /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); @@ -771,24 +743,24 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle * to set handles to NULL. Set handles to NULL so this routine * is idempotent. */ - if (param->pz_handle != NULL) { - atomic_dec(&((struct dapl_pz *)param->pz_handle)->pz_ref_count); - param->pz_handle = NULL; + if (param->pz != NULL) { + atomic_dec(&((struct dapl_pz *)param->pz)->pz_ref_count); + param->pz = NULL; } - if (param->recv_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->recv_evd_handle)-> + if (param->recv_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->recv_evd)-> evd_ref_count); - param->recv_evd_handle = NULL; + param->recv_evd = NULL; } - if (param->request_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->request_evd_handle)-> + if (param->request_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->request_evd)-> evd_ref_count); - param->request_evd_handle = NULL; + param->request_evd = NULL; } - if (param->connect_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->connect_evd_handle)-> + if (param->connect_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->connect_evd)-> evd_ref_count); - param->connect_evd_handle = NULL; + param->connect_evd = NULL; } /* @@ -804,7 +776,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ save_qp_state = ep_ptr->qp_state; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Free the QP. If the EP has never been used, the QP is invalid */ if (save_qp_state != DAPL_QP_STATE_UNATTACHED) { @@ -828,7 +800,7 @@ bail: } -u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, @@ -839,7 +811,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_connect(%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", - ep_handle, + ep, remote_ia_address->sa_data[2], remote_ia_address->sa_data[3], remote_ia_address->sa_data[4], @@ -847,19 +819,19 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han remote_conn_qual, timeout, private_data_size, private_data, qos, connect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state. The connection handle must be good * at this point. */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD)) { + if (!ep_ptr->param.connect_evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; @@ -889,9 +861,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * * Assign the QP and release the lock. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ)) - { + if (ep_ptr->param.pz == NULL) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); @@ -899,7 +869,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } alloc_ep = *ep_ptr; - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, &alloc_ep, ep_ptr); if (dat_status != DAT_SUCCESS) { dat_status = @@ -908,15 +878,15 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han goto bail; } - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * PZ shouldn't have changed since we're only racing with * dapl_cr_accept() */ if (ep_ptr->qp_state != DAPL_QP_STATE_UNATTACHED) { /* Bail, cleaning up. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = dapl_ib_qp_free(&alloc_ep); if (dat_status != DAT_SUCCESS) { dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -932,42 +902,30 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->qp = alloc_ep.qp; ep_ptr->qp_state = alloc_ep.qp_state; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); } /* * We do state checks and transitions under lock. * The only code we're racing against is dapl_cr_accept. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required - */ - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL - && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required + */ + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); goto bail; @@ -977,12 +935,12 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * we need to attach one now. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_TEP); @@ -991,8 +949,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1003,8 +961,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han /* * At this point we only support one QOS level */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_MODEL_NOT_SUPPORTED, 0); goto bail; } @@ -1018,7 +976,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * At this point we're committed, and done with the endpoint * except for the connect, so we can drop the lock. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Copy the connection qualifiers */ memcpy(ep_ptr->param.remote_ia_address_ptr, remote_ia_address, @@ -1026,7 +984,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->param.remote_port_qual = remote_conn_qual; memcpy(ep_ptr->private_data,private_data,private_data_size); - dat_status = dapl_ib_connect(ep_ptr, remote_ia_address, + dat_status = dapl_ib_connect((struct dat_ep *)ep_ptr, remote_ia_address, remote_conn_qual, timeout, private_data_size, ep_ptr->private_data); @@ -1043,10 +1001,10 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han if (dat_status == DAT_ERROR(DAT_INTERNAL_ERROR, 1)) { dapl_evd_post_connection_event((struct dapl_evd *) ep_ptr->param. - connect_evd_handle, + connect_evd, DAT_CONNECTION_EVENT_UNREACHABLE, - (DAT_HANDLE) ep_ptr, 0, - NULL); + (struct dat_ep *)ep_ptr, + 0, NULL); dat_status = DAT_SUCCESS; } } @@ -1058,7 +1016,7 @@ bail: return dat_status; } -u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, +u32 dapl_ep_dup_connect(struct dat_ep *ep, struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { @@ -1067,13 +1025,13 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep DAT_CONN_QUAL remote_conn_qual; u32 dat_status; - ep_dup_ptr = (struct dapl_ep *)ep_dup_handle; + ep_dup_ptr = (struct dapl_ep *)ep_dup; /* * Verify the dup handle, which must be connected. All other * parameters will be verified by dapl_ep_connect */ - if (DAPL_BAD_HANDLE(ep_dup_handle, DAPL_MAGIC_EP)) { + if (!ep_dup) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1089,10 +1047,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep * there is a problem. Do this under a lock and pull out * the connection parameters for atomicity. */ - spin_lock_irqsave(&ep_dup_ptr->header.lock, ep_dup_ptr->header.flags); + spin_lock_irqsave(&ep_dup_ptr->common.lock, ep_dup_ptr->common.flags); if (ep_dup_ptr->param.ep_state != DAT_EP_STATE_CONNECTED) { - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_dup_ptr)); @@ -1100,10 +1058,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep } remote_ia_address_ptr = ep_dup_ptr->param.remote_ia_address_ptr; remote_conn_qual = ep_dup_ptr->param.remote_port_qual; - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); - dat_status = dapl_ep_connect(ep_handle, + dat_status = dapl_ep_connect(ep, remote_ia_address_ptr, remote_conn_qual, timeout, @@ -1114,7 +1072,7 @@ bail: return dat_status; } -u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags disconnect_flags) { struct dapl_ep *ep_ptr; @@ -1124,14 +1082,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_disconnect (%p, %x)\n", - ep_handle, disconnect_flags); + ep, disconnect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1141,12 +1099,12 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * Do the verification of parameters and the state change * atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Disconnecting a disconnected EP is a no-op. */ if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; goto bail; } @@ -1160,8 +1118,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ ep_ptr->param.ep_state != DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECT_PENDING) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1173,8 +1131,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * If in state DISCONNECT_PENDING then this must be an * ABRUPT disconnect */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } @@ -1187,18 +1145,19 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ */ ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); /* disconnect and make sure we get no callbacks */ (void)dapl_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); /* clean up connection state */ dapl_sp_remove_ep(ep_ptr); - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; dapl_evd_post_connection_event(evd_ptr, DAT_CONNECTION_EVENT_DISCONNECTED, - (DAT_HANDLE) ep_ptr, 0, NULL); + (struct dat_ep *)ep_ptr, 0, + NULL); dat_status = DAT_SUCCESS; goto bail; } @@ -1216,14 +1175,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECT_PENDING; } - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); dat_status = dapl_ib_disconnect(ep_ptr, disconnect_flags); /* * Reacquire the lock and make sure we didn't get a callback * that cleaned up. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (disconnect_flags == DAT_CLOSE_ABRUPT_FLAG && ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { /* @@ -1234,8 +1193,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * on the way out. */ cr_ptr = ep_ptr->cr_ptr; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); if (cr_ptr != NULL) { dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, @@ -1246,13 +1205,13 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ DAT_CONNECTION_EVENT_DISCONNECTED, NULL, cr_ptr->sp_ptr); } else { - dapl_evd_connection_callback(ep_ptr->cm_handle, + dapl_evd_connection_callback(ep_ptr->cm, DAT_CONNECTION_EVENT_DISCONNECTED, NULL, ep_ptr); } } else - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, @@ -1261,7 +1220,7 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ return dat_status; } -u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, enum dat_ep_state *ep_state, +u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle) { struct dapl_ep *ep_ptr; @@ -1269,14 +1228,14 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_get_status (%p, %p, %p, %p)\n", - ep_handle, ep_state, in_dto_idle, out_dto_idle); + ep, ep_state, in_dto_idle, out_dto_idle); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1304,15 +1263,15 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ } static inline u32 -dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, +dapl_ep_modify_validate_parameters(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param, struct dapl_ia **ia_ptr, struct dapl_ep **ep_ptr, struct dat_ep_attr *ep_attr_ptr) { - struct dapl_ia *ia; - struct dapl_ep *ep; + struct dapl_ia *dapl_ia; + struct dapl_ep *dapl_ep; struct dat_ep_attr ep_attr; struct dat_ep_attr ep_attr_limit; struct dat_ep_attr ep_attr_request; @@ -1321,14 +1280,14 @@ dapl_ep_modify_validate_parameters(DAT_E *ia_ptr = NULL; *ep_ptr = NULL; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep = (struct dapl_ep *)ep_handle; - ia = ep->header.owner_ia; + dapl_ep = (struct dapl_ep *)ep; + dapl_ia = dapl_ep->common.owner_ia; /* * Verify parameters valid in current EP state @@ -1348,10 +1307,10 @@ dapl_ep_modify_validate_parameters(DAT_E * TENTATIVE_CONNECTION_PENDING (psp PROVIDER allocated EP) */ if ((mask & DAT_EP_FIELD_PZ_HANDLE) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } @@ -1367,41 +1326,21 @@ dapl_ep_modify_validate_parameters(DAT_E DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS | DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV | DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV)) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_RESERVED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_RESERVED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } /* * Validate handles being modified */ - if (mask & DAT_EP_FIELD_PZ_HANDLE) { - if (ep_param->pz_handle != NULL && - DAPL_BAD_HANDLE(ep_param->pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { - if (ep_param->recv_evd_handle != NULL && - (DAPL_BAD_HANDLE(ep_param->recv_evd_handle, DAPL_MAGIC_EVD) - || !((struct dapl_evd *)ep_param->recv_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - - if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) { - if (ep_param->request_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->request_evd_handle, - DAPL_MAGIC_EVD)) { + if (ep_param->recv_evd != NULL && + !((struct dapl_evd *)ep_param->recv_evd)-> + evd_flags & DAT_EVD_DTO_FLAG) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; @@ -1409,10 +1348,8 @@ dapl_ep_modify_validate_parameters(DAT_E } if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) { - if (ep_param->connect_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->connect_evd_handle, - DAPL_MAGIC_EVD) - && !(((struct dapl_evd *)ep_param->connect_evd_handle)-> + if (ep_param->connect_evd != NULL && + !(((struct dapl_evd *)ep_param->connect_evd)-> evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1423,8 +1360,8 @@ dapl_ep_modify_validate_parameters(DAT_E /* * Validate the attributes against the HCA limits */ - ep_attr = ep->param.ep_attr; - ep_attr_limit = ia->hca_ptr->ep_attr; + ep_attr = dapl_ep->param.ep_attr; + ep_attr_limit = dapl_ia->hca_ptr->ep_attr; ep_attr_request = ep_param->ep_attr; if (mask & DAT_EP_FIELD_EP_ATTR_MAX_MESSAGE_SIZE) { @@ -1471,7 +1408,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS) { if (ep_attr_request.max_recv_dtos > ep_attr_limit.max_recv_dtos - || (ep_param->recv_evd_handle == DAT_HANDLE_NULL + || (ep_param->recv_evd == NULL && (ep_attr_request.max_recv_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1484,7 +1421,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS) { if (ep_attr_request.max_request_dtos > ep_attr_limit.max_request_dtos - || (ep_param->request_evd_handle == DAT_HANDLE_NULL + || (ep_param->request_evd == NULL && (ep_attr_request.max_request_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1517,8 +1454,8 @@ dapl_ep_modify_validate_parameters(DAT_E } } - *ia_ptr = ia; - *ep_ptr = ep; + *ia_ptr = dapl_ia; + *ep_ptr = dapl_ep; *ep_attr_ptr = ep_attr; bail: @@ -1534,7 +1471,7 @@ dapl_ep_modify_validate_parameters(DAT_E * the Endpoint. * * Input: - * ep_handle + * ep * ep_args_mask * * Output: @@ -1546,7 +1483,7 @@ dapl_ep_modify_validate_parameters(DAT_E * DAT_INVALID_ATTRIBUTE * DAT_INVALID_STATE */ -u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, +u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param) { struct dapl_ia *ia; @@ -1572,7 +1509,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand boolean_t rqst_iov_used = FALSE; boolean_t recv_iov_used = FALSE; - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep1, &ep_attr1); @@ -1586,16 +1523,16 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep = *ep1; alloc_ep.param.ep_attr = ep_attr1; if (mask & DAT_EP_FIELD_PZ_HANDLE) - alloc_ep.param.pz_handle = ep_param->pz_handle; + alloc_ep.param.pz = ep_param->pz; if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) - alloc_ep.param.recv_evd_handle = ep_param->recv_evd_handle; + alloc_ep.param.recv_evd = ep_param->recv_evd; if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) - alloc_ep.param.request_evd_handle = ep_param->request_evd_handle; + alloc_ep.param.request_evd = ep_param->request_evd; if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) - alloc_ep.param.connect_evd_handle = ep_param->connect_evd_handle; + alloc_ep.param.connect_evd = ep_param->connect_evd; /* * Allocate everything that might be needed. @@ -1648,18 +1585,18 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * There's a lot of work done here, but there should be no * allocation or blocking. */ - spin_lock_irqsave(&ep1->header.lock, ep1->header.flags); + spin_lock_irqsave(&ep1->common.lock, ep1->common.flags); /* * Revalidate parameters; make sure that races haven't * changed anything important. */ - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep2, &ep_attr2); if (DAT_SUCCESS != dat_status) { - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); goto bail; } @@ -1688,10 +1625,10 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * is stuff changed by dapl_cr_accept, and neither PZ nor EVD is in that * list. */ - new_ep.param.pz_handle = alloc_ep.param.pz_handle; - new_ep.param.recv_evd_handle = alloc_ep.param.recv_evd_handle; - new_ep.param.request_evd_handle = alloc_ep.param.request_evd_handle; - new_ep.param.connect_evd_handle = alloc_ep.param.connect_evd_handle; + new_ep.param.pz = alloc_ep.param.pz; + new_ep.param.recv_evd = alloc_ep.param.recv_evd; + new_ep.param.request_evd = alloc_ep.param.request_evd; + new_ep.param.connect_evd = alloc_ep.param.connect_evd; /* Deal with each of the allocation fields. */ if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS @@ -1753,50 +1690,50 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand if (DAPL_QP_STATE_UNATTACHED != new_ep.qp_state) { dat_status = dapl_ib_qp_modify(ia, ep2, &ep_attr2); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep2->header.lock, - ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, + ep2->common.flags); goto bail; } } *ep2 = new_ep; - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); /* * Modify reference counts, incrementing new ones * and then decrementing old ones (so if they're the same * the refcount never drops to zero). */ - tmp_pz = (struct dapl_pz *)new_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)new_ep.param.pz; if (tmp_pz) atomic_inc(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.request_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); /* decreament the old reference counts */ - tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz; if (tmp_pz) atomic_dec(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); @@ -1844,7 +1781,7 @@ bail: return dat_status; } -u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1855,12 +1792,12 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_read (%p, %d, %p, %p, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1875,7 +1812,7 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE return dat_status; } -u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1885,12 +1822,12 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_write (%p, %d, %p, %P, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1905,7 +1842,7 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL return dat_status; } -u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1916,15 +1853,15 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_recv (%p, %d, %p, %P, %x)\n", - ep_handle, num_segments, local_iov, user_cookie.as_64, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *) ep_handle; + ep_ptr = (struct dapl_ep *) ep; /* * Synchronization ok since this buffer is only used for receive @@ -1954,7 +1891,7 @@ bail: return status; } -u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1964,11 +1901,11 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_send (%p, %d, %p, %P, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1982,21 +1919,21 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h return dat_status; } -u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) +u32 dapl_ep_query(struct dat_ep *ep, struct dat_ep_param *ep_param) { struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_query (%p, %p)\n", - ep_handle, ep_param); + ep, ep_param); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } @@ -2012,12 +1949,10 @@ u32 dapl_ep_query(DAT_EP_HANDLE ep_handl * N.B. Just slam all values into the user structure, there * is nothing to be gained by checking for each bit. */ - if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { + if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) /* obtain the remote IP address */ - status = - dapl_ib_cm_remote_addr((DAT_HANDLE) ep_handle, - &ep_ptr->remote_ia_address); - } + status = dapl_ib_cm_remote_addr(ep, &ep_ptr->remote_ia_address); + ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *)&ep_ptr->remote_ia_address; *ep_param = ep_ptr->param; @@ -2026,21 +1961,21 @@ bail: return status; } -u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, int *nbufs_allocate, +u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocate, int *bufs_alloc_span) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", - ep_handle, nbufs_allocate, bufs_alloc_span); + ep, nbufs_allocate, bufs_alloc_span); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2053,17 +1988,17 @@ bail: } -u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_reset(struct dat_ep *ep) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2085,7 +2020,7 @@ bail: return dat_status; } -u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -2093,14 +2028,14 @@ u32 dapl_ep_set_watermark(DAT_EP_HANDLE u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", - ep_handle, soft_high_watermark, hard_high_watermark); + ep, soft_high_watermark, hard_high_watermark); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; Index: linux-kernel/dat-provider/dapl_set_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_set_consumer_context.c (revision 2563) +++ linux-kernel/dat-provider/dapl_set_consumer_context.c (working copy) @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_set_consumer_context - * - * Set a consumer context in the provided dat_handle - * - * Input: - * dat_handle - * context - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - */ -u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, union dat_context context) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - header->user_context = context; - - bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_ia_open.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_open.c (revision 2563) +++ linux-kernel/dat-provider/dapl_ia_open.c (working copy) @@ -45,18 +45,18 @@ * and calls this function. */ u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr) + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr) { u32 dat_status = DAT_SUCCESS; struct dat_provider *provider; struct dapl_hca *hca_ptr = NULL; - struct dapl_ia *ia_ptr = NULL; - struct dapl_evd *evd_ptr; + struct dapl_ia *ia = NULL; + struct dapl_evd *evd; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_open (%s, %d, %p, %p)\n", - name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); + name, async_evd_qlen, async_evd_ptr, ia_ptr); dat_status = dapl_provider_list_search(name, &provider); if (DAT_SUCCESS != dat_status) { @@ -64,25 +64,25 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - /* ia_handle_ptr and async_evd_handle_ptr cannot be NULL */ - if (ia_handle_ptr == NULL) { + /* ia_ptr and async_evd_ptr cannot be NULL */ + if (ia_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } - if (async_evd_handle_ptr == NULL) { + if (async_evd_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; } /* initialize the caller's param */ - *ia_handle_ptr = DAT_HANDLE_NULL; + *ia_ptr = NULL; hca_ptr = (struct dapl_hca *)provider->extension; atomic_inc(&hca_ptr->handle_ref_count); /* Allocate and initialize ia structure */ - ia_ptr = dapl_ia_alloc(provider, hca_ptr); - if (!ia_ptr) { + ia = dapl_ia_alloc(provider, hca_ptr); + if (!ia) { atomic_dec(&hca_ptr->handle_ref_count); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -93,17 +93,16 @@ u32 dapl_ia_open(const char *name, int a * use the one passed in (if non-NULL) or create one */ - evd_ptr = (struct dapl_evd *)*async_evd_handle_ptr; - if (evd_ptr) { - if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || - !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { + evd = (struct dapl_evd *)*async_evd_ptr; + if (evd) { + if (!(evd->evd_flags & DAT_EVD_ASYNC_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - if (evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle != + if (evd->common.owner_ia->hca_ptr->ib_hca_handle != hca_ptr->ib_hca_handle) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, @@ -111,8 +110,8 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - ia_ptr->cleanup_async_error_evd = FALSE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = FALSE; + ia->async_error_evd = evd; } else { /* Verify we have >0 length, and let the provider check the size */ if (async_evd_qlen <= 0) { @@ -120,32 +119,33 @@ u32 dapl_ia_open(const char *name, int a DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } - dat_status = dapl_evd_internal_create(ia_ptr, async_evd_qlen, + dat_status = dapl_evd_internal_create(ia, + async_evd_qlen, DAT_UPCALL_DISABLE, NULL, DAT_EVD_ASYNC_FLAG, - &evd_ptr); + &evd); if (dat_status != DAT_SUCCESS) goto bail; - atomic_inc(&evd_ptr->evd_ref_count); + atomic_inc(&evd->evd_ref_count); /* Register the handlers associated with the async EVD. */ - dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); + dat_status = dapl_ia_setup_callbacks(ia, evd); /* Assign the EVD so it gets cleaned up */ - ia_ptr->cleanup_async_error_evd = TRUE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = TRUE; + ia->async_error_evd = evd; if (dat_status != DAT_SUCCESS) goto bail; } dat_status = DAT_SUCCESS; - *ia_handle_ptr = ia_ptr; - *async_evd_handle_ptr = evd_ptr; + *ia_ptr = (struct dat_ia *)ia; + *async_evd_ptr = (struct dat_evd *)evd; - bail: - if (dat_status != DAT_SUCCESS) { - if (ia_ptr) - dapl_ia_close(ia_ptr, DAT_CLOSE_ABRUPT_FLAG); - } +bail: + if (dat_status != DAT_SUCCESS) + if (ia) + dapl_ia_close((struct dat_ia *)ia, + DAT_CLOSE_ABRUPT_FLAG); dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_ia_open () returns 0x%x\n", dat_status); Index: linux-kernel/dat-provider/dapl_mr_util.c =================================================================== --- linux-kernel/dat-provider/dapl_mr_util.c (revision 2563) +++ linux-kernel/dat-provider/dapl_mr_util.c (working copy) @@ -59,7 +59,7 @@ u64 dapl_mr_get_address(DAT_REGION_DESCR case DAT_MEM_TYPE_VIRTUAL: return (u64) (unsigned long) desc.for_va; case DAT_MEM_TYPE_LMR: - lmr = (struct dapl_lmr *)desc.for_lmr_handle; + lmr = (struct dapl_lmr *)desc.for_lmr; /* Since this function is recoursive we cannot inline it */ return dapl_mr_get_address(lmr->param.region_desc, Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2563) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -51,7 +51,7 @@ static void dapl_evd_upcall_trigger(stru } for (;;) { - status = dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); + status = dapl_evd_dequeue((struct dat_evd *)evd, &event); if (DAT_SUCCESS != status) return; @@ -157,17 +157,12 @@ struct dapl_evd *dapl_evd_alloc(struct d /* zero the structure */ memset(evd, 0, sizeof *evd); - /* - * initialize the header - */ - evd->header.provider = ia->header.provider; - evd->header.magic = DAPL_MAGIC_EVD; - evd->header.handle_type = DAT_HANDLE_TYPE_EVD; - evd->header.owner_ia = ia; - evd->header.user_context.as_64 = 0; - evd->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&evd->header.ia_list_entry); - spin_lock_init(&evd->header.lock); + evd->evd.provider = ia->ia.provider; + evd->common.owner_ia = ia; + evd->common.user_context.as_64 = 0; + evd->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&evd->common.ia_list_entry); + spin_lock_init(&evd->common.lock); /* * Initialize the body @@ -249,7 +244,6 @@ static u32 dapl_evd_dealloc(struct dapl_ u32 status = DAT_SUCCESS; struct dapl_ia *ia; - dapl_os_assert(evd->header.magic == DAPL_MAGIC_EVD); dapl_os_assert(atomic_read(&evd->evd_ref_count) == 0); /* @@ -257,7 +251,7 @@ static u32 dapl_evd_dealloc(struct dapl_ * up from it. */ if (evd->cq != NULL) { - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; status = dapl_ib_cq_free(evd); if (status != DAT_SUCCESS) { @@ -265,12 +259,6 @@ static u32 dapl_evd_dealloc(struct dapl_ } } - /* - * We should now be safe to invalidate the EVD; reset the - * magic to prevent reuse. - */ - evd->header.magic = DAPL_MAGIC_INVALID; - /* If the ring buffer allocation failed, then the dapl_rbuf_destroy */ /* function will detect that the ring buffer's internal data (ex. base */ /* pointer) are invalid and will handle the situation appropriately */ @@ -314,14 +302,14 @@ static struct dat_event *dapl_evd_get_ev struct dat_event *event; if (evd->evd_producer_locking_needed) { - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); } event = (struct dat_event *) dapl_rbuf_remove(&evd->free_event_queue); /* Release the lock if it was taken and the call failed. */ if (!event && evd->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); return event; } @@ -348,8 +336,8 @@ static void dapl_evd_post_event(struct d dapl_os_assert(evd->evd_state == DAPL_EVD_STATE_OPEN); if (evd->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd->header.lock, - evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, + evd->common.flags); dapl_evd_upcall_trigger(evd); } @@ -363,12 +351,12 @@ static void dapl_evd_format_overflow_eve { struct dapl_ia *ia; - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; event->event_number = DAT_ASYNC_ERROR_EVD_OVERFLOW; event->event_data.asynch_error_event_data.dat_handle = - (DAT_HANDLE) ia; + (struct dat_ia *)ia; } /* @@ -409,9 +397,9 @@ static struct dat_event *dapl_evd_get_an event = dapl_evd_get_event(evd); if (!event) dapl_evd_post_overflow_event( - evd->header.owner_ia->async_error_evd, evd); + evd->common.owner_ia->async_error_evd, evd); else { - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; event->event_number = evno; } @@ -419,10 +407,10 @@ static struct dat_event *dapl_evd_get_an } u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp, struct sockaddr *ia_address, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle) + struct dat_cr *cr) { struct dat_event *event; event = dapl_evd_get_and_init_event(evd, DAT_CONNECTION_REQUEST_EVENT); @@ -436,11 +424,11 @@ u32 dapl_evd_post_cr_arrival_event(struc return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event->event_data.cr_arrival_event_data.sp_handle = sp_handle; + event->event_data.cr_arrival_event_data.sp = sp; event->event_data.cr_arrival_event_data.local_ia_address_ptr = ia_address; event->event_data.cr_arrival_event_data.conn_qual = conn_qual; - event->event_data.cr_arrival_event_data.cr_handle = cr_handle; + event->event_data.cr_arrival_event_data.cr = cr; dapl_evd_post_event(evd, event); @@ -449,7 +437,7 @@ u32 dapl_evd_post_cr_arrival_event(struc u32 dapl_evd_post_connection_event(struct dapl_evd *evd, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, int private_data_size, void *private_data) { @@ -466,7 +454,7 @@ u32 dapl_evd_post_connection_event(struc DAT_RESOURCE_MEMORY); } - event->event_data.connect_event_data.ep_handle = ep_handle; + event->event_data.connect_event_data.ep = ep; event->event_data.connect_event_data.private_data_size = private_data_size; event->event_data.connect_event_data.private_data = private_data; @@ -492,7 +480,7 @@ static u32 dapl_evd_post_async_error_eve return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event->event_data.asynch_error_event_data.dat_handle = (DAT_HANDLE) ia; + event->event_data.asynch_error_event_data.dat_handle = ia; dapl_evd_post_event(evd, event); @@ -539,12 +527,8 @@ static void dapl_evd_wc_to_event(struct ep = cookie->ep; dapl_os_assert(NULL != ep); - if (ep->header.magic != DAPL_MAGIC_EP) { - /* ep may have been freed, just return */ - return; - } - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; switch (cookie->type) { case DAPL_COOKIE_TYPE_DTO: @@ -563,7 +547,7 @@ static void dapl_evd_wc_to_event(struct } event->event_number = DAT_DTO_COMPLETION_EVENT; - event_data->ep_handle = cookie->ep; + event_data->ep = (struct dat_ep *)cookie->ep; event_data->user_cookie = cookie->val.dto.cookie; event_data->status = dto_status; @@ -606,7 +590,7 @@ static void dapl_evd_wc_to_event(struct event->event_number = DAT_RMR_BIND_COMPLETION_EVENT; event_data = &event->event_data.rmr_completion_event_data; - event_data->rmr_handle = cookie->val.rmr.rmr; + event_data->rmr = (struct dat_rmr *)cookie->val.rmr.rmr; event_data->user_cookie = cookie->val.rmr.cookie; if (dto_status == DAT_DTO_SUCCESS) { @@ -654,7 +638,7 @@ void dapl_evd_qp_async_error_callback(st u32 status; ep = (struct dapl_ep *)context; - async_evd = (struct dapl_evd *)ep->header.owner_ia->async_error_evd; + async_evd = (struct dapl_evd *)ep->common.owner_ia->async_error_evd; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback: " @@ -679,7 +663,7 @@ void dapl_evd_qp_async_error_callback(st * we are not interested in. */ (void)dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback () returns\n"); @@ -698,11 +682,11 @@ void dapl_evd_cq_async_error_callback(st if (!evd) dapl_os_panic("NULL == context\n"); - async_evd = evd->header.owner_ia->async_error_evd; + async_evd = evd->common.owner_ia->async_error_evd; status = dapl_evd_post_async_error_event(async_evd, DAT_ASYNC_ERROR_EVD_OVERFLOW, - async_evd->header.owner_ia); + async_evd->common.owner_ia); if (status != DAT_SUCCESS) { dapl_os_panic("async EVD overflow\n"); @@ -738,7 +722,7 @@ void dapl_evd_un_async_error_callback(st */ dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_un_async_error_callback () returns\n"); @@ -764,14 +748,14 @@ void dapl_evd_connection_callback(struct "ep: %p\n", cm_handle, event, ep); - evd = (struct dapl_evd *)ep->param.connect_evd_handle; + evd = (struct dapl_evd *)ep->param.connect_evd; private_data_size = 0; /* * All operations effect the EP, so lock it once and unlock * when necessary */ - spin_lock_irqsave(&ep->header.lock, ep->header.flags); + spin_lock_irqsave(&ep->common.lock, ep->common.flags); switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: @@ -784,13 +768,13 @@ void dapl_evd_connection_callback(struct /* If someone pulled the plug on the connection, just * exit */ - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); status = DAT_SUCCESS; break; } ep->param.ep_state = DAT_EP_STATE_CONNECTED; - ep->cm_handle = cm_handle; + ep->cm = cm_handle; if (private_data == NULL) { private_data_size = 0; } else { @@ -798,7 +782,7 @@ void dapl_evd_connection_callback(struct memcpy(ep->private_data, private_data, private_data_size); } - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -811,8 +795,8 @@ void dapl_evd_connection_callback(struct */ ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); break; } @@ -820,7 +804,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -828,7 +812,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -836,7 +820,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -844,14 +828,14 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, FALSE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } case DAT_CONNECTION_REQUEST_EVENT: default: { - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); evd = NULL; dapl_os_assert(0); /* shouldn't happen */ @@ -865,7 +849,7 @@ void dapl_evd_connection_callback(struct */ if (evd != NULL) { status = dapl_evd_post_connection_event(evd, event, - (DAT_HANDLE) ep, + (struct dat_ep *)ep, private_data_size, /* 0 except on CONNECT */ ep->private_data); @@ -877,10 +861,10 @@ void dapl_evd_connection_callback(struct * expect a callback on an ABRUPT disconnect. */ dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); - spin_lock_irqsave(&ep->header.lock, ep->header.flags); + spin_lock_irqsave(&ep->common.lock, ep->common.flags); ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); } } @@ -901,7 +885,6 @@ void dapl_evd_dto_callback(struct ib_cq evd = (struct dapl_evd *)user_context; dapl_os_assert(evd->cq == cq); - dapl_os_assert(evd->header.magic == DAPL_MAGIC_EVD); /* Read once. */ state = *(volatile DAPL_EVD_STATE *)&evd->evd_state; @@ -927,9 +910,9 @@ void dapl_evd_dto_callback(struct ib_cq status = dapl_ib_completion_notify(evd); if (DAT_SUCCESS != status) (void)dapl_evd_post_async_error_event( - evd->header.owner_ia->async_error_evd, + evd->common.owner_ia->async_error_evd, DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR, - evd->header.owner_ia); + evd->common.owner_ia); dapl_evd_upcall_trigger(evd); } @@ -1008,10 +991,10 @@ bail: return status; } -u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int min_qlen, +u32 dapl_evd_kcreate(struct dat_ia *ia_handle, int min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object *upcall, - enum dat_evd_flags flags, DAT_EVD_HANDLE *evd_handle) + enum dat_evd_flags flags, struct dat_evd **evd_handle) { struct dapl_ia *ia; struct dapl_evd *evd; @@ -1025,7 +1008,7 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha evd = NULL; *evd_handle = NULL; - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -1053,17 +1036,17 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha if (status != DAT_SUCCESS) goto bail; - *evd_handle = (DAT_EVD_HANDLE) evd; + *evd_handle = (struct dat_evd *)evd; bail: if (status != DAT_SUCCESS) if (evd) - dapl_evd_free(evd); + dapl_evd_free((struct dat_evd *)evd); return status; } -u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) +u32 dapl_evd_free(struct dat_evd *evd_handle) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; @@ -1072,7 +1055,7 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1083,11 +1066,11 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han goto bail; } - dapl_ia_unlink_evd(evd->header.owner_ia, evd); + dapl_ia_unlink_evd(evd->common.owner_ia, evd); status = dapl_evd_dealloc(evd); if (status != DAT_SUCCESS) { - dapl_ia_link_evd(evd->header.owner_ia, evd); + dapl_ia_link_evd(evd->common.owner_ia, evd); } bail: @@ -1097,7 +1080,7 @@ bail: return status; } -u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) +u32 dapl_evd_kquery(struct dat_evd *evd_handle, struct dat_evd_param *evd_param) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; @@ -1108,11 +1091,11 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h } /* Note: the spec. allows for events to be directed to a NULL EVD */ - /* with handle of type DAT_HANDLE_NULL. See 6.3.1 */ - if (DAT_HANDLE_NULL == evd_handle) { + /* with handle of type NULL. See 6.3.1 */ + if (NULL == evd_handle) { memset(evd_param, 0, sizeof *evd_param); } else { - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1128,7 +1111,7 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h * about reading the state variable atomically when we add * in waitable/unwaitable. */ - evd_param->ia_handle = evd->header.owner_ia; + evd_param->ia = (struct dat_ia *)evd->common.owner_ia; evd_param->evd_qlen = evd->qlen; evd_param->evd_flags = evd->evd_flags; evd_param->upcall_policy = evd->upcall_policy; @@ -1139,7 +1122,7 @@ bail: return status; } -u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +u32 dapl_evd_modify_upcall(struct dat_evd *evd_handle, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object *upcall) { @@ -1150,7 +1133,7 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1162,14 +1145,14 @@ bail: return status; } -u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) +u32 dapl_evd_post_se(struct dat_evd *evd_handle, const struct dat_event *event) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1196,7 +1179,7 @@ u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_ return status; } -u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) +u32 dapl_evd_dequeue(struct dat_evd *evd_handle, struct dat_event *event) { struct dapl_evd *evd; struct dat_event *local_event; @@ -1207,7 +1190,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1222,7 +1205,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ * restricts us from having multiple threads in CQ poll, and the * DAT API allows multiple threads in dat_evd_dequeue() */ - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); /* * Make sure there are no other waiters and the evd is active. @@ -1230,7 +1213,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ */ if (evd->evd_state != DAPL_EVD_STATE_OPEN || evd->catastrophic_overflow) { - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); status = DAT_ERROR(DAT_INVALID_STATE, 0); goto bail; } @@ -1249,7 +1232,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ status = DAT_ERROR(DAT_QUEUE_EMPTY, 0); } - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_dequeue () returns 0x%x\n", status); @@ -1257,7 +1240,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ return status; } -u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) +u32 dapl_evd_resize(struct dat_evd *evd_handle, int evd_qlen) { struct dapl_ia *ia; struct dapl_evd *evd; @@ -1267,13 +1250,13 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", evd_handle, evd_qlen); - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } evd = (struct dapl_evd *)evd_handle; - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; if (evd_qlen == evd->qlen) { status = DAT_SUCCESS; @@ -1285,7 +1268,7 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; } - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); pend_cnt = dapl_rbuf_count(&evd->pending_event_queue); if (pend_cnt > evd_qlen) { @@ -1302,6 +1285,6 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; bail: - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); return status; } Index: linux-kernel/dat-provider/dapl_ia_query.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_query.c (revision 2563) +++ linux-kernel/dat-provider/dapl_ia_query.c (working copy) @@ -38,12 +38,12 @@ * Provide the consumer with Interface Adapter and Provider parameters. * * Input: - * ia_handle + * ia * ia_mask * provider_mask * * Output: - * async_evd_handle + * async_evd * ia_parameters * provider_parameters * @@ -51,7 +51,7 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE *async_evd_handle, +u32 dapl_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -66,21 +66,21 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_query (%p, %p, %p, %p)\n", - ia_handle, - async_evd_handle, + ia, + async_evd, ia_attr, provider_attr); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (NULL != async_evd_handle) { - *async_evd_handle = ia_ptr->async_error_evd; + if (NULL != async_evd) { + *async_evd = (struct dat_evd *)ia_ptr->async_error_evd; } if (NULL != ia_attr) @@ -88,7 +88,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl if (NULL != provider_attr) { strncpy(provider_attr->provider_name, - ia_ptr->header.provider->device_name, + ia_ptr->ia.provider->device_name, DAT_NAME_MAX_LENGTH); provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; Index: linux-kernel/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel/dat-provider/dapl_sp_util.c (revision 2563) +++ linux-kernel/dat-provider/dapl_sp_util.c (working copy) @@ -65,22 +65,16 @@ struct dapl_sp *dapl_sp_alloc(struct dap /* zero the structure */ memset(sp_ptr, 0, sizeof *sp_ptr); - /* - * initialize the header - */ - sp_ptr->header.provider = ia_ptr->header.provider; - if (is_psp) { - sp_ptr->header.magic = DAPL_MAGIC_PSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_PSP; - } else { - sp_ptr->header.magic = DAPL_MAGIC_RSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_RSP; - } - sp_ptr->header.owner_ia = ia_ptr; - sp_ptr->header.user_context.as_64 = 0; - sp_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&sp_ptr->header.ia_list_entry); - spin_lock_init(&sp_ptr->header.lock); + sp_ptr->sp.provider = ia_ptr->ia.provider; + if (is_psp) + sp_ptr->sp.type = DAT_SP_TYPE_PSP; + else + sp_ptr->sp.type = DAT_SP_TYPE_RSP; + sp_ptr->common.owner_ia = ia_ptr; + sp_ptr->common.user_context.as_64 = 0; + sp_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&sp_ptr->common.ia_list_entry); + spin_lock_init(&sp_ptr->common.lock); /* * Initialize the Body (set to NULL above) @@ -107,12 +101,8 @@ struct dapl_sp *dapl_sp_alloc(struct dap */ void dapl_sp_dealloc(struct dapl_sp *sp_ptr) { - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); dapl_os_assert(dapl_llist_is_empty(&sp_ptr->cr_list_head)); - sp_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - kfree(sp_ptr); } @@ -134,11 +124,11 @@ void dapl_sp_dealloc(struct dapl_sp *sp_ */ void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) { - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_llist_add_tail(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry, cr_ptr); + &cr_ptr->common.ia_list_entry, cr_ptr); sp_ptr->cr_list_count++; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, sp_ptr->common.flags); } /* @@ -174,7 +164,7 @@ struct dapl_cr *dapl_sp_search_cr(struct if (cr_ptr->ib_cm_handle == ib_cm_handle) return cr_ptr; - cr_ptr = cr_ptr->header.ia_list_entry.flink->data; + cr_ptr = cr_ptr->common.ia_list_entry.flink->data; } while ((void *)cr_ptr != (void *)sp_ptr->cr_list_head->data); return NULL; @@ -208,7 +198,7 @@ void dapl_sp_remove_cr(struct dapl_sp *s } dapl_llist_remove_entry(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); sp_ptr->cr_list_count--; } @@ -238,13 +228,13 @@ void dapl_sp_remove_ep(struct dapl_ep *e if (cr_ptr != NULL) { sp_ptr = cr_ptr->sp_ptr; - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* Remove the CR from the queue */ dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); Index: linux-kernel/dat-provider/dapl_evd.h =================================================================== --- linux-kernel/dat-provider/dapl_evd.h (revision 2563) +++ linux-kernel/dat-provider/dapl_evd.h (working copy) @@ -60,14 +60,14 @@ extern void dapl_evd_un_async_error_call */ extern u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp_handle, struct sockaddr *ia_address, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle); + struct dat_cr *cr_handle); extern u32 dapl_evd_post_connection_event(struct dapl_evd *evd, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep_handle, int private_data_size, void *private_data); Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2563) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -46,23 +46,6 @@ * * *********************************************************************/ -typedef enum dapl_magic { - /* magic number values for verification & debug */ - DAPL_MAGIC_IA = 0xCafeF00d, - DAPL_MAGIC_EVD = 0xFeedFace, - DAPL_MAGIC_EP = 0xDeadBabe, - DAPL_MAGIC_LMR = 0xBeefCafe, - DAPL_MAGIC_RMR = 0xABadCafe, - DAPL_MAGIC_PZ = 0xDeafBeef, - DAPL_MAGIC_PSP = 0xBeadeD0c, - DAPL_MAGIC_RSP = 0xFab4Feed, - DAPL_MAGIC_SRQ = 0xC001Babe, - DAPL_MAGIC_CR = 0xBe12Cee1, - DAPL_MAGIC_CR_DESTROYED = 0xB12bDead, - DAPL_MAGIC_CNO = 0xDeadF00d, - DAPL_MAGIC_INVALID = 0xFFFFFFFF -} DAPL_MAGIC; - typedef enum dapl_evd_state { DAPL_EVD_STATE_OPEN, DAPL_EVD_STATE_DEAD = 0xDEAD @@ -92,23 +75,6 @@ typedef enum dapl_evd_completion { /********************************************************************* * * - * Macros * - * * - *********************************************************************/ - -/* - * Simple macro to verify a handle is bad. Conditions: - * - pointer is NULL - * - pointer is not word aligned - * - pointer's magic number is wrong - */ -#define DAPL_BAD_HANDLE(h, magicNum) ( \ - ((h) == NULL) || \ - ((unsigned long)(h) & 3) || \ - (((struct dapl_header *)(h))->magic != (magicNum))) - -/********************************************************************* - * * * Typedefs * * * *********************************************************************/ @@ -170,11 +136,8 @@ struct dapl_hca { struct dat_ep_attr ep_attr; }; -/* DAPL Objects always have the following header */ -struct dapl_header { - struct dat_provider *provider; /* required by DAT - must be first */ - DAPL_MAGIC magic; /* magic number for verification */ - enum dat_handle_type handle_type; +/* DAPL Objects always have the following items */ +struct dapl_common { struct dapl_ia *owner_ia; struct dapl_llist_entry ia_list_entry; union dat_context user_context; /* user context - opaque to DAPL */ @@ -182,9 +145,9 @@ struct dapl_header { unsigned long flags; /* saved lock flag values */ }; -/* struct dapl_ia maps to DAT_IA_HANDLE */ struct dapl_ia { - struct dapl_header header; /* WARNING: must be first */ + struct dat_ia ia; + struct dapl_common common; struct dapl_hca *hca_ptr; struct dapl_evd *async_error_evd; boolean_t cleanup_async_error_evd; @@ -200,9 +163,9 @@ struct dapl_ia { struct dapl_llist_entry *srq_list_head; /* SRQ queue */ }; -/* struct dapl_evd maps to DAT_EVD_HANDLE */ struct dapl_evd { - struct dapl_header header; /* WARNING: must be first */ + struct dat_evd evd; + struct dapl_common common; DAPL_EVD_STATE evd_state; enum dat_evd_flags evd_flags; @@ -230,9 +193,9 @@ struct dapl_evd { struct dat_upcall_object upcall; }; -/* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { - struct dapl_header header; + struct dat_ep ep; + struct dapl_common common; /* What the DAT Consumer asked for */ struct dat_ep_param param; @@ -241,7 +204,7 @@ struct dapl_ep { enum ib_qp_state qp_state; /* communications manager handle */ - struct dapl_cm_id *cm_handle; + struct dapl_cm_id *cm; /* store the remote IA address here, reference from the param * struct which only has a pointer, no storage */ @@ -267,33 +230,33 @@ struct dapl_ep { int send_iov_num; }; -/* struct dapl_srq maps to DAT_SRQ_HANDLE */ struct dapl_srq { - struct dapl_header header; /* WARNING: must be first */ + struct dat_srq srq; + struct dapl_common common; struct dat_srq_param param; atomic_t srq_ref_count; struct dapl_cookie_buffer recv_buffer; atomic_t recv_count; }; -/* struct dapl_pz maps to DAT_PZ_HANDLE */ struct dapl_pz { - struct dapl_header header; /* WARNING: must be first */ + struct dat_pz pz; + struct dapl_common common; struct ib_pd *pd; atomic_t pz_ref_count; }; -/* struct dapl_lmr maps to DAT_LMR_HANDLE */ struct dapl_lmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_lmr lmr; + struct dapl_common common; struct dat_lmr_param param; struct ib_mr *mr; atomic_t lmr_ref_count; }; -/* struct dapl_rmr maps to DAT_RMR_HANDLE */ struct dapl_rmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_rmr rmr; + struct dapl_common common; struct dat_rmr_param param; struct dapl_ep *ep; struct dapl_pz *pz; @@ -310,17 +273,17 @@ typedef enum dapl_sp_state { DAPL_SP_STATE_RSP_PENDING } DAPL_SP_STATE; -/* struct dapl_sp maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ struct dapl_sp { - struct dapl_header header; /* WARNING: must be first */ + struct dat_sp sp; + struct dapl_common common; DAPL_SP_STATE state; /* type and queue of the SP */ /* PSP/RSP PARAM fields */ - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; /* maintenance fields */ boolean_t listening; /* PSP is registered & active */ @@ -329,9 +292,9 @@ struct dapl_sp { int cr_list_count; /* count of CRs on queue */ }; -/* struct dapl_cr maps to DAT_CR_HANDLE */ struct dapl_cr { - struct dapl_header header; + struct dat_cr cr; + struct dapl_common common; /* for convenience the data is kept as a struct dat_cr_param. * however, the "local_endpoint" field is always NULL * so this wastes a pointer. This is probably ok to @@ -396,91 +359,88 @@ struct dapl_cookie { */ extern u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr); + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr); -extern u32 dapl_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dapl_ia_close(struct dat_ia *, enum dat_close_flags); -extern u32 dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, +extern u32 dapl_ia_query(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); /* helper functions */ -extern u32 dapl_set_consumer_context(DAT_HANDLE handle, +extern u32 dapl_set_consumer_context(void *handle, union dat_context context); -extern u32 dapl_get_consumer_context(DAT_HANDLE handle, +extern u32 dapl_get_consumer_context(void *handle, union dat_context *context); -extern u32 dapl_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type); - /* CR Functions */ -extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); +extern u32 dapl_cr_query(struct dat_cr *, struct dat_cr_param *); -extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +extern u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data); -extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); +extern u32 dapl_cr_reject(struct dat_cr *cr); -extern u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL handoff); +extern u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff); /* EVD Functions */ -extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, +extern u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, u64 *suggested_alignment); -extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, +extern u32 dapl_evd_kcreate(struct dat_ia *ia, int evd_min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE * evd_handle); + struct dat_evd ** evd); -extern u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_kquery(struct dat_evd *evd, struct dat_evd_param *evd_args); -extern u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall); -extern u32 dapl_evd_enable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_enable(struct dat_evd *evd); -extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_disable(struct dat_evd *evd); -extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen); +extern u32 dapl_evd_resize(struct dat_evd *evd, int evd_qlen); -extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_post_se(struct dat_evd *evd, const struct dat_event *event); -extern u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event); +extern u32 dapl_evd_dequeue(struct dat_evd *evd, struct dat_event *event); -extern u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_free(struct dat_evd *evd); -extern u32 dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_set_unwaitable(struct dat_evd *evd); -extern u32 dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_clear_unwaitable(struct dat_evd *evd); /* EP functions */ -extern u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE in_dto_completion_evd_handle, - DAT_EVD_HANDLE out_dto_completion_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, +extern u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_dto_completion_evd, + struct dat_evd *out_dto_completion_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *ep_parameters, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); +extern u32 dapl_ep_query(struct dat_ep *, struct dat_ep_param *); -extern u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask ep_args_mask, const struct dat_ep_param *ep_args); -extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, @@ -489,161 +449,161 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); -extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, +extern u32 dapl_ep_dup_connect(struct dat_ep *ep, + struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); -extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags close_flags); -extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle); -extern u32 dapl_ep_free(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_free(struct dat_ep *ep); -extern u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_reset(struct dat_ep *ep); -extern u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_ep_create_with_srq(struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq_handle, const struct dat_ep_attr *ep_attributes, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocated, int *bufs_alloc_span); -extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark); /* LMR functions */ -extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +extern u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address); -extern u32 dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); +extern u32 dapl_lmr_query(struct dat_lmr *, struct dat_lmr_param *); -extern u32 dapl_lmr_free(DAT_LMR_HANDLE); +extern u32 dapl_lmr_free(struct dat_lmr *); -extern u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); -extern u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); /* RMR Functions */ -extern u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle); +extern u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr); -extern u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_query(struct dat_rmr *rmr, struct dat_rmr_param *rmr_args); -extern u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_bind(struct dat_rmr *rmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *context); -extern u32 dapl_rmr_free(DAT_RMR_HANDLE); +extern u32 dapl_rmr_free(struct dat_rmr *rmr); /* PSP Functions */ -extern u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, +extern u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, +extern u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); +extern u32 dapl_psp_query(struct dat_sp *, struct dat_psp_param *); -extern u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle); +extern u32 dapl_psp_free(struct dat_sp *psp); /* RSP Functions */ -extern u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle); +extern u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp); -extern u32 dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); +extern u32 dapl_rsp_query(struct dat_sp *, struct dat_rsp_param *); -extern u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle); +extern u32 dapl_rsp_free(struct dat_sp *rsp); /* PZ Functions */ -extern u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle); +extern u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz); -extern u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_args); +extern u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_args); -extern u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle); +extern u32 dapl_pz_free(struct dat_pz *pz); /* SRQ functions */ -extern u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, +extern u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *srq_attr, - DAT_SRQ_HANDLE *srq_handle); + struct dat_srq **srq); -extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); +extern u32 dapl_srq_free(struct dat_srq *srq); -extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +extern u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie); -extern u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param); -extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto); -extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark); +extern u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark); /* * DAPL internal utility function prototypes Index: linux-kernel/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_util.c (revision 2563) +++ linux-kernel/dat-provider/dapl_ia_util.c (working copy) @@ -65,17 +65,12 @@ struct dapl_ia *dapl_ia_alloc(struct dat /* zero the structure */ memset(ia_ptr, 0, sizeof *ia_ptr); - /* - * initialize the header - */ - ia_ptr->header.provider = provider; - ia_ptr->header.magic = DAPL_MAGIC_IA; - ia_ptr->header.handle_type = DAT_HANDLE_TYPE_IA; - ia_ptr->header.owner_ia = ia_ptr; - ia_ptr->header.user_context.as_64 = 0; - ia_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&ia_ptr->header.ia_list_entry); - spin_lock_init(&ia_ptr->header.lock); + ia_ptr->ia.provider = provider; + ia_ptr->common.owner_ia = ia_ptr; + ia_ptr->common.user_context.as_64 = 0; + ia_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&ia_ptr->common.ia_list_entry); + spin_lock_init(&ia_ptr->common.lock); /* * initialize the body @@ -127,9 +122,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rmr_list_head)); while (rmr_ptr != NULL) { next_rmr_ptr = dapl_llist_next_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header. + &rmr_ptr->common. ia_list_entry); - dat_status = dapl_rmr_free(rmr_ptr); + dat_status = dapl_rmr_free((struct dat_rmr *)rmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rmr_free(%p) returns %x\n", @@ -141,9 +136,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rsp_list_head)); while (sp_ptr != NULL) { next_sp_ptr = dapl_llist_next_entry(&ia_ptr->rsp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); - dat_status = dapl_rsp_free(sp_ptr); + dat_status = dapl_rsp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rsp_free(%p) returns %x\n", @@ -155,7 +150,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->ep_list_head)); while (ep_ptr != NULL) { next_ep_ptr = dapl_llist_next_entry(&ia_ptr->ep_list_head, - &ep_ptr->header. + &ep_ptr->common. ia_list_entry); /* * Issue a disconnect if the EP needs it @@ -163,21 +158,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - || ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING - || ep_ptr->param.ep_state == - DAT_EP_STATE_DISCONNECT_PENDING) { - dat_status = - dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + DAT_EP_STATE_ACTIVE_CONNECTION_PENDING || + ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { + dat_status = dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_disconnect(%p) returns %x\n", ep_ptr, dat_status); } /* force the EP into error state to force flush all posted DTOs. */ - (void) dapl_modify_qp_state_to_error(ep_ptr->qp); + (void)dapl_modify_qp_state_to_error(ep_ptr->qp); - dat_status = dapl_ep_free(ep_ptr); + dat_status = dapl_ep_free((struct dat_ep *)ep_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_free(%p) returns %x\n", @@ -189,9 +182,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->lmr_list_head)); while (lmr_ptr != NULL) { next_lmr_ptr = dapl_llist_next_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header. + &lmr_ptr->common. ia_list_entry); - dat_status = dapl_lmr_free(lmr_ptr); + dat_status = dapl_lmr_free((struct dat_lmr *)lmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): lmr_free(%p) returns %x\n", @@ -213,7 +206,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status); next_sp_ptr = dapl_llist_next_entry(&ia_ptr->psp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); /* Remove CR's from this PSP and clean them up */ @@ -222,18 +215,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia while (cr_ptr != NULL) { next_cr_ptr = dapl_llist_next_entry( &sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); /* Remove the CR from the queue & cleanup */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); cr_ptr = next_cr_ptr; } - dat_status = dapl_psp_free(sp_ptr); + dat_status = dapl_psp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): psp_free(%p) returns %x\n", @@ -246,9 +240,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->pz_list_head)); while (pz_ptr != NULL) { next_pz_ptr = dapl_llist_next_entry(&ia_ptr->pz_list_head, - &pz_ptr->header. + &pz_ptr->common. ia_list_entry); - dat_status = dapl_pz_free(pz_ptr); + dat_status = dapl_pz_free((struct dat_pz *)pz_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): pz_free(%p) returns %x\n", @@ -266,11 +260,11 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->evd_list_head)); while (evd_ptr != NULL) { next_evd_ptr = dapl_llist_next_entry(&ia_ptr->evd_list_head, - &evd_ptr->header. + &evd_ptr->common. ia_list_entry); if (evd_ptr != ia_ptr->async_error_evd) { /* it isn't the async EVD; delete it. */ - dat_status = dapl_evd_free(evd_ptr); + dat_status = dapl_evd_free((struct dat_evd *)evd_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): evd_free(%p) returns %x\n", @@ -289,7 +283,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status = dapl_ia_teardown_callbacks(ia_ptr); atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - dat_status = dapl_evd_free(ia_ptr->async_error_evd); + dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != dat_status) dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -404,7 +398,7 @@ u32 dapl_ia_graceful_close(struct dapl_i if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - cur_dat_status = dapl_evd_free(ia_ptr->async_error_evd); + cur_dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; @@ -424,8 +418,6 @@ bail: */ void dapl_ia_free(struct dapl_ia *ia_ptr) { - dapl_os_assert(ia_ptr->header.magic == DAPL_MAGIC_IA); - dapl_os_assert(ia_ptr->async_error_evd == NULL); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->lmr_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rmr_list_head)); @@ -434,12 +426,8 @@ void dapl_ia_free(struct dapl_ia *ia_ptr dapl_os_assert(dapl_llist_is_empty(&ia_ptr->psp_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rsp_list_head)); - /* - * deinitialize the header - */ dapl_hca_unlink_ia(ia_ptr->hca_ptr, ia_ptr); - ia_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - /* no need to destroy ia_ptr->header.lock */ + /* no need to destroy ia_ptr->common.lock */ kfree(ia_ptr); } @@ -449,10 +437,10 @@ void dapl_ia_free(struct dapl_ia *ia_ptr */ void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry, ep_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry, ep_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -460,10 +448,10 @@ void dapl_ia_link_ep(struct dapl_ia *ia_ */ void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -471,10 +459,10 @@ void dapl_ia_unlink_ep(struct dapl_ia *i */ void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry, srq_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry, srq_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -482,10 +470,10 @@ void dapl_ia_link_srq(struct dapl_ia *ia */ void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -493,10 +481,10 @@ void dapl_ia_unlink_srq(struct dapl_ia * */ void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry, lmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry, lmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -504,10 +492,10 @@ void dapl_ia_link_lmr(struct dapl_ia *ia */ void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -515,10 +503,10 @@ void dapl_ia_unlink_lmr(struct dapl_ia * */ void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry, rmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry, rmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -526,10 +514,10 @@ void dapl_ia_link_rmr(struct dapl_ia *ia */ void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -537,10 +525,10 @@ void dapl_ia_unlink_rmr(struct dapl_ia * */ void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry, pz_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry, pz_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -548,10 +536,10 @@ void dapl_ia_link_pz(struct dapl_ia *ia_ */ void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -559,10 +547,10 @@ void dapl_ia_unlink_pz(struct dapl_ia *i */ void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry, evd_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry, evd_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -570,10 +558,10 @@ void dapl_ia_link_evd(struct dapl_ia *ia */ void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -581,10 +569,10 @@ void dapl_ia_unlink_evd(struct dapl_ia * */ void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->psp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -594,17 +582,17 @@ void dapl_ia_unlink_sp(struct dapl_ia *i { struct dapl_llist_entry **list_head; - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_PSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_PSP) { list_head = &ia_ptr->psp_list_head; } else { - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == + DAT_SP_TYPE_RSP); list_head = &ia_ptr->rsp_list_head; } - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); - dapl_llist_remove_entry(list_head, &sp_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); + dapl_llist_remove_entry(list_head, &sp_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -634,7 +622,7 @@ struct dapl_sp *dapl_ia_sp_search(struct else list_head = &ia_ptr->rsp_list_head; - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); sp_ptr = (dapl_llist_is_empty(list_head) ? NULL : dapl_llist_peek_head(list_head)); @@ -644,10 +632,10 @@ struct dapl_sp *dapl_ia_sp_search(struct break; sp_ptr = dapl_llist_next_entry(list_head, - &sp_ptr->header.ia_list_entry); + &sp_ptr->common.ia_list_entry); } - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); return sp_ptr; } @@ -670,10 +658,10 @@ struct dapl_sp *dapl_ia_sp_search(struct */ void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rsp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, Index: linux-kernel/dat-provider/dapl_cr_query.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_query.c (revision 2563) +++ linux-kernel/dat-provider/dapl_cr_query.c (working copy) @@ -34,16 +34,16 @@ #include "dapl.h" #include "dapl_openib_util.h" -u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) +u32 dapl_cr_query(struct dat_cr *cr, struct dat_cr_param *cr_param) { struct dapl_cr *cr_ptr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", - cr_handle, cr_param); + cr, cr_param); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (cr == NULL) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } @@ -53,10 +53,10 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* obtain the remote IP address */ - (void) dapl_ib_cm_remote_addr((DAT_HANDLE) cr_handle, + (void) dapl_ib_cm_remote_addr((void *) cr, &cr_ptr->remote_ia_address); /* since the arguments are easily accessible, ignore the mask */ Index: linux-kernel/dat-provider/dapl_provider.c =================================================================== --- linux-kernel/dat-provider/dapl_provider.c (revision 2563) +++ linux-kernel/dat-provider/dapl_provider.c (working copy) @@ -71,9 +71,8 @@ static struct dat_provider g_dapl_provid .ia_close_func = &dapl_ia_close, .ia_memtype_hint_func = &dapl_ia_memtype_hint, - .set_consumer_context_func = &dapl_set_consumer_context, - .get_consumer_context_func = &dapl_get_consumer_context, - .get_handle_type_func = &dapl_get_handle_type, + .set_consumer_context_func = NULL, + .get_consumer_context_func = NULL, .cr_query_func = &dapl_cr_query, .cr_accept_func = &dapl_cr_accept, Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2563) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -70,7 +70,7 @@ static void dapl_destroy_cm_id(struct da conn, conn->cm_id); ib_destroy_cm_id(conn->cm_id); if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } } @@ -172,7 +172,7 @@ static int dapl_cm_active_cb_handler(str spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -285,7 +285,7 @@ static int dapl_cm_passive_cb_handler(st spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -413,7 +413,7 @@ error: * Initiate a connection with the passive listener on another node * * Input: - * ep_handle + * ep * remote_ia_address * remote_conn_qual * private_data_size size of private data and structure @@ -428,7 +428,7 @@ error: * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data) @@ -438,7 +438,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han struct dapl_cm_id *conn; int status; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->qp == NULL) { printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); return DAT_INTERNAL_ERROR; @@ -460,9 +460,9 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han kfree(conn); return DAT_INTERNAL_ERROR; } - conn->ep->cm_handle = conn; + conn->ep->cm = conn; - ia_ptr = conn->ep->header.owner_ia; + ia_ptr = conn->ep->common.owner_ia; /* Setup QP/CM parameters */ memset(&conn->param, 0, sizeof conn->param); @@ -507,7 +507,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han * Disconnect an EP * * Input: - * ep_handle + * ep * close_flags * * Output: @@ -521,7 +521,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han */ u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags) { - struct dapl_cm_id *conn = ep_ptr->cm_handle; + struct dapl_cm_id *conn = ep_ptr->cm; int status; dapl_dbg_log(DAPL_DBG_TYPE_CM, @@ -537,7 +537,7 @@ u32 dapl_ib_disconnect(struct dapl_ep *e status = ib_send_cm_dreq(conn->cm_id, NULL, 0); if (status) printk(KERN_ERR "dapl_ib_disconnect: CM ID 0x%p " - "status %d\n", ep_ptr->cm_handle, status); + "status %d\n", ep_ptr->cm, status); } return DAT_SUCCESS; @@ -607,14 +607,14 @@ void dapl_ib_reinit_ep(struct dapl_ep *e dapl_dbg_log(DAPL_DBG_TYPE_CM, " >>> dapl_ib_reinit_ep: EP %p\n", ep_ptr); - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* * What if QP is connected? Do we need to destroy the cm_id * here as well, send a DREQ, REJ? What cm_id state are we in? * we may not care if we just destroy the cm_id - * ib_destroy_cm_id(ep_ptr->cm_handle); - * supporting this call may require setting the cm_handle to + * ib_destroy_cm_id(ep_ptr->cm); + * supporting this call may require setting the cm to * something like NULL wherever it is destroyed */ @@ -685,7 +685,7 @@ u32 dapl_ib_setup_conn_listener(struct d * Have the CM remove a connection listener. * * Input: - * ia_handle IA handle + * ia IA handle * ServiceID IB Channel Service ID * * Output: @@ -719,7 +719,7 @@ u32 dapl_ib_remove_conn_listener(struct * Perform necessary steps to reject a connection * * Input: - * cm_handle + * cm * * Output: * none @@ -730,24 +730,24 @@ u32 dapl_ib_remove_conn_listener(struct * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle) +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm) { int status; - if (cm_handle == NULL) { + if (cm == NULL) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_reject_connection: invalid cm handle\n"); return DAT_SUCCESS; } - status = ib_send_cm_rej(cm_handle->cm_id, IB_CM_REJ_CONSUMER_DEFINED, + status = ib_send_cm_rej(cm->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_reject_connection: " "ib_send_cm_rej failed: %d\n", status); return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); } - dapl_destroy_cm_id(cm_handle); + dapl_destroy_cm_id(cm); return DAT_SUCCESS; } @@ -757,8 +757,8 @@ u32 dapl_ib_reject_connection(struct dap * Perform necessary steps to accept a connection * * Input: - * cr_handle - * ep_handle + * cr + * ep * private_data_size * private_data * @@ -771,7 +771,7 @@ u32 dapl_ib_reject_connection(struct dap * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *prd_ptr) { @@ -783,9 +783,9 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN struct ib_cm_rep_param passive_params; struct dapl_cm_id *conn; - cr_ptr = (struct dapl_cr *)cr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; - ia_ptr = ep_ptr->header.owner_ia; + cr_ptr = (struct dapl_cr *)cr; + ep_ptr = (struct dapl_ep *)ep; + ia_ptr = ep_ptr->common.owner_ia; conn = cr_ptr->ib_cm_handle; /* Obtain size of private data structure & contents */ @@ -810,8 +810,8 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN } } - cr_ptr->param.local_ep_handle = ep_handle; - ep_ptr->cm_handle = conn; + cr_ptr->param.local_ep = ep; + ep_ptr->cm = conn; conn->ep = ep_ptr; memset(&passive_params, 0, sizeof passive_params); @@ -851,7 +851,7 @@ reject: /* * ib_cm_get_remote_gid */ -static int ib_cm_get_remote_gid(struct dapl_cm_id *cm_handle, u8 *remote_gid) +static int ib_cm_get_remote_gid(struct dapl_cm_id *cm, u8 *remote_gid) { return DAT_INTERNAL_ERROR; /* for now!!! */ } @@ -862,7 +862,7 @@ static int ib_cm_get_remote_gid(struct d * Obtain the remote IP address given a connection * * Input: - * cr_handle + * cr * private data structure handle (only if not using ATS) * * Output: @@ -874,7 +874,7 @@ static int ib_cm_get_remote_gid(struct d * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address) { struct dapl_ia *ia_ptr; @@ -883,7 +883,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da struct dapl_cr *cr_ptr = (struct dapl_cr *)dat_handle; - ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia_handle; + ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia; rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); if (rc < 0) Index: linux-kernel/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel/patches/alt_dat_provider_makefile (revision 2563) +++ linux-kernel/patches/alt_dat_provider_makefile (working copy) @@ -21,8 +21,6 @@ PROVIDER_MODULES := \ dapl_cr_util \ dapl_ep \ dapl_evd \ - dapl_get_consumer_context \ - dapl_get_handle_type \ dapl_hash \ dapl_hca_util \ dapl_ia_close \ @@ -39,7 +37,6 @@ PROVIDER_MODULES := \ dapl_ring_buffer_util \ dapl_rmr \ dapl_rsp \ - dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ dapl_util Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2563) +++ linux-kernel/dat/api.c (working copy) @@ -350,7 +350,7 @@ static u32 dat_strerror_minor(u32 value, } u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, + struct dat_evd **async_event_handle, struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; @@ -387,13 +387,12 @@ u32 dat_ia_openv(const char *name, int a return status; } - return ia_open_func(name, async_event_qlen, async_event_handle, - ia_handle); + return ia_open_func(name, async_event_qlen, async_event_handle, ia); } EXPORT_SYMBOL(dat_ia_openv); -u32 dat_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags flags) +u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; @@ -401,17 +400,16 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle const char *ia_name; dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "%s(ia:%x, flags:%x)\n", __func__, ia_handle, flags); + "%s(ia:%x, flags:%x)\n", __func__, ia, flags); - provider = DAT_HANDLE_TO_PROVIDER(ia_handle); + provider = DAT_HANDLE_TO_PROVIDER(ia); ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia_handle, NULL, NULL, + if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); - else if (DAT_SUCCESS != (status = provider->ia_close_func(ia_handle, - flags))) + else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "close function for IA %s failed\n", ia_name); else { Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2563) +++ linux-kernel/dat/dat.h (working copy) @@ -333,20 +333,6 @@ enum dat_return_subtype { #define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) #define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) -/* dat handle types */ -enum dat_handle_type { - DAT_HANDLE_TYPE_CR, - DAT_HANDLE_TYPE_EP, - DAT_HANDLE_TYPE_EVD, - DAT_HANDLE_TYPE_IA, - DAT_HANDLE_TYPE_LMR, - DAT_HANDLE_TYPE_PSP, - DAT_HANDLE_TYPE_PZ, - DAT_HANDLE_TYPE_RMR, - DAT_HANDLE_TYPE_RSP, - DAT_HANDLE_TYPE_SRQ -}; - #define DAT_NAME_MAX_LENGTH 256 struct dat_named_attr { @@ -385,21 +371,6 @@ enum dat_completion_flags { #define DAT_TIMEOUT_MAX ULONG_MAX -/* dat handles */ -typedef void *DAT_HANDLE; -typedef DAT_HANDLE DAT_CR_HANDLE; -typedef DAT_HANDLE DAT_EP_HANDLE; -typedef DAT_HANDLE DAT_EVD_HANDLE; -typedef DAT_HANDLE DAT_IA_HANDLE; -typedef DAT_HANDLE DAT_LMR_HANDLE; -typedef DAT_HANDLE DAT_PSP_HANDLE; -typedef DAT_HANDLE DAT_PZ_HANDLE; -typedef DAT_HANDLE DAT_RMR_HANDLE; -typedef DAT_HANDLE DAT_RSP_HANDLE; -typedef DAT_HANDLE DAT_SRQ_HANDLE; - -#define DAT_HANDLE_NULL ((DAT_HANDLE)NULL) - typedef u64 DAT_CONN_QUAL; typedef u64 DAT_PORT_QUAL; @@ -541,8 +512,8 @@ enum dat_mem_priv_flags { /* RMR Arguments */ struct dat_rmr_param { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; struct dat_lmr_triplet lmr_triplet; enum dat_mem_priv_flags mem_priv; DAT_RMR_CONTEXT rmr_context; @@ -588,17 +559,17 @@ enum dat_ep_state { }; struct dat_ep_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_ep_state ep_state; struct sockaddr *local_ia_address_ptr; DAT_PORT_QUAL local_port_qual; struct sockaddr *remote_ia_address_ptr; DAT_PORT_QUAL remote_port_qual; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EVD_HANDLE request_evd_handle; - DAT_EVD_HANDLE connect_evd_handle; - DAT_SRQ_HANDLE srq_handle; + struct dat_pz *pz; + struct dat_evd *recv_evd; + struct dat_evd *request_evd; + struct dat_evd *connect_evd; + struct dat_srq *srq; struct dat_ep_attr ep_attr; }; @@ -654,9 +625,9 @@ struct dat_srq_attr { }; struct dat_srq_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_srq_state srq_state; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; int max_recv_dtos; int max_recv_iov; int low_watermark; @@ -667,25 +638,25 @@ struct dat_srq_param { /* PZ Parameters */ struct dat_pz_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; }; /* PSP Parameters */ struct dat_psp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; }; /* RSP Parameters */ struct dat_rsp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_ep *ep; }; /* Connection Request Parameters. @@ -712,11 +683,11 @@ struct dat_cr_param { /* The local Endpoint provided by the Service Point for the requested * connection. It is the only Endpoint that can accept a Connection - * Request on this Service Point. The value DAT_HANDLE_NULL + * Request on this Service Point. The value NULL * represents that there is no associated local Endpoint for the * requested connection. */ - DAT_EP_HANDLE local_ep_handle; + struct dat_ep *local_ep; }; /************************** Events ******************************/ @@ -753,7 +724,7 @@ enum dat_dto_completion_status { /* DTO completion event data */ /* transfered_length is not defined if status is not DAT_SUCCESS */ struct dat_dto_completion_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; DAT_DTO_COOKIE user_cookie; enum dat_dto_completion_status status; u64 transfered_length; @@ -761,25 +732,20 @@ struct dat_dto_completion_event_data { /* RMR bind completion event data */ struct dat_rmr_bind_completion_event_data { - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr *rmr; DAT_RMR_COOKIE user_cookie; enum dat_dto_completion_status status; }; -typedef union dat_sp_handle { - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; -} DAT_SP_HANDLE; - /* Connection Request Arrival event data */ struct dat_cr_arrival_event_data { /* Handle to the Service Point that received the Connection Request * from the remote side. If the Service Point was Reserved, sp is - * DAT_HANDLE_NULL because the reserved Service Point is + * NULL because the reserved Service Point is * automatically destroyed upon generating this event. Can be PSP * or RSP. */ - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; /* Address of the IA on which the Connection Request arrived. */ struct sockaddr *local_ia_address_ptr; @@ -791,30 +757,30 @@ struct dat_cr_arrival_event_data { /* The Connection Request instance created by a Provider for the * arrived Connection Request. Consumers can find out private_data - * passed by a remote Consumer from cr_handle. It is up to a Consumer + * passed by a remote Consumer from cr. It is up to a Consumer * to dat_cr_accept or dat_cr_reject of the Connection Request. */ - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; }; /* Connection event data */ struct dat_connection_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; int private_data_size; void *private_data; }; /* Async Error event data */ -/* For unaffiliated asynchronous event dat_handle is ia_handle. For - * Endpoint affiliated asynchronous event dat_handle is ep_handle. For - * EVD affiliated asynchronous event dat_handle is evd_handle. For SRQ +/* For unaffiliated asynchronous event dat_handle is ia. For + * Endpoint affiliated asynchronous event dat_handle is ep. For + * EVD affiliated asynchronous event dat_handle is evd. For SRQ * affiliated asynchronous event dat_handle is srq_handle. For Memory - * affiliated asynchronous event dat_handle is either lmr_handle, - * rmr_handle or pz_handle. + * affiliated asynchronous event dat_handle is either lmr, + * rmr_handle or pz. */ struct dat_asynch_error_event_data { - DAT_HANDLE dat_handle; /* either IA, EP, EVD, SRQ, */ + void *dat_handle; /* either IA, EP, EVD, SRQ, */ /* LMR, RMR, or PZ handle */ int reason; /* object specific */ }; @@ -894,7 +860,7 @@ typedef union dat_event_data { struct dat_event { enum dat_event_number event_number; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; DAT_EVENT_DATA event_data; }; @@ -934,7 +900,7 @@ struct dat_provider_info { }; struct dat_evd_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; int evd_qlen; enum dat_upcall_policy upcall_policy; struct dat_upcall_object upcall; @@ -960,7 +926,7 @@ enum dat_mem_optimize_flags { typedef union dat_region_description { void *for_va; - DAT_LMR_HANDLE for_lmr_handle; + struct dat_lmr *for_lmr; void *for_pointer; /* For kdapl only */ void *for_array; /* For kdapl only */ u64 for_pa; /* For kdapl only */ @@ -968,11 +934,11 @@ typedef union dat_region_description { /* LMR Arguments */ struct dat_lmr_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_mem_type mem_type; DAT_REGION_DESCRIPTION region_desc; u64 length; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; enum dat_mem_priv_flags mem_priv; DAT_LMR_CONTEXT lmr_context; DAT_RMR_CONTEXT rmr_context; @@ -1083,178 +1049,176 @@ struct dat_provider_attr { * ****************************************************************/ -typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *); +typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **); -typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *, u32, u32, boolean_t); +typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **, u32, u32, boolean_t); -typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum dat_close_flags); -typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_HANDLE, DAT_EVD_HANDLE *, +typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); - -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(void *, union dat_context); -typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, enum dat_handle_type *); +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(void *, union dat_context *); -typedef u32 (*DAT_CR_QUERY_FUNC)(DAT_CR_HANDLE, struct dat_cr_param *); +typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct dat_cr_param *); -typedef u32 (*DAT_CR_ACCEPT_FUNC)(DAT_CR_HANDLE, DAT_EP_HANDLE, int, +typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct dat_ep *, int, const void*); -typedef u32 (*DAT_CR_REJECT_FUNC)(DAT_CR_HANDLE); +typedef u32 (*DAT_CR_REJECT_FUNC)(struct dat_cr *); -typedef u32 (*DAT_CR_HANDOFF_FUNC)(DAT_CR_HANDLE, DAT_CONN_QUAL); +typedef u32 (*DAT_CR_HANDOFF_FUNC)(struct dat_cr *, DAT_CONN_QUAL); -typedef u32 (*DAT_EVD_RESIZE_FUNC)(DAT_EVD_HANDLE, int); +typedef u32 (*DAT_EVD_RESIZE_FUNC)(struct dat_evd *, int); -typedef u32 (*DAT_EVD_POST_SE_FUNC)(DAT_EVD_HANDLE, const struct dat_event *); +typedef u32 (*DAT_EVD_POST_SE_FUNC)(struct dat_evd *, const struct dat_event *); -typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(DAT_EVD_HANDLE, struct dat_event *); +typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(struct dat_evd *, struct dat_event *); -typedef u32 (*DAT_EVD_FREE_FUNC)(DAT_EVD_HANDLE); +typedef u32 (*DAT_EVD_FREE_FUNC)(struct dat_evd *); -typedef u32 (*DAT_EVD_QUERY_FUNC)(DAT_EVD_HANDLE, struct dat_evd_param *); +typedef u32 (*DAT_EVD_QUERY_FUNC)(struct dat_evd *, struct dat_evd_param *); -typedef u32 (*DAT_EP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - const struct dat_ep_attr *, DAT_EP_HANDLE *); +typedef u32 (*DAT_EP_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, struct dat_evd *, + struct dat_evd *, struct dat_evd *, + const struct dat_ep_attr *, struct dat_ep **); -typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_SRQ_HANDLE, +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_evd *, struct dat_evd *, + struct dat_evd *, struct dat_srq *, const struct dat_ep_attr *, - DAT_EP_HANDLE *); + struct dat_ep **); -typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_HANDLE, struct dat_ep_param *); +typedef u32 (*DAT_EP_QUERY_FUNC)(struct dat_ep *, struct dat_ep_param *); -typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, enum dat_ep_param_mask, +typedef u32 (*DAT_EP_MODIFY_FUNC)(struct dat_ep *, enum dat_ep_param_mask, const struct dat_ep_param *); -typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, +typedef u32 (*DAT_EP_CONNECT_FUNC)(struct dat_ep *, struct sockaddr *, DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); -typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(struct dat_ep *, struct dat_ep *, unsigned long timeout, int, const void *, enum dat_qos); -typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_EP_DISCONNECT_FUNC)(struct dat_ep *, enum dat_close_flags); -typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_SEND_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RECV_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RECV_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, enum dat_ep_state *, +typedef u32 (*DAT_EP_GET_STATUS_FUNC)(struct dat_ep *, enum dat_ep_state *, boolean_t *, boolean_t *); -typedef u32 (*DAT_EP_FREE_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_FREE_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RESET_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_RESET_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(DAT_EP_HANDLE, int *, int *); +typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(struct dat_ep *, int *, int *); -typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(DAT_EP_HANDLE, int, int); +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(struct dat_ep *, int, int); -typedef u32 (*DAT_LMR_FREE_FUNC)(DAT_LMR_HANDLE); +typedef u32 (*DAT_LMR_FREE_FUNC)(struct dat_lmr *); -typedef u32 (*DAT_LMR_QUERY_FUNC)(DAT_LMR_HANDLE, struct dat_lmr_param *); +typedef u32 (*DAT_LMR_QUERY_FUNC)(struct dat_lmr *, struct dat_lmr_param *); -typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_PZ_HANDLE, DAT_RMR_HANDLE *); +typedef u32 (*DAT_RMR_CREATE_FUNC)(struct dat_pz *, struct dat_rmr * *); -typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RMR_HANDLE, struct dat_rmr_param *); +typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); -typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, - enum dat_mem_priv_flags, DAT_EP_HANDLE, +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, + enum dat_mem_priv_flags, struct dat_ep *, DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); -typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); +typedef u32 (*DAT_RMR_FREE_FUNC)(struct dat_rmr *); -typedef u32 (*DAT_PSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, - enum dat_psp_flags, DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_evd *, + enum dat_psp_flags, struct dat_sp **); -typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL *, - DAT_EVD_HANDLE, enum dat_psp_flags, - DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(struct dat_ia *, DAT_CONN_QUAL *, + struct dat_evd *, enum dat_psp_flags, + struct dat_sp **); -typedef u32 (*DAT_PSP_QUERY_FUNC)(DAT_PSP_HANDLE, struct dat_psp_param *); +typedef u32 (*DAT_PSP_QUERY_FUNC)(struct dat_sp *, struct dat_psp_param *); -typedef u32 (*DAT_PSP_FREE_FUNC)(DAT_PSP_HANDLE); +typedef u32 (*DAT_PSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_RSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, - DAT_EVD_HANDLE, DAT_RSP_HANDLE *); +typedef u32 (*DAT_RSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_ep *, + struct dat_evd *, struct dat_sp **); -typedef u32 (*DAT_RSP_QUERY_FUNC)(DAT_RSP_HANDLE, struct dat_rsp_param *); +typedef u32 (*DAT_RSP_QUERY_FUNC)(struct dat_sp *, struct dat_rsp_param *); -typedef u32 (*DAT_RSP_FREE_FUNC)(DAT_RSP_HANDLE); +typedef u32 (*DAT_RSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_PZ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE *); +typedef u32 (*DAT_PZ_CREATE_FUNC)(struct dat_ia *, struct dat_pz **); -typedef u32 (*DAT_PZ_QUERY_FUNC)(DAT_PZ_HANDLE, struct dat_pz_param *); +typedef u32 (*DAT_PZ_QUERY_FUNC)(struct dat_pz *, struct dat_pz_param *); -typedef u32 (*DAT_PZ_FREE_FUNC)(DAT_PZ_HANDLE); +typedef u32 (*DAT_PZ_FREE_FUNC)(struct dat_pz *); -typedef u32 (*DAT_SRQ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - struct dat_srq_attr *, DAT_SRQ_HANDLE *); +typedef u32 (*DAT_SRQ_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_srq_attr *, struct dat_srq **); -typedef u32 (*DAT_SRQ_SET_LW_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_SET_LW_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_FREE_FUNC)(DAT_SRQ_HANDLE); +typedef u32 (*DAT_SRQ_FREE_FUNC)(struct dat_srq *); -typedef u32 (*DAT_SRQ_QUERY_FUNC)(DAT_SRQ_HANDLE, struct dat_srq_param *); +typedef u32 (*DAT_SRQ_QUERY_FUNC)(struct dat_srq *, struct dat_srq_param *); -typedef u32 (*DAT_SRQ_RESIZE_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_RESIZE_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DAT_SRQ_HANDLE, int, +typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(struct dat_srq *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, +typedef u32 (*DAT_LMR_KCREATE_FUNC)(struct dat_ia *, enum dat_mem_type, DAT_REGION_DESCRIPTION, u64, - DAT_PZ_HANDLE, enum dat_mem_priv_flags, + struct dat_pz *, enum dat_mem_priv_flags, enum dat_mem_optimize_flags, - DAT_LMR_HANDLE *, + struct dat_lmr **, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, u64 *, u64 *); -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, u64, +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(struct dat_ia *, enum dat_mem_type, u64, enum dat_mem_optimize_flags, u64 *, u64 *); -typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, enum dat_upcall_policy, +typedef u32 (*DAT_EVD_KCREATE_FUNC) (struct dat_ia *, int, enum dat_upcall_policy, const struct dat_upcall_object *, - enum dat_evd_flags, DAT_EVD_HANDLE *); + enum dat_evd_flags, struct dat_evd **); -typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(struct dat_evd *, enum dat_upcall_policy, const struct dat_upcall_object *); @@ -1269,7 +1233,6 @@ struct dat_provider { DAT_SET_CONSUMER_CONTEXT_FUNC set_consumer_context_func; DAT_GET_CONSUMER_CONTEXT_FUNC get_consumer_context_func; - DAT_GET_HANDLE_TYPE_FUNC get_handle_type_func; DAT_CR_QUERY_FUNC cr_query_func; DAT_CR_ACCEPT_FUNC cr_accept_func; @@ -1338,6 +1301,52 @@ struct dat_provider { DAT_SRQ_SET_LW_FUNC srq_set_lw_func; }; +struct dat_ep { + struct dat_provider *provider; +}; + +struct dat_ia { + struct dat_provider *provider; +}; + +struct dat_evd { + struct dat_provider *provider; +}; + +struct dat_cno { + struct dat_provider *provider; +}; + +struct dat_srq { + struct dat_provider *provider; +}; + +struct dat_pz { + struct dat_provider *provider; +}; + +struct dat_lmr { + struct dat_provider *provider; +}; + +struct dat_rmr { + struct dat_provider *provider; +}; + +enum dat_sp_type { + DAT_SP_TYPE_RSP, + DAT_SP_TYPE_PSP +}; + +struct dat_sp { + struct dat_provider *provider; + enum dat_sp_type type; +}; + +struct dat_cr { + struct dat_provider *provider; +}; + /*************************************************************** * * FUNCTION PROTOTYPES @@ -1359,8 +1368,8 @@ extern u32 dat_registry_remove_provider( * DAT registry functions for consumers */ extern u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, - DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, + struct dat_evd **async_event_handle, + struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety); #define dat_ia_open(name, qlen, async_evd, ia) \ @@ -1368,7 +1377,7 @@ extern u32 dat_ia_openv(const char *name DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ DAT_THREADSAFE) -extern u32 dat_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); extern u32 dat_strerror(u32, const char **, const char **); @@ -1383,27 +1392,21 @@ extern u32 dat_registry_list_providers(i #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) -static inline u32 dat_set_consumer_context(DAT_HANDLE handle, +static inline u32 dat_set_consumer_context(void *handle, union dat_context context) { return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, context); } -static inline u32 dat_get_consumer_context(DAT_HANDLE handle, +static inline u32 dat_get_consumer_context(void *handle, union dat_context *context) { return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, context); } -static inline u32 dat_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type) -{ - return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); -} - -static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, +static inline u32 dat_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimize, @@ -1415,7 +1418,7 @@ static inline u32 dat_ia_memtype_hint(DA preferred_alignment); } -static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE *async_evd, +static inline u32 dat_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -1423,49 +1426,49 @@ static inline u32 dat_ia_query(DAT_IA_HA ia_query_func, ia, async_evd, ia_attr, provider_attr); } -static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, +static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data) { return DAT_CALL_PROVIDER_FUNC( cr_accept_func, cr, ep, private_data_size, private_data); } -static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) +static inline u32 dat_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff) { return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); } -static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) +static inline u32 dat_cr_query(struct dat_cr *cr, struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } -static inline u32 dat_cr_reject(DAT_CR_HANDLE cr) +static inline u32 dat_cr_reject(struct dat_cr *cr) { return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); } -static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) +static inline u32 dat_evd_dequeue(struct dat_evd *evd, struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } -static inline u32 dat_evd_free(DAT_EVD_HANDLE evd) +static inline u32 dat_evd_free(struct dat_evd *evd) { return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); } -static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, +static inline u32 dat_evd_kcreate(struct dat_ia *ia, int qlen, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall, enum dat_evd_flags flags, - DAT_EVD_HANDLE * evd) + struct dat_evd ** evd) { return DAT_CALL_PROVIDER_FUNC(evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); } -static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall) { @@ -1473,23 +1476,23 @@ static inline u32 dat_evd_modify_upcall( upcall); } -static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_post_se(struct dat_evd *evd, const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } -static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) +static inline u32 dat_evd_query(struct dat_evd *evd, struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } -static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, int qlen) +static inline u32 dat_evd_resize(struct dat_evd *evd, int qlen) { return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, +static inline u32 dat_ep_connect(struct dat_ep *ep, struct sockaddr *ia_addr, DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, @@ -1500,36 +1503,36 @@ static inline u32 dat_ep_connect(DAT_EP_ qos, flags); } -static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, +static inline u32 dat_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, struct dat_evd *out_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_func, ia, pz, in_evd, out_evd, connect_evd, attr, ep); } -static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, - DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, - DAT_SRQ_HANDLE srq, +static inline u32 dat_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, + struct dat_evd *out_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_with_srq_func, ia, pz, in_evd, out_evd, connect_evd, srq, attr, ep); } -static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_disconnect(struct dat_ep *ep, enum dat_close_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } -static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, +static inline u32 dat_ep_dup_connect(struct dat_ep *ep, struct dat_ep *dup_ep, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) @@ -1538,25 +1541,25 @@ static inline u32 dat_ep_dup_connect(DAT private_data_size, private_data, qos); } -static inline u32 dat_ep_free(DAT_EP_HANDLE ep) +static inline u32 dat_ep_free(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, enum dat_ep_state *state, +static inline u32 dat_ep_get_status(struct dat_ep *ep, enum dat_ep_state *state, boolean_t *recv_idle, boolean_t *req_idle) { return DAT_CALL_PROVIDER_FUNC(ep_get_status_func, ep, state, recv_idle, req_idle); } -static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, enum dat_ep_param_mask mask, +static inline u32 dat_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } -static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_read(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1566,7 +1569,7 @@ static inline u32 dat_ep_post_rdma_read( local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_write(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1576,7 +1579,7 @@ static inline u32 dat_ep_post_rdma_write local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_recv(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, enum dat_completion_flags flags) @@ -1585,7 +1588,7 @@ static inline u32 dat_ep_post_recv(DAT_E cookie, flags); } -static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_send(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1595,24 +1598,24 @@ static inline u32 dat_ep_post_send(DAT_E ep_post_send_func, ep, size, local_iov, cookie, flags); } -static inline u32 dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) +static inline u32 dat_ep_query(struct dat_ep *ep, struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } -static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, int *bufs_alloc, +static inline u32 dat_ep_recv_query(struct dat_ep *ep, int *bufs_alloc, int *bufs_avail) { return DAT_CALL_PROVIDER_FUNC(ep_recv_query_func, ep, bufs_alloc, bufs_avail); } -static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) +static inline u32 dat_ep_reset(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); } -static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, +static inline u32 dat_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -1620,17 +1623,17 @@ static inline u32 dat_ep_set_watermark(D soft_high_watermark, hard_high_watermark); } -static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) +static inline u32 dat_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } -static inline u32 dat_lmr_free(DAT_LMR_HANDLE lmr) +static inline u32 dat_lmr_free(struct dat_lmr *lmr) { return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); } -static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1638,7 +1641,7 @@ static inline u32 dat_lmr_sync_rdma_read num_iovs); } -static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1646,20 +1649,20 @@ static inline u32 dat_lmr_sync_rdma_writ num_iovs); } -static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE *rmr) +static inline u32 dat_rmr_create(struct dat_pz *pz, struct dat_rmr * *rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline u32 dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) +static inline u32 dat_rmr_query(struct dat_rmr * rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, +static inline u32 dat_rmr_bind(struct dat_rmr * rmr, const struct dat_lmr_triplet *iov, enum dat_mem_priv_flags mem_flags, - DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, + struct dat_ep *ep, DAT_RMR_COOKIE cookie, enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) { @@ -1667,84 +1670,84 @@ static inline u32 dat_rmr_bind(DAT_RMR_H cookie, comp_flags, context); } -static inline u32 dat_rmr_free(DAT_RMR_HANDLE rmr) +static inline u32 dat_rmr_free(struct dat_rmr * rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) +static inline u32 dat_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd, +static inline u32 dat_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_any_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) +static inline u32 dat_psp_query(struct dat_sp *psp, struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } -static inline u32 dat_psp_free(DAT_PSP_HANDLE psp) +static inline u32 dat_psp_free(struct dat_sp *psp) { return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); } -static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, - DAT_RSP_HANDLE *rsp) +static inline u32 dat_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_create_func, ia, conn_qual, ep, evd, rsp); } -static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) +static inline u32 dat_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } -static inline u32 dat_rsp_free(DAT_RSP_HANDLE rsp) +static inline u32 dat_rsp_free(struct dat_sp *rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); } -static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE *pz) +static inline u32 dat_pz_create(struct dat_ia *ia, struct dat_pz **pz) { return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) +static inline u32 dat_pz_query(struct dat_pz *pz, struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } -static inline u32 dat_pz_free(DAT_PZ_HANDLE pz) +static inline u32 dat_pz_free(struct dat_pz *pz) { return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); } -static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, +static inline u32 dat_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *attr, - DAT_SRQ_HANDLE *srq) + struct dat_srq **srq) { return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); } -static inline u32 dat_srq_free(DAT_SRQ_HANDLE srq) +static inline u32 dat_srq_free(struct dat_srq *srq) { return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); } -static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, int num_iovs, +static inline u32 dat_srq_post_recv(struct dat_srq *srq, int num_iovs, struct dat_lmr_triplet *iovs, DAT_DTO_COOKIE cookie) { @@ -1752,34 +1755,34 @@ static inline u32 dat_srq_post_recv(DAT_ cookie); } -static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) +static inline u32 dat_srq_query(struct dat_srq *srq, struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } -static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, int max_recv_dtos) +static inline u32 dat_srq_resize(struct dat_srq *srq, int max_recv_dtos) { return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); } -static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, int low_watermark) +static inline u32 dat_srq_set_lw(struct dat_srq *srq, int low_watermark) { return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, enum dat_mem_type type, +static inline u32 dat_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type type, DAT_REGION_DESCRIPTION region, u64 len, - DAT_PZ_HANDLE pz, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { return DAT_CALL_PROVIDER_FUNC(lmr_kcreate_func, ia, type, region, len, - pz, privileges, optimization, lmr_handle, + pz, privileges, optimization, lmr, lmr_context, rmr_context, registered_length, registered_address); } From halr at voltaire.com Mon Jun 6 14:46:39 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 Jun 2005 17:46:39 -0400 Subject: [openib-general] Re: [ib_at] oops in ib_at when running kdapltest In-Reply-To: <20050605123618.GA3562@mellanox.co.il> References: <20050605123618.GA3562@mellanox.co.il> Message-ID: <1118094374.4493.68.camel@hal.voltaire.com> On Sun, 2005-06-05 at 08:36, Itamar wrote: > Hi All, > I was running today some kdapltest and after about 10 tests i got this oops. > the test was : > ./kdapltest -T T -s 11.1.4.43 -D mthca0a -d -t 2 -w 8 -i 10 client SR server SR > (the same test pass before it got the oops) > my system is : > dual x86 xeon 2G with 1GB memory + suse9.1 + k2.6.11 + svn rev 2546 + kdapl > my kernel is with CONFIG_DEBUG_PAGEALLOC > > Jun 5 13:45:32 mtvs42 kernel: Unable to handle kernel paging request at virtual address ec298ff4 > Jun 5 13:45:32 mtvs42 kernel: printing eip: > Jun 5 13:45:32 mtvs42 kernel: f8ad3c15 > Jun 5 13:45:32 mtvs42 kernel: *pde = 0047c067 > Jun 5 13:45:32 mtvs42 kernel: Oops: 0002 [#1] > Jun 5 13:45:32 mtvs42 kernel: SMP DEBUG_PAGEALLOC > Jun 5 13:45:32 mtvs42 kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at dat ib_umad ib_ipoib ib_sa edd joydev st sd_mod sr_mod scsi_mod ide_cd cdrom nvram usbserial parport_pc lp parport autofs thermal processor fan button ipv6 battery ac e100 mii ib_mthca ib_mad ib_core i2c_i801 i2c_core uhci_hcd usbcore hw_random evdev dm_mod ext3 jbd > Jun 5 13:45:32 mtvs42 kernel: CPU: 0 > Jun 5 13:45:32 mtvs42 kernel: EIP: 0060:[pg0+946297877/1069155328] Not tainted VLI > Jun 5 13:45:32 mtvs42 kernel: EIP: 0060:[] Not tainted VLI > Jun 5 13:45:32 mtvs42 kernel: EFLAGS: 00010246 (2.6.11-gen2) > Jun 5 13:45:32 mtvs42 kernel: EIP is at resolve_path+0xa5/0x100 [ib_at] > Jun 5 13:45:32 mtvs42 kernel: eax: 00000000 ebx: ec298f3c ecx: f21b7e80 edx: 00000000 > Jun 5 13:45:32 mtvs42 kernel: esi: ec79dae8 edi: 00000064 ebp: ec79dae8 esp: ec79daa8 > Jun 5 13:45:32 mtvs42 kernel: ds: 007b es: 007b ss: 0068 > Jun 5 13:45:32 mtvs42 kernel: Process DT_Mdep_Thread_ (pid: 9737, threadinfo=ec79c000 task=ebd18ad0) > Jun 5 13:45:32 mtvs42 kernel: Stack: 000080fe 00000000 00c90200 51d00001 000080fe 00000000 01c90200 71f79d07 > Jun 5 13:45:32 mtvs42 kernel: 00000000 00000000 00000000 00000000 00000000 ffff0001 00000000 00000000 > Jun 5 13:45:32 mtvs42 kernel: 000080fe 00000000 00c90200 51d00001 000080fe 00000000 01c90200 71f79d07 > Jun 5 13:45:32 mtvs42 kernel: Call Trace: > Jun 5 13:45:32 mtvs42 kernel: [pg0+946298889/1069155328] ib_at_paths_by_route+0xa9/0xd0 [ib_at] > Jun 5 13:45:32 mtvs42 kernel: [] ib_at_paths_by_route+0xa9/0xd0 [ib_at] > Jun 5 13:45:32 mtvs42 kernel: [pg0+963317968/1069155328] dapl_rt_comp_handler+0x135/0x190 [ib_dat_provider] > Jun 5 13:45:32 mtvs42 kernel: [] dapl_rt_comp_handler+0x135/0x190 [ib_dat_provider] > Jun 5 13:45:32 mtvs42 kernel: [pg0+963319747/1069155328] dapl_ib_connect+0x698/0x6ae [ib_dat_provider] > Jun 5 13:45:32 mtvs42 kernel: [] dapl_ib_connect+0x698/0x6ae [ib_dat_provider] > Jun 5 13:45:32 mtvs42 kernel: [pg0+963336373/1069155328] dapl_ep_connect+0x73f/0x7b3 [ib_dat_provider] Can you turn on the DEBUG printing in at.c by changing the following lines: #define DEBUG(fmt, ...) while (0) {} //#define DEBUG(fmt, arg ...) printk("ib_at: %s: " fmt "\n", __FUNCTION__, ## arg); to: //#define DEBUG(fmt, ...) while (0) {} #define DEBUG(fmt, arg ...) printk("ib_at: %s: " fmt "\n", __FUNCTION__, ## arg); rebuild ib_at, and send the output of /var/log/messages. Thanks. -- Hal From halr at voltaire.com Mon Jun 6 14:54:28 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 Jun 2005 17:54:28 -0400 Subject: [openib-general] uverbs_main cleanup Message-ID: <1118094867.4493.72.camel@hal.voltaire.com> Hi Roland, Should the order of unregistering the uverbs event fs and the unmounting be reversed or does this not matter ? static void __exit ib_uverbs_cleanup(void) { ib_unregister_client(&uverbs_client); unregister_filesystem(&uverbs_event_fs); mntput(uverbs_event_mnt); ... static void __exit ib_uverbs_cleanup(void) { ib_unregister_client(&uverbs_client); mntput(uverbs_event_mnt); unregister_filesystem(&uverbs_event_fs); ... -- Hal From iod00d at hp.com Mon Jun 6 15:05:06 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 6 Jun 2005 15:05:06 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup dat/ a bit more In-Reply-To: <469958e0050606121567925a45@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C22@mtvex01.mtv.mtl.com> <469958e00506030430687795a8@mail.gmail.com> <20050603172727.GA3888@lst.de> <469958e0050603115887db8fa@mail.gmail.com> <20050604103058.GB17004@lst.de> <469958e005060416332c9f4f9b@mail.gmail.com> <20050606164138.GA11120@esmail.cup.hp.com> <469958e0050606121567925a45@mail.gmail.com> Message-ID: <20050606220506.GC12068@esmail.cup.hp.com> On Mon, Jun 06, 2005 at 12:15:53PM -0700, Caitlin Bestler wrote: > As your reply indicates below, the OS does *not* > know *when* the device is active. The driver doesn't have to and in fact does not for other technologies either. The device driver knows (or can know) who registered the region since that's done in the process context or on behalf of a kernel user. > The memory > is registered as being device accessible for an > indefinite period of time (must likely until the > machine reboots). Understood. Things leading up to and including Hotplug are generally the events that would cause us to shudown a card. > > extreme prejudice if the resources it's sitting on > > aren't free yet. I don't expect Hotplug support > > to be simple to implement. But it's definitely possible. > > Killing the task will not promptly cause RDMA activity > to cease. Not directly. It will cause files to get closed, memory to get free'dm and someplace along the way DMA regions to get unregistered. In other words, killing a consumer of a resource should result in the recovery of that resource. Maybe the libibverbs or libmthca can register a signal handler to trap such events and clean up properly. > The memory region must be deallocated or > the RDMA device handle must be released. So you're > back to how the kernel verbs are told of this. I don't know which mechanism can be used to communicate and handle RDMA region recovery. > In any event, what you describe would only work for > shutdown. It would not work to migrate pages. The > same interface can be used for both purposes. Ok. I'm only familiar with how to shutdown IO cards. If someone wants to (and can) implement something more sophisticated that's great. grant From tduffy at sun.com Mon Jun 6 16:48:45 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 06 Jun 2005 16:48:45 -0700 Subject: [openib-general] [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: <1118093694.5735.13.camel@duffman> References: <1118093694.5735.13.camel@duffman> Message-ID: <1118101725.30742.2.camel@duffman> Rebased against r2564. James, you think you will apply this? Does it need more work? Signed-off-by: Tom Duffy dat-provider/dapl.h | 258 ++++------- dat-provider/dapl_cr_accept.c | 106 +--- dat-provider/dapl_cr_callback.c | 147 ++---- dat-provider/dapl_cr_handoff.c | 2 dat-provider/dapl_cr_query.c | 10 dat-provider/dapl_cr_reject.c | 28 - dat-provider/dapl_cr_util.c | 21 dat-provider/dapl_ep.c | 605 +++++++++++--------------- dat-provider/dapl_evd.c | 175 +++---- dat-provider/dapl_evd.h | 6 dat-provider/dapl_get_consumer_context.c | 80 --- dat-provider/dapl_get_handle_type.c | 74 --- dat-provider/dapl_ia_close.c | 19 dat-provider/dapl_ia_memtype_hint.c | 10 dat-provider/dapl_ia_open.c | 62 +- dat-provider/dapl_ia_query.c | 20 dat-provider/dapl_ia_util.c | 180 +++---- dat-provider/dapl_lmr.c | 221 ++++----- dat-provider/dapl_mr_util.c | 2 dat-provider/dapl_openib_cm.c | 64 +- dat-provider/dapl_openib_qp.c | 10 dat-provider/dapl_openib_util.c | 12 dat-provider/dapl_openib_util.h | 14 dat-provider/dapl_provider.c | 5 dat-provider/dapl_psp.c | 105 ++-- dat-provider/dapl_pz.c | 79 +-- dat-provider/dapl_rmr.c | 78 +-- dat-provider/dapl_rsp.c | 87 +-- dat-provider/dapl_set_consumer_context.c | 74 --- dat-provider/dapl_sp_util.c | 46 - dat-provider/dapl_srq.c | 117 ++--- dat/api.c | 16 dat/dat.h | 469 ++++++++++---------- patches/alt_dat_provider_makefile | 3 test/dapltest/include/dapl_bpool.h | 10 test/dapltest/include/dapl_fft_util.h | 12 test/dapltest/include/dapl_performance_test.h | 14 test/dapltest/include/dapl_proto.h | 84 +-- test/dapltest/include/dapl_tdep.h | 12 test/dapltest/include/dapl_test_data.h | 20 test/dapltest/include/dapl_transaction_test.h | 18 test/dapltest/kdapl/kdapl_tdep_evd.c | 34 - test/dapltest/test/dapl_bpool.c | 38 - test/dapltest/test/dapl_client.c | 94 ++-- test/dapltest/test/dapl_cnxn.c | 4 test/dapltest/test/dapl_fft_connmgt.c | 4 test/dapltest/test/dapl_fft_dataxfer.c | 4 test/dapltest/test/dapl_fft_dataxfer_client.c | 12 test/dapltest/test/dapl_fft_endpoint.c | 118 ++--- test/dapltest/test/dapl_fft_hwconn.c | 22 test/dapltest/test/dapl_fft_mem.c | 42 - test/dapltest/test/dapl_fft_pz.c | 90 +-- test/dapltest/test/dapl_fft_queryinfo.c | 190 ++++---- test/dapltest/test/dapl_fft_util.c | 102 ++-- test/dapltest/test/dapl_limit.c | 204 ++++---- test/dapltest/test/dapl_performance_client.c | 24 - test/dapltest/test/dapl_performance_server.c | 68 +- test/dapltest/test/dapl_performance_util.c | 88 +-- test/dapltest/test/dapl_server.c | 102 ++-- test/dapltest/test/dapl_test_util.c | 108 ++-- test/dapltest/test/dapl_transaction_test.c | 164 +++---- test/dapltest/test/dapl_transaction_util.c | 50 +- 62 files changed, 2227 insertions(+), 2710 deletions(-) Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -33,7 +33,7 @@ boolean_t DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -61,10 +61,10 @@ DT_Performance_Test_Create ( test_ptr->remote_ia_addr = remote_ia_addr; test_ptr->is_remote_little_endian = is_remote_little_endian; test_ptr->base_port = (DAT_CONN_QUAL) pt_ptr->Server_Info.first_port_number; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; - ret = dat_ia_query (test_ptr->ia_handle, + ret = dat_ia_query (test_ptr->ia, NULL, &test_ptr->ia_attr, NULL); @@ -95,17 +95,17 @@ DT_Performance_Test_Create ( test_ptr->creq_evd_length = DT_PERF_DFLT_EVD_LENGTH; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; return FALSE; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->recv_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG, @@ -114,11 +114,11 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; return FALSE; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->reqt_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -127,7 +127,7 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: 0x%x dat_evd_create (request) error: %s\n", test_ptr->reqt_evd_length,test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; return FALSE; } @@ -135,30 +135,30 @@ DT_Performance_Test_Create ( if ( is_server ) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->creq_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CR_FLAG, &test_ptr->creq_evd_hdl); /* cr */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; return FALSE; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->conn_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CONNECTION_FLAG, &test_ptr->conn_evd_hdl); /* conn */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; return FALSE; } @@ -176,18 +176,18 @@ DT_Performance_Test_Create ( test_ptr->ep_context.ep_attr.max_request_dtos = pipeline_len; /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &test_ptr->ep_context.ep_attr, /* EP attrs */ - &test_ptr->ep_context.ep_handle); + &test_ptr->ep_context.ep); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->ep_context.ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context.ep = NULL; return FALSE; } @@ -197,10 +197,10 @@ DT_Performance_Test_Create ( */ test_ptr->ep_context.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, + NULL, /* rmr */ DT_PERF_SYNC_BUFF_SIZE, 2, /* 2 RMIs */ 255, /* FIXME should query for this */ @@ -226,7 +226,7 @@ DT_Performance_Test_Create ( * Post recv and sync buffers */ if ( !DT_post_recv_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_RECV_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE) ) @@ -269,13 +269,13 @@ DT_Performance_Test_Destroy ( boolean_t is_server) { u32 ret; - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; DT_Tdep_Print_Head *phead; int status = 0; phead = pt_ptr->Params.phead; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ if (test_ptr->ep_context.op.bp) @@ -310,9 +310,9 @@ DT_Performance_Test_Destroy ( * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context.ep_handle) + if (test_ptr->ep_context.ep) { - ret = dat_ep_disconnect (test_ptr->ep_context.ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context.ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -323,7 +323,7 @@ DT_Performance_Test_Destroy ( status = 1; } else if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -331,10 +331,10 @@ DT_Performance_Test_Destroy ( } } - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free error: %s\n", @@ -394,9 +394,9 @@ DT_Performance_Test_Destroy ( } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -417,7 +417,7 @@ DT_Performance_Test_Destroy ( boolean_t DT_performance_post_rdma_op ( Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats) { unsigned int j; @@ -456,7 +456,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -473,7 +473,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -502,17 +502,17 @@ DT_performance_post_rdma_op ( unsigned int DT_performance_reap ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Mode_Type mode, Performance_Stats_t *stats) { if ( BLOCKING_MODE == mode ) { - return DT_performance_wait (phead, evd_handle, stats); + return DT_performance_wait (phead, evd, stats); } else { - return DT_performance_poll (phead, evd_handle, stats); + return DT_performance_poll (phead, evd, stats); } } @@ -520,7 +520,7 @@ DT_performance_reap ( unsigned int DT_performance_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { int i; @@ -537,7 +537,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_wait ( evd_handle, + ret = DT_Tdep_evd_wait ( evd, DAT_TIMEOUT_MAX, &event); @@ -571,7 +571,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); @@ -611,7 +611,7 @@ DT_performance_wait ( unsigned int DT_performance_poll ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { u32 ret; @@ -626,7 +626,7 @@ DT_performance_poll ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: linux-kernel/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_client.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_performance_client.c (working copy) @@ -34,7 +34,7 @@ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr) { Performance_Test_t *test_ptr = NULL; @@ -47,7 +47,7 @@ DT_Performance_Test_Client ( DT_Tdep_PT_Debug (1,(phead,"Client: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - ia_handle, + ia, remote_ia_addr, FALSE, pt_ptr->Server_Info.is_little_endian, @@ -102,7 +102,7 @@ DT_Performance_Test_Client_Connect ( test_ptr->base_port, test_ptr->ep_context.port)); retry: - ret = dat_ep_connect (test_ptr->ep_context.ep_handle, + ret = dat_ep_connect (test_ptr->ep_context.ep, test_ptr->remote_ia_addr, test_ptr->ep_context.port, DAT_TIMEOUT_MAX, @@ -119,7 +119,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -297,7 +297,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -308,7 +308,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -375,9 +375,9 @@ DT_Performance_Test_Client_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -406,7 +406,7 @@ DT_Performance_Test_Client_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) @@ -461,7 +461,7 @@ DT_Performance_Test_Client_Exchange ( DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Sending Sync Msg\n", test_ptr->base_port)); if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -478,7 +478,7 @@ DT_Performance_Test_Client_Exchange ( if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Client_Sync_Send")) Index: linux-kernel/test/dapltest/test/dapl_cnxn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_cnxn.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_cnxn.c (working copy) @@ -29,7 +29,7 @@ /****************************************************************************/ int -get_ep_connection_state (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle) +get_ep_connection_state (DT_Tdep_Print_Head *phead, struct dat_ep *ep) { enum dat_ep_state ep_state; boolean_t in_dto_idle; @@ -39,7 +39,7 @@ get_ep_connection_state (DT_Tdep_Print_H char *req_status = "Idle"; - ret = dat_ep_get_status (ep_handle, &ep_state, &in_dto_idle, + ret = dat_ep_get_status (ep, &ep_state, &in_dto_idle, &out_dto_idle); if (ret != 0) { Index: linux-kernel/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_server.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_server.c (working copy) @@ -103,7 +103,7 @@ DT_cs_Server (Params_t * params_ptr) ret = dat_ia_open (Server_Cmd->dapl_name, DFLT_QLEN, &ps_ptr->async_evd_hdl, - &ps_ptr->ia_handle); + &ps_ptr->ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -111,28 +111,28 @@ DT_cs_Server (Params_t * params_ptr) module, Server_Cmd->dapl_name, DT_RetToString (ret)); - ps_ptr->ia_handle = DAT_HANDLE_NULL; + ps_ptr->ia = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: IA %s opened\n", module, Server_Cmd->dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ps_ptr->ia_handle, &ps_ptr->pz_handle); + ret = dat_pz_create (ps_ptr->ia, &ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->pz_handle = DAT_HANDLE_NULL; + ps_ptr->pz = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: PZ created\n", module)); /* Create 4 events - recv, request, connection-request, connect */ - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -142,11 +142,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->recv_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -156,11 +156,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->reqt_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CR_FLAG, @@ -170,11 +170,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (cr) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->creq_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -184,44 +184,44 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->conn_evd_hdl = NULL; status = 1; goto server_exit; } /* Create the EP */ - ret = dat_ep_create (ps_ptr->ia_handle, /* IA */ - ps_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (ps_ptr->ia, /* IA */ + ps_ptr->pz, /* PZ */ ps_ptr->recv_evd_hdl, /* recv */ ps_ptr->reqt_evd_hdl, /* request */ ps_ptr->conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ps_ptr->ep_handle); + &ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->ep_handle = DAT_HANDLE_NULL; + ps_ptr->ep = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: EP created\n", module)); /* Create PSP */ - ret = dat_psp_create (ps_ptr->ia_handle, + ret = dat_psp_create (ps_ptr->ia, SERVER_PORT_NUMBER, ps_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &ps_ptr->psp_handle); + &ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->psp_handle = DAT_HANDLE_NULL; + ps_ptr->psp = NULL; status = 1; goto server_exit; } @@ -233,10 +233,10 @@ DT_cs_Server (Params_t * params_ptr) */ ps_ptr->bpool = DT_BpoolAlloc (NULL, phead, - ps_ptr->ia_handle, - ps_ptr->pz_handle, - ps_ptr->ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ps_ptr->ia, + ps_ptr->pz, + ps_ptr->ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ 256, /* FIXME query for this value */ @@ -275,7 +275,7 @@ DT_cs_Server (Params_t * params_ptr) */ for (;/* EVER */;) { - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; @@ -299,7 +299,7 @@ DT_cs_Server (Params_t * params_ptr) /* Server_Info, Client_Info, Params set up below */ /* Gather whatever info we want about defaults */ - if (!DT_query (pt_ptr, ps_ptr->ia_handle, ps_ptr->ep_handle)) + if (!DT_query (pt_ptr, ps_ptr->ia, ps_ptr->ep)) { status = 1; goto server_exit; @@ -308,7 +308,7 @@ DT_cs_Server (Params_t * params_ptr) /* Post recv buffers for ClientInfo and Transaction_Cmd_t */ DT_Tdep_PT_Debug (1,(phead,"%s: Posting 2 recvs\n", module)); if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 0, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0))) @@ -320,7 +320,7 @@ DT_cs_Server (Params_t * params_ptr) goto server_exit; } if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 1, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1))) @@ -343,9 +343,9 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_cr_event_wait (phead, ps_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check (phead, &cr_stat, - ps_ptr->psp_handle, + ps_ptr->psp, SERVER_PORT_NUMBER, - &cr_handle, + &cr, module)) { @@ -357,7 +357,7 @@ DT_cs_Server (Params_t * params_ptr) } DT_Tdep_PT_Debug (1,(phead,"%s: Accepting Connection Request\n", module)); - ret = dat_cr_accept (cr_handle, ps_ptr->ep_handle, 0, (void *)0); + ret = dat_cr_accept (cr, ps_ptr->ep, 0, (void *)0); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -370,7 +370,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Debug (1,(phead,"%s: Awaiting connection ...\n", module)); if (!DT_conn_event_wait (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->conn_evd_hdl, &event_num)) { @@ -384,7 +384,7 @@ DT_cs_Server (Params_t * params_ptr) if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ps_ptr->ep_handle); + get_ep_connection_state (phead, ps_ptr->ep); } /* Wait for Client_Info */ @@ -395,7 +395,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0), dto_cookie, "Client_Info_Recv")) @@ -420,7 +420,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1), dto_cookie, "Client_Cmd_Recv")) @@ -585,7 +585,7 @@ DT_cs_Server (Params_t * params_ptr) /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -601,7 +601,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) @@ -619,7 +619,7 @@ DT_cs_Server (Params_t * params_ptr) /* we passed the pt_ptr to the thread and must now 'forget' it */ pt_ptr = NULL; - ret = dat_ep_disconnect (ps_ptr->ep_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect fails: %s\n", @@ -635,7 +635,7 @@ DT_cs_Server (Params_t * params_ptr) } /* reset the EP to get back into the game */ - dat_ep_reset (ps_ptr->ep_handle); + dat_ep_reset (ps_ptr->ep); DT_Tdep_PT_Debug (1,(phead,"%s: Waiting for another client...\n", module)); } /* end loop accepting connections */ @@ -669,9 +669,9 @@ server_exit: * we also get here on error, hence abrupt closure to * flush any lingering buffers posted. */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_disconnect (ps_ptr->ep_handle, + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -704,9 +704,9 @@ server_exit: } /* Free the PSP */ - if (ps_ptr->psp_handle) + if (ps_ptr->psp) { - ret = dat_psp_free (ps_ptr->psp_handle); + ret = dat_psp_free (ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_free error: %s\n", @@ -718,9 +718,9 @@ server_exit: } /* Free the EP */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_free (ps_ptr->ep_handle); + ret = dat_ep_free (ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free error: %s\n", @@ -781,9 +781,9 @@ server_exit: } /* Free the PZ */ - if (ps_ptr->pz_handle) + if (ps_ptr->pz) { - ret = dat_pz_free (ps_ptr->pz_handle); + ret = dat_pz_free (ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free error: %s\n", @@ -794,17 +794,17 @@ server_exit: } /* Close the IA */ - if (ps_ptr->ia_handle) + if (ps_ptr->ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) error: %s\n", module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -871,7 +871,7 @@ send_control_data ( sizeof (Server_Info_t)); DT_Server_Info_Endian ((Server_Info_t *) buffp); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -886,7 +886,7 @@ send_control_data ( if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) Index: linux-kernel/test/dapltest/test/dapl_fft_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) @@ -55,22 +55,22 @@ void DT_assert_fail (DT_Tdep_Print_Head } /* helper function to open an IA */ -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle) +int DT_ia_open (char *dev_name, struct dat_ia **ia) { - DAT_EVD_HANDLE evd_handle; - evd_handle = DAT_HANDLE_NULL; - return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, ia_handle); + struct dat_evd *evd; + evd = NULL; + return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, ia); } /* helper function to create an endpoint and its associated EVDs */ int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle) + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep) { u32 status; DT_Tdep_Print_Head *phead; @@ -80,7 +80,7 @@ int DT_ep_create (Params_t *params_ptr, *cr_evd = NULL; phead = params_ptr->phead; - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CR_FLAG, cr_evd); if (status != DAT_SUCCESS) { @@ -88,7 +88,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CONNECTION_FLAG, conn_evd); if (status != DAT_SUCCESS) { @@ -96,7 +96,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, send_evd); if (status != DAT_SUCCESS) @@ -105,7 +105,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, recv_evd); if (status != DAT_SUCCESS) { @@ -113,8 +113,8 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = dat_ep_create (ia_handle, pz_handle, *recv_evd, - *send_evd, *conn_evd, NULL, ep_handle); + status = dat_ep_create (ia, pz, *recv_evd, + *send_evd, *conn_evd, NULL, ep); if (status != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "dat_ep_create failed %s\n", DT_RetToString (status)); @@ -125,15 +125,15 @@ int DT_ep_create (Params_t *params_ptr, /* function that initializes the connection struct */ void DT_fft_init_conn_struct (FFT_Connection_t *conn) { - conn->ia_handle = NULL; - conn->pz_handle = NULL; - conn->psp_handle = NULL; - conn->ep_handle = NULL; + conn->ia = NULL; + conn->pz = NULL; + conn->psp = NULL; + conn->ep = NULL; conn->cr_evd = NULL; conn->send_evd = NULL; conn->conn_evd = NULL; conn->recv_evd = NULL; - conn->cr_handle = NULL; + conn->cr = NULL; conn->remote_netaddr = NULL; conn->bpool = NULL; conn->pt_ptr = NULL; @@ -154,27 +154,27 @@ void DT_fft_init_client (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); if (rc != DAT_SUCCESS) { /* make sure the handle has an invalid value */ - conn->ia_handle = NULL; + conn->ia = NULL; } DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* if a server name is given, allocate memory for a net address and set it @@ -198,11 +198,11 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_Print_Head *phead; phead = params_ptr->phead; - if (conn->ep_handle) + if (conn->ep) { if (conn->connected) { - rc = dat_ep_disconnect (conn->ep_handle, DAT_CLOSE_DEFAULT); + rc = dat_ep_disconnect (conn->ep, DAT_CLOSE_DEFAULT); DT_assert_clean (phead, rc == DAT_SUCCESS); if (!DT_disco_event_wait ( phead, conn->cr_evd, NULL )) @@ -211,16 +211,16 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_PT_Printf (phead, "DT_fft_destroy_conn_struct: bad disconnect event\n"); } } - rc = dat_ep_free (conn->ep_handle); + rc = dat_ep_free (conn->ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->bpool) { DT_Bpool_Destroy (NULL, phead, conn->bpool); } - if (conn->psp_handle) + if (conn->psp) { - rc = dat_psp_free (conn->psp_handle); + rc = dat_psp_free (conn->psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->cr_evd) @@ -247,14 +247,14 @@ int DT_fft_destroy_conn_struct (Params_t { DT_Free_Per_Test_Data (conn->pt_ptr); } - if (conn->pz_handle) + if (conn->pz) { - rc = dat_pz_free (conn->pz_handle); + rc = dat_pz_free (conn->pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn->ia_handle) + if (conn->ia) { - rc = dat_ia_close (conn->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (conn->ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return rc; @@ -272,31 +272,31 @@ void DT_fft_init_server (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PSP */ - rc = dat_psp_create (conn->ia_handle, SERVER_PORT_NUMBER, conn->cr_evd, - DAT_PSP_CONSUMER_FLAG, &conn->psp_handle); + rc = dat_psp_create (conn->ia, SERVER_PORT_NUMBER, conn->cr_evd, + DAT_PSP_CONSUMER_FLAG, &conn->psp); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate memory for buffers */ - conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia_handle, conn->pz_handle, + conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia, conn->pz, NULL, NULL, 8192, 2, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, conn->bpool); @@ -316,19 +316,19 @@ void DT_fft_listen (Params_t *params_ptr DT_assert_dat (phead, DT_cr_event_wait (phead, conn->cr_evd, &conn->cr_stat) && DT_cr_check (phead, &conn->cr_stat, - conn->psp_handle, + conn->psp, SERVER_PORT_NUMBER, - &conn->cr_handle, + &conn->cr, "DT_fft_listen")); /* accept the connection */ - rc =dat_cr_accept (conn->cr_handle, conn->ep_handle, 0, (void *)0); + rc =dat_cr_accept (conn->cr, conn->ep, 0, (void *)0); DT_assert_dat (phead, rc == DAT_SUCCESS); /* wait on a conn event via the conn EVD */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); conn->connected = TRUE; @@ -351,7 +351,7 @@ int DT_fft_connect (Params_t *params_ptr DT_Tdep_PT_Printf (phead, "Connection to server, attempt #%d\n", wait_count+1); /* attempt to connect, timeout = 10 secs */ - rc = dat_ep_connect (conn->ep_handle, conn->remote_netaddr, + rc = dat_ep_connect (conn->ep, conn->remote_netaddr, SERVER_PORT_NUMBER, 10*1000000, 0, (void *)0, DAT_QOS_BEST_EFFORT, DAT_CONNECT_DEFAULT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); @@ -359,7 +359,7 @@ int DT_fft_connect (Params_t *params_ptr /* wait on conn event */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); Index: linux-kernel/test/dapltest/test/dapl_fft_mem.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_mem.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_mem.c (working copy) @@ -37,28 +37,28 @@ int DT_mem_generic (Params_t *params_ptr FFT_Connection_t conn; DAT_REGION_DESCRIPTION region; u64 reg_size; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; DAT_LMR_CONTEXT lmr_context; u64 reg_addr; unsigned char *alloc_ptr; int res; u64 buffer_size; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; rc = 0; expect = 0; res = 1; - lmr_handle = NULL; + lmr = NULL; lmr_context = 0; reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; DT_fft_init_client (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); if (flag == 2) { @@ -76,24 +76,24 @@ int DT_mem_generic (Params_t *params_ptr memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - ia_handle = conn.ia_handle; + ia = conn.ia; if (flag != 3) { - pz_handle = conn.pz_handle; + pz = conn.pz; } if (flag != 4) { DT_Tdep_PT_Printf (phead, "Registering memory\n"); - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -110,21 +110,21 @@ int DT_mem_generic (Params_t *params_ptr } if (flag == 1) { - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_dat (phead, rc == DAT_SUCCESS); } - lmr_handle = NULL; + lmr = NULL; - rc = dat_lmr_kcreate (conn.ia_handle, + rc = dat_lmr_kcreate (conn.ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -133,9 +133,9 @@ int DT_mem_generic (Params_t *params_ptr } cleanup: - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (alloc_ptr) @@ -197,7 +197,7 @@ int DT_mem_case4 ( Params_t *params_ptr, DT_Tdep_Print_Head *phead; phead = params_ptr->phead; DT_Tdep_PT_Printf (phead, "\ - Description: Try to deregister memory with null lmr_handle\n"); + Description: Try to deregister memory with null lmr\n"); return DT_mem_generic (params_ptr, cmd, 4); } Index: linux-kernel/test/dapltest/test/dapl_limit.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) @@ -33,13 +33,13 @@ static boolean_t more_handles (DT_Tdep_Print_Head *phead, - DAT_HANDLE **old_ptrptr, /* pointer to current pointer */ + void ***old_ptrptr, /* pointer to current pointer */ unsigned int *old_count, /* number pointed to */ unsigned int size) /* size of one datum */ { unsigned int count = *old_count; - DAT_HANDLE *old_handles = *old_ptrptr; - DAT_HANDLE *handle_tmp = DT_Mdep_Malloc (count * 2 * size); + void * *old_handles = *old_ptrptr; + void * *handle_tmp = DT_Mdep_Malloc (count * 2 * size); if (!handle_tmp) { @@ -74,19 +74,19 @@ limit_test ( DT_Tdep_Print_Head *phead, Limit_Cmd_t *cmd, Limit_Index depth) { - DAT_EVD_HANDLE conn_handle; + struct dat_evd * conn_handle; typedef struct obj_set { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; - DAT_LMR_HANDLE lmr_handle; + struct dat_evd * evd; + struct dat_ep * ep; + struct dat_lmr * lmr; char * lmr_buffer; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; } Obj_Set; @@ -135,19 +135,19 @@ limit_test ( DT_Tdep_Print_Head *phead, for (w = 0; w < cmd->width; w++) { /* Specify that we want to get back an async EVD. */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdl_sets[w].ia_async_handle, - &hdl_sets[w].ia_handle); + &hdl_sets[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", module, cmd->device_name, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; - hdl_sets[w].ia_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; + hdl_sets[w].ia = NULL; goto clean_up_now; } } @@ -159,8 +159,8 @@ limit_test ( DT_Tdep_Print_Head *phead, */ typedef struct _ia { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; } OneOpen; unsigned int count = START_COUNT; @@ -178,7 +178,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -187,11 +187,11 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } /* Specify that we want to get back an async EVD. */ - hdlptr[w].ia_async_handle = DAT_HANDLE_NULL; + hdlptr[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdlptr[w].ia_async_handle, - &hdlptr[w].ia_handle); + &hdlptr[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", @@ -209,14 +209,14 @@ limit_test ( DT_Tdep_Print_Head *phead, for (tmp = 0; tmp < w; tmp++) { DT_Mdep_Schedule(); - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) fails: %s\n", module, DT_RetToString (ret)); retval = FALSE; - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -245,14 +245,14 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_pz_create (hdl_sets[w].ia_handle, - &hdl_sets[w].pz_handle); + ret = dat_pz_create (hdl_sets[w].ia, + &hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].pz_handle = DAT_HANDLE_NULL; + hdl_sets[w].pz = NULL; goto clean_up_now; } } @@ -263,7 +263,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PZs we can create */ unsigned int count = START_COUNT; - DAT_PZ_HANDLE *hdlptr = (DAT_PZ_HANDLE *) + struct dat_pz **hdlptr = (struct dat_pz **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PZ Exhaustion test loop */ @@ -279,7 +279,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -287,7 +287,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_pz_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_pz_create (hdl_sets[w % cmd->width].ia, &hdlptr[w]); if (ret != DAT_SUCCESS) { @@ -335,13 +335,13 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_cno_create (hdl_sets[w].ia_handle, + ret = dat_cno_create (hdl_sets[w].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdl_sets[w].cno_handle); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { DT_Tdep_PT_Printf (phead, "%s: dat_cno_create unimplemented\n", module); - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; /* ignore this error */ break; } @@ -350,7 +350,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_cno_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; goto clean_up_now; } } @@ -377,7 +377,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -385,7 +385,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_cno_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_cno_create (hdl_sets[w % cmd->width].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) @@ -446,7 +446,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -456,22 +456,22 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; goto clean_up_now; } for (w = 0; w < cmd->width; w++) { - ret = DT_Tdep_evd_create (hdl_sets[w].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w].ia, DFLT_QLEN, hdl_sets[w].cno_handle, flags, - &hdl_sets[w].evd_handle); + &hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].evd_handle = DAT_HANDLE_NULL; + hdl_sets[w].evd = NULL; goto clean_up_now; } } @@ -482,7 +482,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EVDs we can create */ unsigned int count = START_COUNT; - DAT_EVD_HANDLE *hdlptr = (DAT_EVD_HANDLE *) + struct dat_evd **hdlptr = (struct dat_evd **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); enum dat_evd_flags flags = ( DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG @@ -500,7 +500,7 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -510,13 +510,13 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->maximum; w++) { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -524,7 +524,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia, DFLT_QLEN, hdl_sets[w % cmd->width].cno_handle, flags, @@ -542,10 +542,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; /* EVD Cleanup loop */ - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (tmp = 0; tmp < w; tmp++) { @@ -578,19 +578,19 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_ep_create (hdl_sets[w].ia_handle, - hdl_sets[w].pz_handle, - hdl_sets[w].evd_handle, /* recv */ - hdl_sets[w].evd_handle, /* request */ + ret = dat_ep_create (hdl_sets[w].ia, + hdl_sets[w].pz, + hdl_sets[w].evd, /* recv */ + hdl_sets[w].evd, /* request */ conn_handle, /* connect */ (struct dat_ep_attr *) NULL, - &hdl_sets[w].ep_handle); + &hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ep_handle = DAT_HANDLE_NULL; + hdl_sets[w].ep = NULL; goto clean_up_now; } } @@ -601,7 +601,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EPs we can create */ unsigned int count = START_COUNT; - DAT_EP_HANDLE *hdlptr = (DAT_EP_HANDLE *) + struct dat_ep **hdlptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* EP Exhaustion test loop */ @@ -615,7 +615,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -623,10 +623,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, /* connect */ (struct dat_ep_attr *) NULL, &hdlptr[w]); @@ -672,9 +672,9 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many RSPs we can create */ unsigned int count = START_COUNT; - DAT_RSP_HANDLE *hdlptr = (DAT_RSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); - DAT_EP_HANDLE *epptr = (DAT_EP_HANDLE *) + struct dat_ep **epptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*epptr)); /* RSP Exhaustion test loop */ @@ -692,7 +692,7 @@ limit_test ( DT_Tdep_Print_Head *phead, unsigned int count1 = count; unsigned int count2 = count; - if (!more_handles (phead, (DAT_HANDLE **) &hdlptr, + if (!more_handles (phead, (void ***) &hdlptr, &count1, sizeof (*hdlptr))) { @@ -700,7 +700,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - if (!more_handles (phead, (DAT_HANDLE **) &epptr, + if (!more_handles (phead, (void ***) &epptr, &count2, sizeof (*epptr))) { @@ -723,10 +723,10 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * Each RSP needs a unique EP, so create one first */ - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, (struct dat_ep_attr *) NULL, &epptr[w]); @@ -738,10 +738,10 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } - ret = dat_rsp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_rsp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, epptr[w], - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { @@ -808,7 +808,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PSPs we can create */ unsigned int count = START_COUNT; - DAT_PSP_HANDLE *hdlptr = (DAT_PSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PSP Exhaustion test loop */ @@ -822,7 +822,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -830,9 +830,9 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_psp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_psp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, DAT_PSP_CONSUMER_FLAG, &hdlptr[w]); if (ret != DAT_SUCCESS) @@ -840,7 +840,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_psp_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); retval = TRUE; - hdlptr[w] = DAT_HANDLE_NULL; + hdlptr[w] = NULL; break; } } @@ -897,14 +897,14 @@ limit_test ( DT_Tdep_Print_Head *phead, memset (®ion, 0, sizeof (region)); region.for_va = hdl_sets[w].lmr_buffer; - ret = dat_lmr_kcreate (hdl_sets[w].ia_handle, + ret = dat_lmr_kcreate (hdl_sets[w].ia, DAT_MEM_TYPE_VIRTUAL, region, DFLT_BUFFSZ, - hdl_sets[w].pz_handle, + hdl_sets[w].pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &hdl_sets[w].lmr_handle, + &hdl_sets[w].lmr, &hdl_sets[w].lmr_context, NULL, /* FIXME */ ®_size, ®_addr); @@ -913,7 +913,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_lmr_kcreate #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].lmr_handle = DAT_HANDLE_NULL; + hdl_sets[w].lmr = NULL; goto clean_up_now; } if ((uintptr_t)reg_addr > (uintptr_t)hdl_sets[w].lmr_buffer @@ -949,7 +949,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -966,10 +966,10 @@ limit_test ( DT_Tdep_Print_Head *phead, */ hdlptr[w] = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].ep_handle, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].ep, + hdl_sets[w % cmd->width].evd, DFLT_BUFFSZ, 1, 256, /* FIXME should query for this */ @@ -1026,7 +1026,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, cmd->width * sizeof (*hdlptr))) { @@ -1054,7 +1054,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_ep_post_recv #%d\n", module, w * cmd->width + i + 1); - ret = dat_ep_post_recv (hdl_sets[i].ep_handle, + ret = dat_ep_post_recv (hdl_sets[i].ep, 1, iovp, cookie, @@ -1091,7 +1091,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * outstanding recv DTOs in error, and otherwise * be a no-op. */ - ret = dat_ep_reset (hdl_sets[i].ep_handle); + ret = dat_ep_reset (hdl_sets[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect (abrupt) fails: %s\n", @@ -1107,7 +1107,7 @@ limit_test ( DT_Tdep_Print_Head *phead, */ do { - ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd_handle, + ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd, &event); } while (ret == DAT_SUCCESS); } @@ -1130,10 +1130,10 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Mdep_Schedule(); test_bpool = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[0].ia_handle, - hdl_sets[0].pz_handle, - hdl_sets[0].ep_handle, - hdl_sets[0].evd_handle, + hdl_sets[0].ia, + hdl_sets[0].pz, + hdl_sets[0].ep, + hdl_sets[0].evd, test_size, 1, 256, /* FIXME should query for this */ @@ -1185,9 +1185,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].lmr_handle) + if (hdl_sets[w].lmr) { - ret = dat_lmr_free (hdl_sets[w].lmr_handle); + ret = dat_lmr_free (hdl_sets[w].lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_lmr_free fails: %s\n", @@ -1220,9 +1220,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ep_handle) + if (hdl_sets[w].ep) { - ret = dat_ep_free (hdl_sets[w].ep_handle); + ret = dat_ep_free (hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free fails: %s\n", @@ -1237,16 +1237,16 @@ clean_up_now: { unsigned int w; - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].evd_handle) + if (hdl_sets[w].evd) { - ret = DT_Tdep_evd_free (hdl_sets[w].evd_handle); + ret = DT_Tdep_evd_free (hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_free fails: %s\n", @@ -1284,9 +1284,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].pz_handle) + if (hdl_sets[w].pz) { - ret = dat_pz_free (hdl_sets[w].pz_handle); + ret = dat_pz_free (hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free fails: %s\n", @@ -1303,10 +1303,10 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ia_handle) + if (hdl_sets[w].ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { @@ -1318,7 +1318,7 @@ clean_up_now: * we may as well try the largest hammer we have. */ retval = FALSE; - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { Index: linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -37,31 +37,31 @@ int DT_queryinfo_basic (Params_t *params u32 result_wanted) { char *dev_name; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; struct dat_ia_attr ia_attributes; struct dat_provider_attr provider_attributes; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE cr_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *cr_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; + struct dat_ep *ep; struct dat_ep_param ep_param; DAT_CNO_HANDLE cno_handle; #ifndef __KDAPLTEST__ DAT_CNO_PARAM cno_param; #endif struct dat_evd_param evd_param; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * psp; struct dat_psp_param psp_param; - DAT_RSP_HANDLE rsp_handle; + struct dat_sp * rsp; struct dat_rsp_param rsp_param; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; struct dat_pz_param pz_param; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; struct dat_lmr_param lmr_param; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; struct dat_rmr_param rmr_param; DAT_REGION_DESCRIPTION region; u64 reg_size; @@ -77,30 +77,30 @@ int DT_queryinfo_basic (Params_t *params reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - lmr_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + lmr = NULL; rmr_handle = NULL; - pz_handle = NULL; - psp_handle = NULL; - rsp_handle = NULL; + pz = NULL; + psp = NULL; + rsp = NULL; cno_handle = NULL; - evd_handle = DAT_HANDLE_NULL; - conn_evd_handle = DAT_HANDLE_NULL; - cr_evd_handle = DAT_HANDLE_NULL; - recv_evd_handle = DAT_HANDLE_NULL; - send_evd_handle = DAT_HANDLE_NULL; + evd = NULL; + conn_evd = NULL; + cr_evd = NULL; + recv_evd = NULL; + send_evd = NULL; dev_name = cmd->device_name; - /* All functions require an ia_handle to be created */ + /* All functions require an ia to be created */ rc = dat_ia_open ((const char *)dev_name, DEFAULT_QUEUE_LEN, - &evd_handle, - &ia_handle); + &evd, + &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - /* These functions require a pz_handle to be created */ + /* These functions require a pz to be created */ if ( (object_to_query == QUERY_EVD) || (object_to_query == QUERY_RMR) || (object_to_query == QUERY_LMR) || @@ -108,43 +108,43 @@ int DT_queryinfo_basic (Params_t *params (object_to_query == QUERY_RSP) || (object_to_query == QUERY_PZ) ) { - rc = dat_pz_create (ia_handle, - &pz_handle); + rc = dat_pz_create (ia, + &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - /* These functions require a ep_handle to be created */ + /* These functions require a ep to be created */ if ( (object_to_query == QUERY_EP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CONNECTION_FLAG, - &conn_evd_handle); + &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, - pz_handle, - recv_evd_handle, - send_evd_handle, - conn_evd_handle, + rc = dat_ep_create (ia, + pz, + recv_evd, + send_evd, + conn_evd, NULL, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -152,11 +152,11 @@ int DT_queryinfo_basic (Params_t *params if ( (object_to_query == QUERY_PSP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CR_FLAG, - &cr_evd_handle); + &cr_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -165,8 +165,8 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -177,15 +177,15 @@ int DT_queryinfo_basic (Params_t *params * NULL out ia_attr and for the DAT_IA_ATTR_MASK to * have values */ - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, NULL, &provider_attributes); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_ia_query (evd_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -197,11 +197,11 @@ int DT_queryinfo_basic (Params_t *params #ifndef __KDAPLTEST__ #if defined(WIN32) - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, NULLPROXY, &cno_handle); #else - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &cno_handle); #endif @@ -220,7 +220,7 @@ int DT_queryinfo_basic (Params_t *params } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_cno_query (ia_handle, + rc = dat_cno_query (ia, &cno_param); } #endif @@ -230,17 +230,17 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, &evd_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_evd_query (ia_handle, + rc = dat_evd_query (evd, &evd_param); } } @@ -248,25 +248,25 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_psp_query function */ else if (object_to_query == QUERY_PSP) { - rc = dat_psp_create (ia_handle, + rc = dat_psp_create (ia, SERVER_PORT_NUMBER, - cr_evd_handle, + cr_evd, DAT_PSP_PROVIDER_FLAG, - &psp_handle); + &psp); DT_assert_dat (phead, rc == DAT_SUCCESS); if (result_wanted == DAT_SUCCESS) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, &psp_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_psp_query (evd_handle, + rc = dat_psp_query (psp, &psp_param); } } @@ -274,13 +274,13 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_rsp_query function */ else if (object_to_query == QUERY_RSP) { - rc = dat_rsp_create (ia_handle, + rc = dat_rsp_create (ia, SERVER_PORT_NUMBER, - ep_handle, - cr_evd_handle, - &rsp_handle); + ep, + cr_evd, + &rsp); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_rsp_query (rsp_handle, + rc = dat_rsp_query (rsp, &rsp_param); } @@ -294,14 +294,14 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_ep_query function */ else if (object_to_query == QUERY_EP) { - rc = dat_ep_query (ep_handle, + rc = dat_ep_query (ep, &ep_param); } /* Test dat_pz_query function */ else if (object_to_query == QUERY_PZ) { - rc = dat_pz_query (pz_handle, + rc = dat_pz_query (pz, &pz_param); } @@ -312,27 +312,27 @@ int DT_queryinfo_basic (Params_t *params DT_assert (phead, alloc_ptr); memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, ®_addr); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_lmr_query (lmr_handle, + rc = dat_lmr_query (lmr, &lmr_param); } /* Test dat_rmr_query function */ else if (object_to_query == QUERY_RMR) { - rc = dat_rmr_create (pz_handle, + rc = dat_rmr_create (pz, &rmr_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_rmr_query (rmr_handle, @@ -342,39 +342,39 @@ int DT_queryinfo_basic (Params_t *params DT_assert_dat (phead, DAT_GET_TYPE (rc) == result_wanted); cleanup: - if (rsp_handle) + if (rsp) { - rc = dat_rsp_free (rsp_handle); + rc = dat_rsp_free (rsp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -391,27 +391,27 @@ cleanup: DT_assert_clean (phead, rc == DAT_SUCCESS); } #endif - if (psp_handle) + if (psp) { - rc = dat_psp_free (psp_handle); + rc = dat_psp_free (psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (cr_evd_handle) + if (cr_evd) { - rc = DT_Tdep_evd_free (cr_evd_handle); + rc = DT_Tdep_evd_free (cr_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -47,7 +47,7 @@ /****************************************************************************/ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr) { Transaction_Cmd_t *cmd = &pt_ptr->Params.u.Transaction_Cmd; @@ -68,7 +68,7 @@ DT_Transaction_Test_Client (Per_Test_Dat DT_Tdep_PT_Debug (1,(phead,"Client: Starting Client side of test\n")); if (!DT_Transaction_Create_Test (pt_ptr, - ia_handle, + ia, FALSE, port_num, pt_ptr->Server_Info.is_little_endian, @@ -110,7 +110,7 @@ DT_Transaction_Test_Server (void *params + i * cmd->eps_per_thread; if (!DT_Transaction_Create_Test (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, TRUE, port_num, pt_ptr->Client_Info.is_little_endian, @@ -156,7 +156,7 @@ DT_Transaction_Test_Server (void *params boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -181,7 +181,7 @@ DT_Transaction_Create_Test (Per_Test_Dat test_ptr->remote_is_little_endian = remote_is_little_endian; test_ptr->is_server = is_server; test_ptr->pt_ptr = pt_ptr; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->base_port = (DAT_CONN_QUAL) port_num; test_ptr->cmd = &pt_ptr->Params.u.Transaction_Cmd; test_ptr->time_out = DFLT_TMO * 1000; /* DFLT_TMO seconds */ @@ -244,18 +244,18 @@ DT_Transaction_Main (void *param) private_data_str = "DAPL and RDMA rule! Test 4321."; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; status = 1; goto test_failure; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG, @@ -264,12 +264,12 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; status = 1; goto test_failure; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -278,7 +278,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (request) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; status = 1; goto test_failure; } @@ -286,7 +286,7 @@ DT_Transaction_Main (void *param) if (pt_ptr->local_is_server) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CR_FLAG, @@ -295,13 +295,13 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; status = 1; goto test_failure; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CONNECTION_FLAG, @@ -310,7 +310,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; status = 1; goto test_failure; } @@ -363,18 +363,18 @@ DT_Transaction_Main (void *param) } /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &ep_attr, /* EP attrs */ - &test_ptr->ep_context[i].ep_handle); + &test_ptr->ep_context[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - test_ptr->ep_context[i].ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context[i].ep = NULL; status = 1; goto test_failure; } @@ -385,10 +385,10 @@ DT_Transaction_Main (void *param) */ test_ptr->ep_context[i].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ buff_size, 4, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -423,7 +423,7 @@ DT_Transaction_Main (void *param) * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -433,7 +433,7 @@ DT_Transaction_Main (void *param) goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -457,11 +457,11 @@ DT_Transaction_Main (void *param) * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->creq_evd_hdl, - &test_ptr->ep_context[i].rsp_handle); + &test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create #%d error: %s\n", @@ -472,11 +472,11 @@ DT_Transaction_Main (void *param) } else { - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &test_ptr->ep_context[i].psp_handle); + &test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create #%d error: %s\n", @@ -519,7 +519,7 @@ DT_Transaction_Main (void *param) RemoteMemoryInfo *RemoteMemInfo; struct dat_dto_completion_event_data dto_stat; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; /* * Establish the connection @@ -538,16 +538,16 @@ DT_Transaction_Main (void *param) &cr_stat) || !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].rsp_handle, + test_ptr->ep_context[i].rsp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error:(%x) %s\n", @@ -571,21 +571,21 @@ DT_Transaction_Main (void *param) } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, + ret = dat_cr_accept (cr, NULL, /* NULL for RSP */ 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -594,7 +594,7 @@ DT_Transaction_Main (void *param) goto test_failure; } /* throw away single-use PSP */ - ret = dat_rsp_free (test_ptr->ep_context[i].rsp_handle); + ret = dat_rsp_free (test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free #%d error: %s\n", @@ -622,16 +622,16 @@ DT_Transaction_Main (void *param) if ( !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].psp_handle, + test_ptr->ep_context[i].psp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error: %s\n", @@ -656,33 +656,33 @@ DT_Transaction_Main (void *param) /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context[i].ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context[i].ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ - (void) dat_psp_free (test_ptr->ep_context[i].psp_handle); + /* cr consumed on failure */ + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { /* error message printed by DT_cr_event_wait */ - (void) dat_psp_free (&test_ptr->ep_context[i].psp_handle); + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* throw away single-use PSP */ - ret = dat_psp_free (test_ptr->ep_context[i].psp_handle); + ret = dat_psp_free (test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free #%d error: %s\n", @@ -709,7 +709,7 @@ DT_Transaction_Main (void *param) test_ptr->ep_context[i].ia_port)); retry: - ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_connect (test_ptr->ep_context[i].ep, test_ptr->remote_ia_addr, test_ptr->ep_context[i].ia_port, DAT_TIMEOUT_MAX, @@ -726,7 +726,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -742,7 +742,7 @@ retry: struct dat_event event; int drained = 0; - dat_ep_reset (test_ptr->ep_context[i].ep_handle); + dat_ep_reset (test_ptr->ep_context[i].ep); do { ret = DT_Tdep_evd_dequeue ( test_ptr->recv_evd_hdl, @@ -756,7 +756,7 @@ retry: * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -766,7 +766,7 @@ retry: goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -835,9 +835,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -871,9 +871,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -906,10 +906,10 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -976,7 +976,7 @@ retry: /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_SEND_BUFFER_ID, buff_size)) @@ -993,7 +993,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Send" @@ -1017,7 +1017,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Recv" @@ -1082,9 +1082,9 @@ test_failure: /* Foreach EP */ for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ for (j = 0; j < test_ptr->cmd->num_ops; j++) @@ -1122,9 +1122,9 @@ test_failure: * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context[i].ep_handle) + if (test_ptr->ep_context[i].ep) { - ret = dat_ep_disconnect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context[i].ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -1148,7 +1148,7 @@ test_failure: { if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -1164,16 +1164,16 @@ test_failure: */ for (j = 0; j < test_ptr->cmd->eps_per_thread; j++) { - if ( test_ptr->ep_context[j].ep_handle == ep_handle ) + if ( test_ptr->ep_context[j].ep == ep ) { - test_ptr->ep_context[j].ep_handle = NULL; + test_ptr->ep_context[j].ep = NULL; } } } } else /* !success - QP may be in error state */ { - ep_handle = test_ptr->ep_context[i].ep_handle; + ep = test_ptr->ep_context[i].ep; } /* @@ -1182,7 +1182,7 @@ test_failure: * disconnected as we are racing with the remote side * disconnects. */ - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { struct dat_event event; /* @@ -1195,7 +1195,7 @@ test_failure: &event); } while (ret == DAT_SUCCESS); /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free #%d error: %s\n", @@ -1259,9 +1259,9 @@ test_failure: } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -1357,7 +1357,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1432,7 +1432,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1923,8 +1923,8 @@ DT_Print_Transaction_Test (DT_Tdep_Print test_ptr->remote_is_little_endian); DT_Tdep_PT_Printf (phead, "TransTest.base_port : " F64x "\n", test_ptr->base_port); - DT_Tdep_PT_Printf (phead, "TransTest.pz_handle : %p\n", - test_ptr->pz_handle); + DT_Tdep_PT_Printf (phead, "TransTest.pz : %p\n", + test_ptr->pz); /* statistics */ DT_Tdep_PT_Printf (phead, "TransTest.bytes_send : %d\n", test_ptr->stats.stat_bytes_send); Index: linux-kernel/test/dapltest/test/dapl_performance_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_server.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_performance_server.c (working copy) @@ -41,7 +41,7 @@ DT_Performance_Test_Server ( DT_Tdep_PT_Debug (1,(phead,"Server: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, (struct sockaddr *) 0, TRUE, pt_ptr->Client_Info.is_little_endian, @@ -97,15 +97,15 @@ DT_Performance_Test_Server_Connect ( { u32 ret; boolean_t status; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; enum dat_event_number event_num; - rsp_handle = DAT_HANDLE_NULL; - psp_handle = DAT_HANDLE_NULL; + rsp = NULL; + psp = NULL; #if 0 /* FIXME */ if (test_ptr->cmd->use_rsp) { @@ -113,11 +113,11 @@ DT_Performance_Test_Server_Connect ( * Server - create a single-use RSP and * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context.port, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->creq_evd_hdl, - &rsp_handle); + &rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create error: %s\n", @@ -132,9 +132,9 @@ DT_Performance_Test_Server_Connect ( /* wait for the connection request */ if (!DT_cr_event_wait (test_ptr->conn_evd_hdl, &cr_stat) || !DT_cr_check ( &cr_stat, - DAT_HANDLE_NULL, + NULL, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -142,20 +142,20 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ - if (!DT_conn_event_wait ( test_ptr->ep_context.ep_handle, + if (!DT_conn_event_wait ( test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -173,17 +173,17 @@ DT_Performance_Test_Server_Connect ( */ status = TRUE; - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context.port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &psp_handle); + &psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); status = FALSE; - psp_handle = DAT_HANDLE_NULL; + psp = NULL; return (status); } @@ -204,9 +204,9 @@ DT_Performance_Test_Server_Connect ( if (!DT_cr_event_wait (phead, test_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check ( phead, &cr_stat, - psp_handle, + psp, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -214,22 +214,22 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num ) ) { @@ -241,10 +241,10 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_PT_Debug (1, (phead,"Server[" F64x "]: Accept on port 0x" F64x "\n", test_ptr->base_port, test_ptr->ep_context.port)); psp_free: - if ( DAT_HANDLE_NULL != psp_handle ) + if ( NULL != psp ) { /* throw away single-use PSP */ - ret = dat_psp_free (psp_handle); + ret = dat_psp_free (psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free error: %s\n", @@ -252,10 +252,10 @@ psp_free: status = FALSE; } } - if ( DAT_HANDLE_NULL != rsp_handle ) + if ( NULL != rsp ) { /* throw away single-use PSP */ - ret = dat_rsp_free (rsp_handle); + ret = dat_rsp_free (rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free error: %s\n", @@ -282,9 +282,9 @@ DT_Performance_Test_Server_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -340,7 +340,7 @@ DT_Performance_Test_Server_Exchange ( /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -358,7 +358,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Send Sync_Msg") ) @@ -380,7 +380,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (working copy) @@ -40,7 +40,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ u32 rc=0; DT_fft_init_client (phead, cmd, &conn); - DT_assert_dat (phead, conn.ia_handle != NULL) + DT_assert_dat (phead, conn.ia != NULL) DT_assert (phead, DT_fft_connect (phead, &conn)); @@ -58,8 +58,8 @@ int DT_dataxfer_client_generic (DT_Tdep_ { conn.bpool = DT_BpoolAlloc (0, phead, - conn.ia_handle, - conn.pz_handle, + conn.ia, + conn.pz, NULL, NULL, 4096, @@ -68,7 +68,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ FALSE, FALSE); DT_assert (phead, conn.bpool != 0); - rc = DT_post_send_buffer (phead, conn.ep_handle, conn.bpool, 0, + rc = DT_post_send_buffer (phead, conn.ep, conn.bpool, 0, DT_Bpool_GetBuffSize (conn.bpool, 0)); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_evd_wait (conn.send_evd, 10*1000000, 1, &conn.event, @@ -80,11 +80,11 @@ int DT_dataxfer_client_generic (DT_Tdep_ /* cleanup */ cleanup: - if (conn.ep_handle) + if (conn.ep) { /* disconnect */ DT_Tdep_PT_Printf (phead, "Disconnect\n"); - rc = dat_ep_disconnect (conn.ep_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ep_disconnect (conn.ep, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } rc = DT_fft_destroy_conn_struct (phead, &conn); Index: linux-kernel/test/dapltest/test/dapl_fft_connmgt.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (working copy) @@ -39,7 +39,7 @@ int DT_connmgt_case0 (Params_t *params_p Description: Ensure time in dat_evd_wait works correctly\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 10000, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); @@ -62,7 +62,7 @@ int DT_connmgt_case1 (Params_t *params_p Description: Attempt to use timeout of 0 in dat_evd_wait\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 0, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -86,10 +86,10 @@ Bpool * DT_BpoolAlloc ( Per_Test_Data_t *pt_ptr, DT_Tdep_Print_Head *phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia * ia, + struct dat_pz * pz, + struct dat_ep * ep, + struct dat_evd * rmr_evd, int seg_size, int num_segs, int alignment, @@ -131,16 +131,16 @@ DT_BpoolAlloc ( bpool_ptr->alloc_ptr = alloc_ptr; bpool_ptr->alloc_size = alloc_size; - bpool_ptr->pz_handle = pz_handle; + bpool_ptr->pz = pz; bpool_ptr->num_segs = num_segs; - bpool_ptr->ep_handle = ep_handle; + bpool_ptr->ep = ep; bpool_ptr->buffer_size = seg_size * num_segs; bpool_ptr->buffer_start = DT_AlignPtr (alloc_ptr, alignment) + GG_ALLOC_OFFSET; bpool_ptr->tripl_start = (struct dat_lmr_triplet *) (bpool_ptr + 1); bpool_ptr->seg_size = seg_size; bpool_ptr->enable_rdma_write = enable_rdma_write; bpool_ptr->enable_rdma_read = enable_rdma_read; - bpool_ptr->rmr_evd_handle = rmr_evd_handle; + bpool_ptr->rmr_evd = rmr_evd; DT_Tdep_PT_Debug (3, (phead, "lmr_create [%p, " F64x "]\n", @@ -160,14 +160,14 @@ DT_BpoolAlloc ( region.for_pa = virt_to_phys(region.for_va); } - ret = dat_lmr_kcreate (ia_handle, + ret = dat_lmr_kcreate (ia, DT_mem_type, region, bp_len, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &bpool_ptr->lmr_handle, + &bpool_ptr->lmr, &bpool_ptr->lmr_context, &bpool_ptr->rmr_context, &bpool_ptr->reg_size, @@ -213,7 +213,7 @@ DT_BpoolAlloc ( struct dat_rmr_bind_completion_event_data rmr_stat; /* create the RMR */ - ret = dat_rmr_create (pz_handle, &bpool_ptr->rmr_handle); + ret = dat_rmr_create (pz, &bpool_ptr->rmr_handle); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_rmr_create failed %s\n", @@ -238,7 +238,7 @@ DT_BpoolAlloc ( ret = dat_rmr_bind ( bpool_ptr->rmr_handle, &iov, mflags, - bpool_ptr->ep_handle, + bpool_ptr->ep, cookie, DAT_COMPLETION_DEFAULT_FLAG, &bpool_ptr->rmr_context); @@ -253,7 +253,7 @@ DT_BpoolAlloc ( /* await the bind result */ if (!DT_rmr_event_wait (phead, - bpool_ptr->rmr_evd_handle, + bpool_ptr->rmr_evd, &rmr_stat) || !DT_rmr_check (phead, &rmr_stat, @@ -290,9 +290,9 @@ err: DT_RetToString (ret)); } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - ret = dat_lmr_free (bpool_ptr->lmr_handle); + ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -357,9 +357,9 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - u32 ret = dat_lmr_free (bpool_ptr->lmr_handle); + u32 ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -433,8 +433,8 @@ DT_Bpool_print (DT_Tdep_Print_Head *phea "BPOOL alloc_size %x\n", (int) bpool_ptr->alloc_size); DT_Tdep_PT_Printf (phead, - "BPOOL pz_handle %p\n", - bpool_ptr->pz_handle); + "BPOOL pz %p\n", + bpool_ptr->pz); DT_Tdep_PT_Printf (phead, "BPOOL num_segs %x\n", (int) bpool_ptr->num_segs); Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -32,11 +32,11 @@ */ boolean_t DT_query ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle) + struct dat_ia * ia, + struct dat_ep * ep) { unsigned char *module = "DT_query"; - DAT_EVD_HANDLE async_evd_hdl; /* not used */ + struct dat_evd * async_evd_hdl; /* not used */ struct dat_ep_param ep_params; u32 ret; DT_Tdep_Print_Head *phead; @@ -44,7 +44,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, phead = pt_ptr->Params.phead; /* Query the IA */ - ret = dat_ia_query (ia_handle, + ret = dat_ia_query (ia, &async_evd_hdl, &pt_ptr->ia_attr, &pt_ptr->provider_attr); @@ -57,7 +57,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, } /* Query the EP */ - ret = dat_ep_query (ep_handle, &ep_params); + ret = dat_ep_query (ep, &ep_params); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_query error: %s\n", @@ -160,7 +160,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, */ boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -184,7 +184,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Recv #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_recv (ep_handle, + ret = dat_ep_post_recv (ep, 1, iov, cookie, @@ -205,7 +205,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head */ boolean_t DT_post_send_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -228,7 +228,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Send #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_send (ep_handle, + ret = dat_ep_post_send (ep, 1, iov, cookie, @@ -249,7 +249,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head */ boolean_t DT_cr_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p) { int err_cnt; @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -306,8 +306,8 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p */ boolean_t DT_conn_event_wait (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep * ep, + struct dat_evd *evd, enum dat_event_number *event_number) { @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -341,7 +341,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * { /* * Could return struct dat_connection_event_data and verify: - * event.event_data.connect_event_data.ep_handle + * event.event_data.connect_event_data.ep * event.event_data.connect_event_data.private_data_size * event.event_data.connect_event_data.private_data */ @@ -361,15 +361,15 @@ DT_conn_event_wait (DT_Tdep_Print_Head * */ boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ) + struct dat_evd *evd, + struct dat_ep * *ep ) { for (;;) { u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -392,9 +392,9 @@ DT_disco_event_wait ( DT_Tdep_Print_Head if (event.event_number == DAT_CONNECTION_EVENT_DISCONNECTED) { - if ( ep_handle != NULL ) + if ( ep != NULL ) { - *ep_handle = event.event_data.connect_event_data.ep_handle; + *ep = event.event_data.connect_event_data.ep; } return (TRUE); } @@ -412,17 +412,17 @@ DT_disco_event_wait ( DT_Tdep_Print_Head */ boolean_t DT_dto_event_reap (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dto_statusp) { if (poll) { - return DT_dto_event_poll (phead, evd_handle, dto_statusp); + return DT_dto_event_poll (phead, evd, dto_statusp); } else { - return DT_dto_event_wait (phead, evd_handle, dto_statusp); + return DT_dto_event_wait (phead, evd, dto_statusp); } } @@ -432,7 +432,7 @@ DT_dto_event_reap (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_poll (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -440,7 +440,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); if (DAT_GET_TYPE (ret) == DAT_QUEUE_EMPTY) @@ -460,7 +460,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -484,7 +484,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -505,7 +505,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -528,7 +528,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p */ boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_statusp) { for (;;) @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", @@ -572,12 +572,12 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p boolean_t DT_dto_check ( DT_Tdep_Print_Head *phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message ) { - if ( ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) || dto_p->transfered_length != len_expected || dto_p->user_cookie.as_64 != cookie_expected.as_64 || dto_p->status != DAT_DTO_SUCCESS ) @@ -588,10 +588,10 @@ DT_dto_check ( DT_Tdep_Print_Head *phead : (dto_p->status == DAT_DTO_ERR_FLUSHED ? "FAILURE" : "LengthError"))); DT_Test_Error (); - if ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) { DT_Tdep_PT_Printf (phead, "\tEndPoint mismatch (got %p wanted %p)\n", - dto_p->ep_handle, + dto_p->ep, ep_expected); } if (dto_p->transfered_length != len_expected) @@ -620,11 +620,11 @@ DT_dto_check ( DT_Tdep_Print_Head *phead boolean_t DT_rmr_check ( DT_Tdep_Print_Head *phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message) { - if ( rmr_p->rmr_handle != rmr_expected + if ( rmr_p->rmr != rmr_expected || rmr_p->user_cookie.as_ptr != cookie_expected || rmr_p->status != DAT_RMR_BIND_SUCCESS ) { @@ -634,10 +634,10 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead (rmr_p->status == DAT_RMR_BIND_SUCCESS ? "OK" : "FAILURE")); DT_Test_Error (); - if (rmr_p->rmr_handle != rmr_expected) + if (rmr_p->rmr != rmr_expected) { DT_Tdep_PT_Printf (phead, "\tRMR handle mismatch (got 0x%p wanted 0x%p)\n", - rmr_p->rmr_handle, + rmr_p->rmr, rmr_expected); } if (rmr_p->user_cookie.as_ptr != cookie_expected) @@ -659,21 +659,21 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead boolean_t DT_cr_check ( DT_Tdep_Print_Head *phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp * psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr * *crp, char *message) { u32 ret; - if (cr_handlep) + if (crp) { - *cr_handlep = (DAT_CR_HANDLE) 0; + *crp = (struct dat_cr *) 0; } if (cr_stat_p->conn_qual != port_expected || - (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected)) + (psp_expected && + cr_stat_p->sp != psp_expected)) { DT_Tdep_PT_Printf (phead, "Test Error: %s CR data problem\n", message); @@ -684,20 +684,20 @@ DT_cr_check ( DT_Tdep_Print_Head *phead " (got 0x" F64x " wanted 0x" F64x ")\n", cr_stat_p->conn_qual, port_expected); } - if (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected) + if (psp_expected && + cr_stat_p->sp != psp_expected) { DT_Tdep_PT_Printf (phead, "\tPSP mismatch (got 0x%p wanted 0x%p)\n", - cr_stat_p->sp_handle.psp_handle, - psp_handle_expected); + cr_stat_p->sp, + psp_expected); } - if (!cr_stat_p->cr_handle) + if (!cr_stat_p->cr) { - DT_Tdep_PT_Printf (phead, "\tGot NULL cr_handle\n"); + DT_Tdep_PT_Printf (phead, "\tGot NULL cr\n"); } else { - ret = dat_cr_reject (cr_stat_p->cr_handle); + ret = dat_cr_reject (cr_stat_p->cr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "\tdat_cr_reject error: %s\n", @@ -707,9 +707,9 @@ DT_cr_check ( DT_Tdep_Print_Head *phead return ( FALSE ); } - if (cr_handlep) + if (crp) { - *cr_handlep = cr_stat_p->cr_handle; + *crp = cr_stat_p->cr; } return ( TRUE ); } Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -41,13 +41,13 @@ DT_cs_Client (Params_t * params_ptr, u32 total_threads) { Per_Test_Data_t *pt_ptr = NULL; - DAT_IA_HANDLE ia_handle = DAT_HANDLE_NULL; - DAT_PZ_HANDLE pz_handle = DAT_HANDLE_NULL; - DAT_EVD_HANDLE recv_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE reqt_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE conn_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE async_evd_hdl = DAT_HANDLE_NULL; - DAT_EP_HANDLE ep_handle = DAT_HANDLE_NULL; + struct dat_ia *ia = NULL; + struct dat_pz *pz = NULL; + struct dat_evd *recv_evd_hdl = NULL; + struct dat_evd *reqt_evd_hdl = NULL; + struct dat_evd *conn_evd_hdl = NULL; + struct dat_evd *async_evd_hdl = NULL; + struct dat_ep *ep = NULL; Server_Info_t *sinfo = NULL; Transaction_Cmd_t *Transaction_Cmd = NULL; Quit_Cmd_t *Quit_Cmd = NULL; @@ -98,7 +98,7 @@ DT_cs_Client (Params_t * params_ptr, ret = dat_ia_open (dapl_name, DFLT_QLEN, &async_evd_hdl, - &ia_handle); + &ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -106,28 +106,28 @@ DT_cs_Client (Params_t * params_ptr, module, dapl_name, DT_RetToString (ret)); - ia_handle = DAT_HANDLE_NULL; + ia = NULL; status = 1; goto client_exit; } DT_Tdep_PT_Debug (1,(phead, "%s: IA %s opened\n", module, dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ia_handle, &pz_handle); + ret = dat_pz_create (ia, &pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - pz_handle = DAT_HANDLE_NULL; + pz = NULL; status = 1; goto client_exit; } /* Create 3 events - recv, request, connect */ - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -138,11 +138,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - recv_evd_hdl = DAT_HANDLE_NULL; + recv_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -153,11 +153,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - reqt_evd_hdl = DAT_HANDLE_NULL; + reqt_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -168,26 +168,26 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - conn_evd_hdl = DAT_HANDLE_NULL; + conn_evd_hdl = NULL; status = 1; goto client_exit; } /* Create an EP */ - ret = dat_ep_create (ia_handle, /* IA */ - pz_handle, /* PZ */ + ret = dat_ep_create (ia, /* IA */ + pz, /* PZ */ recv_evd_hdl, /* recv */ reqt_evd_hdl, /* request */ conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ep_handle); + &ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ep_handle = DAT_HANDLE_NULL; + ep = NULL; status = 1; goto client_exit; } @@ -197,7 +197,7 @@ DT_cs_Client (Params_t * params_ptr, * Gather whatever info we want about defaults, * and check that we can handle the requested parameters. */ - if (!DT_query (pt_ptr, ia_handle, ep_handle) || + if (!DT_query (pt_ptr, ia, ep) || !DT_check_params (pt_ptr, module)) { status = 1; @@ -206,10 +206,10 @@ DT_cs_Client (Params_t * params_ptr, bpool = DT_BpoolAlloc (pt_ptr, phead, - ia_handle, - pz_handle, - ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ia, + pz, + ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ pt_ptr->provider_attr.optimal_buffer_alignment, @@ -238,7 +238,7 @@ DT_cs_Client (Params_t * params_ptr, DT_Tdep_PT_Debug (1,(phead, "%s: Posting 1 recv buffer\n", module)); retry_repost: if (!DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, DT_Bpool_GetBuffSize (bpool, 0))) @@ -253,7 +253,7 @@ retry_repost: DT_Tdep_PT_Debug (1,(phead, "%s: Connect Endpoint\n", module)); try_connect =1; retry: - ret = dat_ep_connect (ep_handle, + ret = dat_ep_connect (ep, server_netaddr, SERVER_PORT_NUMBER, DAT_TIMEOUT_MAX, @@ -271,7 +271,7 @@ retry: } DT_Tdep_PT_Debug (1,(phead, "%s: Await connection ...\n", module)); - if (!DT_conn_event_wait (phead, ep_handle, conn_evd_hdl, &event_num)) + if (!DT_conn_event_wait (phead, ep, conn_evd_hdl, &event_num)) { if ( event_num == DAT_CONNECTION_EVENT_PEER_REJECTED ) { @@ -287,7 +287,7 @@ retry: * See if any buffers were flushed as a result of * the REJECT; clean them up and repost if so */ - dat_ep_reset (ep_handle); + dat_ep_reset (ep); do { @@ -315,7 +315,7 @@ retry: if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ep_handle); + get_ep_connection_state (phead, ep); } /* Send Client_Info (using 2nd buffer in the pool) */ @@ -326,7 +326,7 @@ retry: sizeof (Client_Info_t)); DT_Client_Info_Endian ((Client_Info_t *) buffp); if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 1, DT_Bpool_GetBuffSize (bpool, 1))) @@ -343,7 +343,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 1), dto_cookie, "Client_Info_Send")) @@ -396,7 +396,7 @@ retry: /* Send the Command buffer */ if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 2, DT_Bpool_GetBuffSize (bpool, 2))) @@ -414,7 +414,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 2), dto_cookie, "Client_Cmd_Send")) @@ -430,7 +430,7 @@ retry: if (!DT_dto_event_wait (phead, recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 0), dto_cookie, "Server_Info_Recv")) @@ -476,7 +476,7 @@ retry: DT_Transaction_Cmd_PT_Print (phead, Transaction_Cmd); } status = DT_Transaction_Test_Client (pt_ptr, - ia_handle, + ia, server_netaddr); params_ptr->Client_Stats_T.bytes_rdma_read = pt_ptr->Client_Stats.bytes_rdma_read; @@ -508,7 +508,7 @@ retry: status = DT_Performance_Test_Client (params_ptr, pt_ptr, - ia_handle, + (struct dat_ia *)ia, server_netaddr); break; } @@ -521,13 +521,13 @@ client_exit: DT_Tdep_PT_Debug (1,(phead, "%s: Cleaning Up ...\n", module)); /* Disconnect the EP */ - if (ep_handle && try_connect) + if (ep && try_connect) { /* * graceful attempt might fail because we got here due to * some error above, so we may as well try harder. */ - ret = dat_ep_disconnect (ep_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ep_disconnect (ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -547,7 +547,7 @@ client_exit: DT_Bpool_Destroy (pt_ptr, phead, bpool); /* Free the EP */ - if (ep_handle) + if (ep) { struct dat_event event; /* @@ -560,7 +560,7 @@ client_exit: &event); } while (ret == DAT_SUCCESS); - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -614,9 +614,9 @@ client_exit: } /* Free the PZ */ - if (pz_handle) + if (pz) { - ret = dat_pz_free (pz_handle); + ret = dat_pz_free (pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -629,10 +629,10 @@ client_exit: } /* Close the IA */ - if (ia_handle) + if (ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -640,7 +640,7 @@ client_exit: module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, Index: linux-kernel/test/dapltest/test/dapl_fft_endpoint.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) @@ -36,59 +36,59 @@ int DT_endpoint_generic (Params_t *param boolean_t destroy_pz_early) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; u32 rc, wanted; int res; DT_Tdep_Print_Head *phead; res = 1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - evd_handle = NULL; - conn_evd_handle = NULL; - send_evd_handle = NULL; - recv_evd_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + evd = NULL; + conn_evd = NULL; + send_evd = NULL; + recv_evd = NULL; dev_name = cmd->device_name; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; phead = params_ptr->phead; - rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, &ia_handle); + rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); if (destroy_pz_early) { - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } } - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, - DAT_EVD_CONNECTION_FLAG, &conn_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, + DAT_EVD_CONNECTION_FLAG, &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, pz_handle, recv_evd_handle, send_evd_handle, - conn_evd_handle, NULL, &ep_handle); + rc = dat_ep_create (ia, pz, recv_evd, send_evd, + conn_evd, NULL, &ep); if (destroy_pz_early) { wanted = DAT_INVALID_HANDLE; @@ -100,39 +100,39 @@ int DT_endpoint_generic (Params_t *param DT_assert_dat (phead, DAT_GET_TYPE (rc) == wanted); cleanup: - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (!destroy_pz_early && pz_handle) + if (!destroy_pz_early && pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; @@ -163,10 +163,10 @@ int DT_endpoint_case1 (Params_t *params_ int DT_endpoint_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE send_evd, conn_evd, recv_evd, cr_evd; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_ep *ep; + struct dat_evd *send_evd, *conn_evd, *recv_evd, *cr_evd; + struct dat_pz *pz; struct dat_event event; Bpool *bpool; int res; @@ -177,39 +177,39 @@ int DT_endpoint_case2 (Params_t *params_ Description: try to destroy ep with descriptor still in working queue\n"); res = 1; bpool = NULL; - pz_handle = NULL; - ia_handle = NULL; - ep_handle = NULL; + pz = NULL; + ia = NULL; + ep = NULL; send_evd = NULL; conn_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name = cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, 4096, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, 4096, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); DT_assert (phead, DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, 4096) == TRUE); - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -229,14 +229,14 @@ cleanup: rc = DT_Bpool_Destroy (NULL, phead, bpool); DT_assert_clean (phead, rc != FALSE); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_util.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c (working copy) @@ -59,7 +59,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the recv */ - ret = dat_ep_post_recv ( ep_context[i].ep_handle, + ret = dat_ep_post_recv ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -85,7 +85,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H boolean_t DT_handle_send_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll) @@ -122,7 +122,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the send */ - ret = dat_ep_post_send ( ep_context[i].ep_handle, + ret = dat_ep_post_send ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -173,7 +173,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -188,7 +188,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Send")) @@ -201,7 +201,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -215,7 +215,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -235,8 +235,8 @@ DT_handle_send_op (DT_Tdep_Print_Head *p boolean_t DT_handle_recv_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -283,7 +283,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Receive: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -298,7 +298,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Recv")) @@ -315,7 +315,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -353,7 +353,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: "F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -374,11 +374,11 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p /* * If we have multiple EPs we can't guarantee the order of - * completions, so disable ep_handle check + * completions, so disable ep check */ if (!DT_dto_check (phead, &dto_stat, - num_eps == 1?ep_context[i].ep_handle: NULL, + num_eps == 1?ep_context[i].ep: NULL, op->num_segs * op->seg_size, dto_cookie, "Send-reaped-on-recv")) @@ -395,7 +395,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -411,7 +411,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -428,7 +428,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -463,7 +463,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -516,7 +516,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (opcode == RDMA_WRITE) { - ret = dat_ep_post_rdma_write (ep_context[i].ep_handle, + ret = dat_ep_post_rdma_write (ep_context[i].ep, op->num_segs, iov, cookie, @@ -527,7 +527,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p else /* opcode == RDMA_READ */ { - ret = dat_ep_post_rdma_read ( ep_context[i].ep_handle, + ret = dat_ep_post_rdma_read ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -574,7 +574,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: %s: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -588,7 +588,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, (opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD"))) @@ -601,7 +601,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: Secondary completion seen for endpoint 0x%p (%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -619,7 +619,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: No completion seen for endpoint 0x%p (#%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); Index: linux-kernel/test/dapltest/test/dapl_fft_pz.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) @@ -35,9 +35,9 @@ int DT_pz_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_evd *evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -47,25 +47,25 @@ int DT_pz_case0 ( Params_t *params_ptr, Description: Test if we can normally create pz and destroy it.\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + ia = NULL; + pz = NULL; + evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); cleanup: - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); } return res; @@ -75,10 +75,10 @@ cleanup: int DT_pz_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE conn_evd, send_evd, recv_evd, cr_evd; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *conn_evd, *send_evd, *recv_evd, *cr_evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -88,42 +88,42 @@ int DT_pz_case1 (Params_t *params_ptr, F Description: try to destroy pz with vi still associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; conn_evd = NULL; send_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } cleanup: /* corrrect order */ - if (ep_handle) + if (ep) { - rc=dat_ep_free (ep_handle); + rc=dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn_evd) @@ -141,15 +141,15 @@ cleanup: rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -160,8 +160,8 @@ cleanup: int DT_pz_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz *pz; Bpool *bpool; u32 rc; int res; @@ -174,26 +174,26 @@ int DT_pz_case2 (Params_t *params_ptr, F associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; bpool = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate and register bpool */ - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, BUFFSIZE, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } @@ -205,15 +205,15 @@ cleanup: DT_Tdep_PT_Printf (phead, "Warning: Destroy bpool fails, reboot for cleanup\n"); return 0; } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: linux-kernel/test/dapltest/test/dapl_fft_hwconn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (working copy) @@ -31,8 +31,8 @@ int DT_hwconn_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE nic_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * nic_handle; + struct dat_evd *evd; u32 rc; int res = 1; DT_Tdep_Print_Head *phead; @@ -43,9 +43,9 @@ int DT_hwconn_case0 ( Params_t *params_p dev_name= cmd->device_name; nic_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); @@ -58,9 +58,9 @@ cleanup: /*--------------------------------------------------------*/ int DT_hwconn_case1 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; char dev_name[100]; int i; DT_Tdep_Print_Head *phead; @@ -94,8 +94,8 @@ int DT_hwconn_case1 ( Params_t *params_p sprintf (dev_name, "%s", "34df"); /* number_letter */ } - evd_handle = DAT_HANDLE_NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + evd = NULL; + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); if (DAT_GET_TYPE (rc) != DAT_PROVIDER_NOT_FOUND) { const char *major_msg, *minor_msg; @@ -120,7 +120,7 @@ int DT_hwconn_case1 ( Params_t *params_p /*--------------------------------------------------------*/ int DT_hwconn_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; int res=1; DT_Tdep_Print_Head *phead; @@ -156,9 +156,9 @@ int DT_hwconn_case3 (Params_t *params_pt DT_fft_init_client (params_ptr, cmd, &conn); /* try to close nic when vi have not destroyed */ - if (conn.ia_handle) + if (conn.ia) { - rc= dat_ia_close (conn.ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc= dat_ia_close (conn.ia, DAT_CLOSE_ABRUPT_FLAG); if (rc !=DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Warning: dat_ia_close fails %s, reboot for cleanup\n", Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (revision 2564) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (working copy) @@ -34,7 +34,7 @@ int DT_dataxfer_generic ( DT_Tdep_Print_ u32 rc=0; int res=1; DT_fft_init_server (phead, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); DT_fft_listen (phead, &conn); @@ -81,7 +81,7 @@ cleanup: int DT_dataxfer_case0 ( DT_Tdep_Print_Head *phead, FFT_Cmd_t *cmd) { DT_Tdep_PT_Printf (phead, "\ - Description: Call dat_ep_post_send with null ep_handle.\n"); + Description: Call dat_ep_post_send with null ep.\n"); return DT_dataxfer_generic (phead, cmd, 0); } Index: linux-kernel/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_tdep.h (revision 2564) +++ linux-kernel/test/dapltest/include/dapl_tdep.h (working copy) @@ -31,7 +31,7 @@ #include "dapl_proto.h" #ifdef __KDAPL__ -typedef DAT_HANDLE DAT_CNO_HANDLE; +typedef void * DAT_CNO_HANDLE; #endif /* function prototypes */ @@ -45,21 +45,21 @@ int DT_Tdep_Execute_Test ( Params_t *params_ptr ) ; u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr); + struct dat_evd * *evd_ptr); u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle); +DT_Tdep_evd_free (struct dat_evd * evd); u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd * evd, unsigned long timeout, struct dat_event *event); u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd * evd, struct dat_event *event); #endif Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2564) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -54,7 +54,7 @@ typedef struct { typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL port; int pipeline_len; @@ -70,17 +70,17 @@ typedef struct boolean_t is_remote_little_endian; DAT_CONN_QUAL base_port; struct dat_ia_attr ia_attr; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; int reqt_evd_length; - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ int recv_evd_length; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ + struct dat_evd * recv_evd_hdl; /* receive */ int conn_evd_length; - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ + struct dat_evd * conn_evd_hdl; /* connect */ int creq_evd_length; - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_evd * creq_evd_hdl; /* "" request */ Performance_Ep_Context_t ep_context; } Performance_Test_t; Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2564) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -81,10 +81,10 @@ extern int g_status; /* dapl_bpool.c */ Bpool * DT_BpoolAlloc (Per_Test_Data_t * pt_ptr, DT_Tdep_Print_Head* phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_ep *ep, + struct dat_evd *rmr_evd, int seg_size, int num_segs, int alignment, @@ -106,7 +106,7 @@ void DT_Bpool_print (DT_Tdep_ /* dapl_cnxn.c */ int get_ep_connection_state (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle); + struct dat_ep *ep); /* dapl_client.c */ int DT_cs_Client (Params_t * params_ptr, @@ -236,7 +236,7 @@ void DT_Performance_Cmd_Endia /* dapl_performance_client.c */ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, struct sockaddr *remote); boolean_t DT_Performance_Test_Client_Connect ( @@ -261,7 +261,7 @@ boolean_t DT_Performance_Test /* dapl_performance_util.c */ boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -272,20 +272,20 @@ int DT_Performance_Test_Dest boolean_t is_server); boolean_t DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats); unsigned int DT_performance_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Mode_Type mode, Performance_Stats_t *stats); unsigned int DT_performance_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); unsigned int DT_performance_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); /* dapl_performance_stats.c */ @@ -346,69 +346,69 @@ void DT_Free_Per_Test_Data (Per_Test_D /* dapl_test_util.c */ boolean_t DT_query (Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle); + struct dat_ia * ia, + struct dat_ep * ep); boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_post_send_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_conn_event_wait (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep *ep, + struct dat_evd *evd, enum dat_event_number *event_number); boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ); + struct dat_evd *evd, + struct dat_ep * *ep ); boolean_t DT_cr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p); boolean_t DT_dto_event_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_ptr); boolean_t DT_dto_check ( DT_Tdep_Print_Head* phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message); boolean_t DT_rmr_check ( DT_Tdep_Print_Head* phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message); boolean_t DT_cr_check (DT_Tdep_Print_Head* phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp *psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr **crp, char *message); /* dapl_thread.c */ @@ -460,13 +460,13 @@ void DT_Transaction_Cmd_Endia boolean_t to_wire); /* dapl_transaction_test.c */ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote); void DT_Transaction_Test_Server (void *params); boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -491,15 +491,15 @@ boolean_t DT_handle_post_recv boolean_t DT_handle_send_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll); boolean_t DT_handle_recv_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -507,7 +507,7 @@ boolean_t DT_handle_recv_op ( boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -600,15 +600,15 @@ void DT_assert_fail (DT_Tdep_Print_H char *file, char *baseFile, int line); -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle); +int DT_ia_open (char *dev_name, struct dat_ia **ia); int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle); + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep); void DT_fft_init_conn_struct (FFT_Connection_t *conn); void DT_fft_init_client (Params_t *params_ptr, FFT_Cmd_t *cmd, Index: linux-kernel/test/dapltest/include/dapl_test_data.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_test_data.h (revision 2564) +++ linux-kernel/test/dapltest/include/dapl_test_data.h (working copy) @@ -46,16 +46,16 @@ typedef struct int NextPortNumber; int num_clients; spinlock_t num_clients_lock; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; - DAT_EVD_HANDLE reqt_evd_hdl; - DAT_EVD_HANDLE conn_evd_hdl; - DAT_EVD_HANDLE creq_evd_hdl; - DAT_EVD_HANDLE async_evd_hdl; - DAT_EVD_HANDLE rmr_evd_hdl; - DAT_EP_HANDLE ep_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_evd *recv_evd_hdl; + struct dat_evd *reqt_evd_hdl; + struct dat_evd *conn_evd_hdl; + struct dat_evd *creq_evd_hdl; + struct dat_evd *async_evd_hdl; + struct dat_evd *rmr_evd_hdl; + struct dat_ep *ep; + struct dat_sp *psp; Bpool *bpool; } Per_Server_Data_t; Index: linux-kernel/test/dapltest/include/dapl_bpool.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2564) +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) @@ -38,21 +38,21 @@ typedef struct Bpool_tag Bpool; struct Bpool_tag { unsigned char *alloc_ptr; uint32_t alloc_size; - DAT_PZ_HANDLE pz_handle; + struct dat_pz * pz; int seg_size; int num_segs; /* num segments */ unsigned char *buffer_start; /* Start of buffer area */ u64 buffer_size; /* Size of data buffer (rounded) */ u64 reg_addr; /* start of registered area */ u64 reg_size; /* size of registered area */ - DAT_EP_HANDLE ep_handle; /* EP area is registered to */ - DAT_LMR_HANDLE lmr_handle; /* local access */ + struct dat_ep * ep; /* EP area is registered to */ + struct dat_lmr * lmr; /* local access */ DAT_LMR_CONTEXT lmr_context; struct dat_lmr_triplet *tripl_start; /* local IOV */ boolean_t enable_rdma_write; /* remote access */ boolean_t enable_rdma_read; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; - DAT_EVD_HANDLE rmr_evd_handle; + struct dat_evd * rmr_evd; }; #endif Index: linux-kernel/test/dapltest/include/dapl_fft_util.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2564) +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) @@ -55,14 +55,14 @@ typedef struct { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_PSP_HANDLE psp_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE cr_evd, conn_evd, send_evd, recv_evd; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_sp *psp; + struct dat_ep *ep; + struct dat_evd *cr_evd, *conn_evd, *send_evd, *recv_evd; struct dat_event event; int count; - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; Bpool *bpool; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2564) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -49,13 +49,13 @@ typedef struct typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL ia_port; Bpool *bp; Transaction_Test_Op_t op[ MAX_OPS ]; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; } Ep_Context_t; @@ -75,7 +75,7 @@ typedef struct boolean_t is_server; boolean_t remote_is_little_endian; Per_Test_Data_t *pt_ptr; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; @@ -84,11 +84,11 @@ typedef struct Thread *thread; /* This group set up by each thread in DT_Transaction_Main() */ - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_pz * pz; + struct dat_evd * recv_evd_hdl; /* receive */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ + struct dat_evd * conn_evd_hdl; /* connect */ + struct dat_evd * creq_evd_hdl; /* "" request */ Ep_Context_t *ep_context; /* Statistics set by DT_Transaction_Run() */ Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2564) +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -39,7 +39,7 @@ typedef struct Tdep_Evd_Tag struct Tdep_Evd_Tag *evd_next; struct Tdep_Event_Tag *event_next; struct Tdep_Event_Tag *event_last; - DAT_EVD_HANDLE evd_handle; + struct dat_evd * evd; DT_WAIT_OBJECT wait_object; } Tdep_Evd; @@ -94,11 +94,11 @@ KDT_Evd_Destroy(void) } u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr) + struct dat_evd **evd_ptr_ptr) { u32 dat_status; @@ -120,12 +120,12 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha upcall.upcall_func = DT_Tdep_Event_Callback; upcall.instance_data = evd_ptr; - dat_status = dat_evd_kcreate (ia_handle, + dat_status = dat_evd_kcreate (ia, evd_min_qlen, DAT_UPCALL_SINGLE_INSTANCE, &upcall, evd_flags, - evd_handle_ptr); + evd_ptr_ptr); if (dat_status != DAT_SUCCESS) { goto error; @@ -134,7 +134,7 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha /* initialize evd_ptr */ evd_ptr->event_next = NULL; evd_ptr->event_last = NULL; - evd_ptr->evd_handle = *evd_handle_ptr; + evd_ptr->evd = *evd_ptr_ptr; DT_Mdep_wait_object_init (&evd_ptr->wait_object); /* add evd_ptr to front of evd list */ @@ -154,7 +154,7 @@ error: return (dat_status); } u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd *evd, struct dat_event *dat_event) { u32 dat_status; @@ -163,11 +163,11 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ unsigned long flags; dat_status = DAT_SUCCESS; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -181,7 +181,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } /* Get event */ spin_lock_irqsave (&DT_Evd_Lock, flags); - event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd_handle */ + event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd */ if (event) { evd_ptr->event_next = event->event_next; /* point to next event */ @@ -212,7 +212,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd *evd, unsigned long timeout, struct dat_event *dat_event) { @@ -223,11 +223,11 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han dat_status = DAT_CLASS_ERROR | DAT_INTERNAL_ERROR; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -290,7 +290,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han } u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle) +DT_Tdep_evd_free (struct dat_evd *evd) { Tdep_Evd *next; Tdep_Evd *last; @@ -298,13 +298,13 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han last = NULL; spin_lock_irq (&DT_Evd_Lock); next = DT_Evd_List; - if (next->evd_handle == evd_handle) + if (next->evd == evd) { DT_Evd_List = next->evd_next; } else { - while (next && next->evd_handle != evd_handle) + while (next && next->evd != evd) { last = next; next = next->evd_next; @@ -317,7 +317,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han spin_unlock_irq (&DT_Evd_Lock); DT_Mdep_Free (next); - return dat_evd_free(evd_handle); + return dat_evd_free(evd); } static void DT_Tdep_Event_Callback ( Index: linux-kernel/dat-provider/dapl_cr_reject.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_reject.c (revision 2564) +++ linux-kernel/dat-provider/dapl_cr_reject.c (working copy) @@ -40,36 +40,36 @@ * Reject a connection request from the active remote side requesting * an Endpoint. */ -u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) +u32 dapl_cr_reject(struct dat_cr *cr) { struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; struct dapl_sp *sp_ptr; u32 dat_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* * Clean up provider created EP if there is one: only if * DAT_PSP_PROVIDER_FLAG was set on the PSP */ - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - entry_ep_handle = cr_ptr->param.local_ep_handle; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; + entry_ep = cr_ptr->param.local_ep; entry_ep_state = 0; if (ep_ptr) { entry_ep_state = ep_ptr->param.ep_state; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - cr_ptr->param.local_ep_handle = NULL; + cr_ptr->param.local_ep = NULL; } dat_status = dapl_ib_reject_connection(cr_ptr->ib_cm_handle); @@ -77,8 +77,8 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand if (ep_ptr != NULL) { /* Revert our state to the beginning */ ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; - cr_ptr->param.local_ep_handle = (DAT_EP_HANDLE) ep_ptr; + cr_ptr->param.local_ep = entry_ep; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; } } else { /* @@ -88,13 +88,13 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand */ sp_ptr = cr_ptr->sp_ptr; if (ep_ptr && sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) - (void)dapl_ep_free(ep_ptr); + (void)dapl_ep_free((struct dat_ep *)ep_ptr); /* Remove the CR from the queue, then free it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); } Index: linux-kernel/dat-provider/dapl_srq.c =================================================================== --- linux-kernel/dat-provider/dapl_srq.c (revision 2564) +++ linux-kernel/dat-provider/dapl_srq.c (working copy) @@ -51,13 +51,9 @@ */ static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) { - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); - - /* reset magic to prevent reuse */ - srq_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); + dapl_ia_unlink_srq(srq_ptr->common.owner_ia, srq_ptr); dapl_cb_free(&srq_ptr->recv_buffer); - /* no need to destroy srq_ptr->header.lock */ + /* no need to destroy srq_ptr->common.lock */ kfree(srq_ptr); } @@ -91,19 +87,14 @@ static struct dapl_srq *dapl_srq_alloc(s /* zero the structure */ memset(srq_ptr, 0, sizeof *srq_ptr); - /* - * initialize the header - */ - srq_ptr->header.provider = ia_ptr->header.provider; - srq_ptr->header.magic = DAPL_MAGIC_SRQ; - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; - srq_ptr->header.owner_ia = ia_ptr; - srq_ptr->header.user_context.as_64 = 0; - srq_ptr->header.user_context.as_ptr = NULL; + srq_ptr->srq.provider = ia_ptr->ia.provider; + srq_ptr->common.owner_ia = ia_ptr; + srq_ptr->common.user_context.as_64 = 0; + srq_ptr->common.user_context.as_ptr = NULL; atomic_set(&srq_ptr->srq_ref_count, 0); - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); - spin_lock_init(&srq_ptr->header.lock); + dapl_llist_init_entry(&srq_ptr->common.ia_list_entry); + spin_lock_init(&srq_ptr->common.lock); /* * Initialize the body. @@ -130,15 +121,15 @@ bail: * dapl_srq_create * * Create an instance of a Shared Receive Queue that is provided to the - * consumer at srq_handle. + * consumer at srq. * * Input: - * ia_handle - * pz_handle + * ia + * pz * srq_attr * * Output: - * srq_handle + * srq * * Returns: * DAT_SUCCESS @@ -148,8 +139,8 @@ bail: * ?DAT_INVALID_ATTRIBUTE?? * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) +u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_srq_attr *srq_attr, struct dat_srq **srq) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; @@ -157,14 +148,14 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_create (%p, %p, %p, %p)\n", - ia_handle, pz_handle, srq_attr, srq_handle); + ia, pz, srq_attr, srq); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -175,12 +166,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - - if (srq_handle == NULL) { + if (srq == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -197,9 +183,9 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han goto bail; } - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; + srq_ptr->param.ia = (struct dat_ia *) ia_ptr; srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; - srq_ptr->param.pz_handle = pz_handle; + srq_ptr->param.pz = pz; /* * XXX Allocate provider resource here!!! @@ -211,7 +197,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han /* Link it onto the IA */ dapl_ia_link_srq(ia_ptr, srq_ptr); - *srq_handle = srq_ptr; + *srq = (struct dat_srq *)srq_ptr; bail: return status; @@ -223,7 +209,7 @@ bail: * Destroy an instance of an SRQ * * Input: - * srq_handle + * srq * * Output: * none @@ -233,22 +219,22 @@ bail: * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) +u32 dapl_srq_free(struct dat_srq *srq) { struct dapl_srq *srq_ptr; struct dapl_ia *ia_ptr; struct dat_srq_param *param; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq); - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; param = &srq_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { + if (!srq_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -263,17 +249,17 @@ u32 dapl_srq_free(DAT_SRQ_HANDLE srq_han goto bail; } - ia_ptr = srq_ptr->header.owner_ia; + ia_ptr = srq_ptr->common.owner_ia; /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_lock_irqsave(&srq_ptr->common.lock, srq_ptr->common.flags); /* Remove the SRQ from the IA */ dapl_ia_unlink_srq(ia_ptr, srq_ptr); - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_unlock_irqrestore(&srq_ptr->common.lock, srq_ptr->common.flags); /* * Finish tearing everything down. @@ -301,7 +287,7 @@ bail: * over a connection of any ep handle into local_iov * * Input: - * srq_handle + * srq * num_segments * local_iov * user_cookie @@ -318,7 +304,7 @@ bail: * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie) { @@ -328,14 +314,14 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_post_recv (%p, %d, %p, %P)\n", - srq_handle, num_segments, local_iov, user_cookie.as_64); + srq, num_segments, local_iov, user_cookie.as_64); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * Synchronization ok since this buffer is only used for receive @@ -369,16 +355,15 @@ bail: return status; } -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) +u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_srq_query (%p, %x, %p)\n", - srq_handle, srq_param); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", srq, + srq_param); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -387,7 +372,7 @@ u32 dapl_srq_query(DAT_SRQ_HANDLE srq_ha goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * XXX Need to calculate available_dto_count and outstanding_dto_count @@ -407,7 +392,7 @@ bail: * Modify the size fo the event queue of a Shared Recieve Queue * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -421,22 +406,22 @@ bail: * DAT_INVALID_STATE */ -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) +u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", - srq_handle, srq_max_recv_dto); + srq, srq_max_recv_dto); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; - ia_ptr = srq_ptr->header.owner_ia; + srq_ptr = (struct dapl_srq *)srq; + ia_ptr = srq_ptr->common.owner_ia; /* * Check for nonsense requests per the spec @@ -463,7 +448,7 @@ bail: * event if it is reached. * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -476,20 +461,20 @@ bail: * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) +u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", - srq_handle, low_watermark); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq, + low_watermark); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* XXX Put implementation here XXX */ Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2564) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -171,7 +171,7 @@ u32 dapl_ib_close_hca(struct dapl_hca *h * Alloc a CQ * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * cqlen minimum QLen * @@ -227,7 +227,7 @@ u32 dapl_ib_cq_resize(struct dapl_ia *ia * Set up CQ completion notifications * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * * Output: @@ -253,7 +253,7 @@ u32 dapl_set_cq_notify(struct dapl_ia *i * Register a virtual memory region * * Input: - * ia_handle IA handle + * ia IA handle * lmr pointer to dapl_lmr struct * virt_addr virtual address of beginning of mem region * length length of memory region @@ -300,7 +300,7 @@ u32 dapl_ib_mr_register_physical(struct iova = buf_list[0].addr; acl = dapl_ib_convert_mem_privileges(privileges); acl |= IB_ACCESS_MW_BIND; - mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz_handle)->pd, + mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz)->pd, buf_list, length, acl, &iova); kfree(buf_list); if (IS_ERR(mr)) { @@ -393,7 +393,7 @@ u32 dapl_ib_mw_alloc(struct dapl_rmr *rm int ib_status; struct ib_mw *mw; - mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz_handle)->pd); + mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz)->pd); if (IS_ERR(mw)) { ib_status = PTR_ERR(mw); dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -540,7 +540,7 @@ u32 dapl_ib_mw_unbind(struct dapl_rmr *r * Set up an asynchronous callbacks of various kinds * * Input: - * ia_handle IA handle + * ia IA handle * handler_type type of handler to set up * callback_handle handle param for completion callbacks * callback callback routine pointer Index: linux-kernel/dat-provider/dapl_ia_memtype_hint.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_memtype_hint.c (revision 2564) +++ linux-kernel/dat-provider/dapl_ia_memtype_hint.c (working copy) @@ -37,7 +37,7 @@ * Provide a hint about optimal memory alignment and size * * Input: - * ia_handle, + * ia, * mem_type, * length, * mem_optimization, @@ -53,7 +53,7 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, @@ -63,11 +63,11 @@ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE i u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", - ia_handle); + ia); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; Index: linux-kernel/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2564) +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) @@ -74,17 +74,17 @@ u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ attr = &ep_ptr->param.ep_attr; - dapl_os_assert(ep_ptr->param.pz_handle != NULL); - ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz_handle)->pd; + dapl_os_assert(ep_ptr->param.pz != NULL); + ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz)->pd; ib_hca_handle = ia_ptr->hca_ptr->ib_hca_handle; cq_recv = NULL; cq_send = NULL; cq_recv = dapl_get_dto_cq(ia_ptr, - (struct dapl_evd *)ep_ptr->param.recv_evd_handle); + (struct dapl_evd *)ep_ptr->param.recv_evd); cq_send = dapl_get_dto_cq(ia_ptr, (struct dapl_evd *)ep_ptr->param. - request_evd_handle); + request_evd); qp_attr.send_cq = cq_send; qp_attr.recv_cq = cq_recv; @@ -160,7 +160,7 @@ struct ib_cq *dapl_get_dto_cq(struct dap int cq_entries; int status; - if (evd_ptr != DAT_HANDLE_NULL) + if (evd_ptr != NULL) cq = evd_ptr->cq; else if (ia_ptr->hca_ptr->null_cq != NULL) cq = ia_ptr->hca_ptr->null_cq; Index: linux-kernel/dat-provider/dapl_openib_util.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.h (revision 2564) +++ linux-kernel/dat-provider/dapl_openib_util.h (working copy) @@ -59,7 +59,7 @@ u32 dapl_ib_qp_free(struct dapl_ep *ep_p u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, struct dat_ep_attr *ep_attr); -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data); @@ -70,12 +70,12 @@ u32 dapl_ib_setup_conn_listener(struct d u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr); -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, + struct dat_ep *ep, int private_data_size, const void *private_data); -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle); +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm); u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, DAPL_ASYNC_HANDLER_TYPE handler_type, @@ -126,14 +126,14 @@ u32 dapl_ib_completion_notify(struct dap enum dat_dto_completion_status dapl_ib_get_dto_status(struct ib_wc *cqe_ptr); -void dapl_ib_reinit_ep(struct dapl_ep * ep_ptr); +void dapl_ib_reinit_ep(struct dapl_ep *ep_ptr); -void dapl_ib_disconnect_clean(struct dapl_ep * ep_ptr, boolean_t passive); +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t passive); u32 dapl_ib_get_async_event(struct ib_event *cause_ptr, enum dat_event_number *async_event); -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address); static inline u32 dapl_ib_status_convert(int32_t ib_status) Index: linux-kernel/dat-provider/dapl_ia_close.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_close.c (revision 2564) +++ linux-kernel/dat-provider/dapl_ia_close.c (working copy) @@ -40,7 +40,7 @@ * Close a provider, clean up resources, etc. * * Input: - * ia_handle + * ia * * Output: * none @@ -50,30 +50,29 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags ia_flags) +u32 dapl_ia_close(struct dat_ia *ia, enum dat_close_flags ia_flags) { struct dapl_ia *ia_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); + "dapl_ia_close (%p, %d)\n", ia, ia_flags); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) { + if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) dat_status = dapl_ia_abrupt_close(ia_ptr); - } else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) { + else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) dat_status = dapl_ia_graceful_close(ia_ptr); - } else { + else dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_cr_callback.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_callback.c (revision 2564) +++ linux-kernel/dat-provider/dapl_cr_callback.c (working copy) @@ -50,9 +50,9 @@ static u32 dapl_connection_request(struc struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; - cr_ptr = dapl_cr_alloc(sp_ptr->header.owner_ia); + cr_ptr = dapl_cr_alloc(sp_ptr->common.owner_ia); if (cr_ptr == NULL) { /* Invoking function will call dapl_ib_cm_reject() */ return DAT_INSUFFICIENT_RESOURCES; @@ -79,7 +79,7 @@ static u32 dapl_connection_request(struc } /* EP will be NULL unless RSP service point */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { /* @@ -89,7 +89,7 @@ static u32 dapl_connection_request(struc * EP we are out of resources and need to tell the * requestor that we cant help them. */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; ep_ptr = dapl_ep_alloc(ia_ptr, NULL); if (ep_ptr == NULL) { dapl_cr_free(cr_ptr); @@ -100,7 +100,7 @@ static u32 dapl_connection_request(struc dapl_ia_link_ep(ia_ptr, ep_ptr); } - cr_ptr->param.local_ep_handle = ep_ptr; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; if (ep_ptr != NULL) { /* Assign valid EP fields: RSP and PSP_PROVIDER_FLAG only */ @@ -109,12 +109,11 @@ static u32 dapl_connection_request(struc DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING; } else { /* RSP */ - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == DAT_SP_TYPE_RSP); ep_ptr->param.ep_state = DAT_EP_STATE_PASSIVE_CONNECTION_PENDING; } - ep_ptr->cm_handle = ib_cm_handle; + ep_ptr->cm = ib_cm_handle; } /* link the CR onto the SP so we can pick it up later */ @@ -122,16 +121,16 @@ static u32 dapl_connection_request(struc /* Post the event. */ /* assign sp_ptr to union to avoid typecast errors from some compilers */ - sp_handle.psp_handle = (DAT_PSP_HANDLE) sp_ptr; - dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp_handle, - (struct sockaddr *) &sp_ptr->header.owner_ia->hca_ptr->hca_address, - sp_ptr->conn_qual, (DAT_CR_HANDLE) cr_ptr); + sp = (struct dat_sp *)sp_ptr; + dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp, + (struct sockaddr *)&sp_ptr->common.owner_ia->hca_ptr->hca_address, + sp_ptr->conn_qual, (struct dat_cr *)cr_ptr); if (dat_status != DAT_SUCCESS) { /* Take the CR off the list, we can't use it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); (void)dapl_ib_reject_connection(ib_cm_handle); @@ -157,13 +156,7 @@ struct dapl_ep *dapl_get_sp_ep(struct da /* * acquire the lock, we may be racing with other threads here */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); - /* Verify under lock that the SP is still valid */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - return NULL; - } + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * There are potentially multiple connections in progress. Need to * go through the list and find the one we are interested @@ -172,16 +165,12 @@ struct dapl_ep *dapl_get_sp_ep(struct da */ cr_ptr = dapl_sp_search_cr(sp_ptr, ib_cm_handle); if (cr_ptr == NULL) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); return NULL; } - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - - /* Quick check to ensure our EP is still valid */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) - ep_ptr = NULL; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; /* The CR record is discarded in all except for the CONNECTED case, * as it will have no further relevance. @@ -211,31 +200,31 @@ struct dapl_ep *dapl_get_sp_ep(struct da "dapl_get_sp_ep disconnect dump sp: %p \n", sp_ptr); /* Decrement the ref count on the EVD */ - if (sp_ptr->evd_handle) { + if (sp_ptr->evd) { atomic_dec(&((struct dapl_evd *)sp_ptr-> - evd_handle)->evd_ref_count); - sp_ptr->evd_handle = NULL; + evd)->evd_ref_count); + sp_ptr->evd = NULL; } sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - (void)dapl_ib_remove_conn_listener(sp_ptr->header. + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + (void)dapl_ib_remove_conn_listener(sp_ptr->common. owner_ia, sp_ptr); dapl_ia_unlink_sp((struct dapl_ia *) - sp_ptr->header.owner_ia, + sp_ptr->common.owner_ia, sp_ptr); dapl_sp_dealloc(sp_ptr); dapl_cr_free(cr_ptr); goto skip_unlock; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); } else - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); skip_unlock: return ep_ptr; @@ -255,16 +244,6 @@ void dapl_cr_callback(struct dapl_cm_id ib_cm_handle, event, sp_ptr); /* - * The SP pointer could have been cleaned up in a racing - * CM callback, check to see if we should just exit here - */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - return; - } - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); - - /* * CONNECT_REQUEST events create an event on the PSP * EVD, which will trigger connection processing. The * sequence is: @@ -277,7 +256,7 @@ void dapl_cr_callback(struct dapl_cm_id */ if (event == DAT_CONNECTION_REQUEST_EVENT) { ep_ptr = NULL; - evd_ptr = sp_ptr->evd_handle; + evd_ptr = (struct dapl_evd *)sp_ptr->evd; } else { /* see if there is an EP connected with this CM handle */ ep_ptr = dapl_get_sp_ep(ib_cm_handle, sp_ptr, event); @@ -286,7 +265,7 @@ void dapl_cr_callback(struct dapl_cm_id if (ep_ptr == NULL) return; - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; /* if something has happened to our EVD, bail. */ if (evd_ptr == NULL) return; @@ -296,30 +275,29 @@ void dapl_cr_callback(struct dapl_cm_id switch (event) { case DAT_CONNECTION_REQUEST_EVENT: - { /* * Requests arriving on a disabled SP are immediately rejected */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); if (sp_ptr->listening == FALSE) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, "---> dapl_cr_callback: conn event on down SP\n"); (void)dapl_ib_reject_connection(ib_cm_handle); return; } - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_RSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_RSP) { /* * RSP connections only allow a single connection. Close * it down NOW so we reject any further connections. */ sp_ptr->listening = FALSE; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* * Only occurs on the passive side of a connection @@ -332,24 +310,20 @@ void dapl_cr_callback(struct dapl_cm_id evd_ptr = NULL; break; - } case DAT_CONNECTION_EVENT_ESTABLISHED: - { /* This is just a notification the connection is now * established, there isn't any private data to deal with. * * Update the EP state and cache a copy of the cm handle, * then let the user know we are ready to go. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->header.magic != DAPL_MAGIC_EP || - ep_ptr->param.ep_state != - DAT_EP_STATE_COMPLETION_PENDING) { + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING) { /* If someone pulled the plug on the EP or connection, * just exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; /* Set evd_ptr = NULL so we don't generate an event below */ evd_ptr = NULL; @@ -358,72 +332,63 @@ void dapl_cr_callback(struct dapl_cm_id } ep_ptr->param.ep_state = DAT_EP_STATE_CONNECTED; - ep_ptr->cm_handle = ib_cm_handle; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + ep_ptr->cm = ib_cm_handle; + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_DISCONNECTED: - { /* * EP is now fully disconnected; initiate any post processing * to reset the underlying QP and get the EP ready for * another connection */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { /* The disconnect has already occurred, we are now * cleaned up and ready to exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); return; } ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: - { /* * After posting an accept the requesting node has * stopped talking. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_BROKEN: - { - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } default: - { evd_ptr = NULL; dapl_os_assert(0); /* shouldn't happen */ break; } - } if (evd_ptr != NULL) dat_status = dapl_evd_post_connection_event(evd_ptr, event, - (DAT_HANDLE) ep_ptr, + (struct dat_ep *)ep_ptr, 0, NULL); if (dat_status != DAT_SUCCESS) { Index: linux-kernel/dat-provider/dapl_cr_accept.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_accept.c (revision 2564) +++ linux-kernel/dat-provider/dapl_cr_accept.c (working copy) @@ -39,42 +39,39 @@ * Establish a connection between active side (remote endpoint) * and passive side (local endpoint). */ -u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - int private_data_size, const void *private_data) +u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, + const void *private_data) { struct dapl_ep *ep_ptr; u32 dat_status; struct dapl_cr *cr_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_cr_accept (%p, %p, %d, %p)\n", - cr_handle, ep_handle, private_data_size, private_data); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_accept (%p, %p, %d, %p)\n", + cr, ep, private_data_size, private_data); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* - * Return an error if we have an ep_handle and the CR already has an + * Return an error if we have an ep and the CR already has an * EP, indicating this is an RSP connection or PSP_PROVIDER_FLAG was * specified. */ - if (ep_handle != NULL && - (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP) || - cr_ptr->param.local_ep_handle != NULL)) { + if (ep != NULL && cr_ptr->param.local_ep != NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* Make sure we have an EP handle in one place or another */ - if (ep_handle == NULL && cr_ptr->param.local_ep_handle == NULL) { + if (ep == NULL && cr_ptr->param.local_ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -86,84 +83,71 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand } /* - * ep_handle is NULL if the user specified DAT_PSP_PROVIDER_FLAG + * ep is NULL if the user specified DAT_PSP_PROVIDER_FLAG * OR this is an RSP connection; retrieve it from the cr. */ - if (ep_handle == NULL) { - ep_handle = cr_ptr->param.local_ep_handle; - if ((((struct dapl_ep *)ep_handle)->param.ep_state != + if (ep == NULL) { + ep = cr_ptr->param.local_ep; + if ((((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) - && (((struct dapl_ep *)ep_handle)->param.ep_state != + && (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING)) { return DAT_INVALID_STATE; } } else { /* ensure this EP isn't connected or in use */ - if (((struct dapl_ep *)ep_handle)->param.ep_state != + if (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_UNCONNECTED) { return DAT_INVALID_STATE; } } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required * We do all verification and state change under lock, at which * point the EP state should protect us from most races. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* The qp must be attached by this point! */ - if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) - { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } entry_ep_state = ep_ptr->param.ep_state; - entry_ep_handle = cr_ptr->param.local_ep_handle; + entry_ep = cr_ptr->param.local_ep; ep_ptr->param.ep_state = DAT_EP_STATE_COMPLETION_PENDING; - ep_ptr->cm_handle = cr_ptr->ib_cm_handle; + ep_ptr->cm = cr_ptr->ib_cm_handle; ep_ptr->cr_ptr = cr_ptr; ep_ptr->param.remote_ia_address_ptr = cr_ptr->param.remote_ia_address_ptr; - cr_ptr->param.local_ep_handle = ep_handle; + cr_ptr->param.local_ep = ep; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); - dat_status = dapl_ib_accept_connection(cr_handle, - ep_handle, + dat_status = dapl_ib_accept_connection(cr, + ep, private_data_size, private_data); /* @@ -176,16 +160,14 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand * system error, app termination, perhaps other reasons. */ dat_status = - dapl_evd_post_connection_event(ep_ptr->param. - request_evd_handle, + dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. + request_evd, DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR, - (DAT_HANDLE) ep_ptr, + (void *) ep_ptr, 0, NULL); - - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } else { ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; + cr_ptr->param.local_ep = entry_ep; ep_ptr->cr_ptr = NULL; ep_ptr->param.remote_ia_address_ptr = NULL; } @@ -199,14 +181,8 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); } - } else { - /* Make this CR invalid. We need to hang on to it until - * the connection terminates, but it's destroyed from - * the app point of view. - */ - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_cr_handoff.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_handoff.c (revision 2564) +++ linux-kernel/dat-provider/dapl_cr_handoff.c (working copy) @@ -37,7 +37,7 @@ * Hand the connection request to another Sevice pont specified by the * Connectin Qualifier. */ -u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) +u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL cr_handoff) { /* handoff */ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); } Index: linux-kernel/dat-provider/dapl_pz.c =================================================================== --- linux-kernel/dat-provider/dapl_pz.c (revision 2564) +++ linux-kernel/dat-provider/dapl_pz.c (working copy) @@ -44,18 +44,13 @@ static struct dapl_pz *dapl_pz_alloc(str /* zero the structure */ memset(pz, 0, sizeof *pz); - /* - * initialize the header - */ - pz->header.provider = ia->header.provider; - pz->header.magic = DAPL_MAGIC_PZ; - pz->header.handle_type = DAT_HANDLE_TYPE_PZ; - pz->header.owner_ia = ia; - pz->header.user_context.as_64 = 0; - pz->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&pz->header.ia_list_entry); + pz->pz.provider = ia->ia.provider; + pz->common.owner_ia = ia; + pz->common.user_context.as_64 = 0; + pz->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&pz->common.ia_list_entry); dapl_ia_link_pz(ia, pz); - spin_lock_init(&pz->header.lock); + spin_lock_init(&pz->common.lock); /* * initialize the body @@ -67,83 +62,81 @@ static struct dapl_pz *dapl_pz_alloc(str static void dapl_pz_dealloc(struct dapl_pz *pz) { - /* reset magic to prevent reuse */ - pz->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_pz(pz->header.owner_ia, pz); + dapl_ia_unlink_pz(pz->common.owner_ia, pz); kfree(pz); } -u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) +u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; int ib_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", - ia_handle, pz_handle); + ia, pz); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error1; } - if (NULL == pz_handle) { + if (NULL == pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto error1; } - ia = (struct dapl_ia *)ia_handle; + dapl_ia = (struct dapl_ia *)ia; - pz = dapl_pz_alloc(ia); - if (NULL == pz) { + dapl_pz = dapl_pz_alloc(dapl_ia); + if (!dapl_pz) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - pz->pd = ib_alloc_pd(ia->hca_ptr->ib_hca_handle); - if (IS_ERR(pz->pd)) { - ib_status = PTR_ERR(pz->pd); + dapl_pz->pd = ib_alloc_pd(dapl_ia->hca_ptr->ib_hca_handle); + if (IS_ERR(dapl_pz->pd)) { + ib_status = PTR_ERR(dapl_pz->pd); dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_alloc_pd failed: %X\n", ib_status); status = dapl_ib_status_convert(ib_status); goto error2; } - *pz_handle = pz; + *pz = (struct dat_pz *)dapl_pz; return DAT_SUCCESS; error2: - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); error1: - *pz_handle = NULL; + *pz = NULL; return status; } -u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) +u32 dapl_pz_free(struct dat_pz *pz) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; int ib_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - if (0 != atomic_read(&pz->pz_ref_count)) { + if (0 != atomic_read(&dapl_pz->pz_ref_count)) { status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_PZ_IN_USE); goto error; } - ib_status = ib_dealloc_pd(pz->pd); + ib_status = ib_dealloc_pd(dapl_pz->pd); if (ib_status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_dealloc_pd failed: %X\n", ib_status); @@ -151,22 +144,22 @@ u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle goto error; } - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); return DAT_SUCCESS; error: return status; } -u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) +u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_param) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query(%p, %x, %p)\n", - pz_handle, pz_param); + pz, pz_param); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } @@ -175,8 +168,8 @@ u32 dapl_pz_query(DAT_PZ_HANDLE pz_handl goto error; } - pz = (struct dapl_pz *)pz_handle; - pz_param->ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; + dapl_pz = (struct dapl_pz *)pz; + pz_param->ia = (struct dat_ia *)dapl_pz->common.owner_ia; return DAT_SUCCESS; error: Index: linux-kernel/dat-provider/dapl_lmr.c =================================================================== --- linux-kernel/dat-provider/dapl_lmr.c (revision 2564) +++ linux-kernel/dat-provider/dapl_lmr.c (working copy) @@ -36,7 +36,7 @@ static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_desc, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags mem_priv) { struct dapl_lmr *lmr; @@ -49,27 +49,22 @@ static struct dapl_lmr *dapl_lmr_alloc(s /* zero the structure */ memset(lmr, 0, sizeof *lmr); - /* - * initialize the header - */ - lmr->header.provider = ia->header.provider; - lmr->header.magic = DAPL_MAGIC_LMR; - lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; - lmr->header.owner_ia = ia; - lmr->header.user_context.as_64 = 0; - lmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&lmr->header.ia_list_entry); + lmr->lmr.provider = ia->ia.provider; + lmr->common.owner_ia = ia; + lmr->common.user_context.as_64 = 0; + lmr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&lmr->common.ia_list_entry); dapl_ia_link_lmr(ia, lmr); - spin_lock_init(&lmr->header.lock); + spin_lock_init(&lmr->common.lock); /* * initialize the body */ - lmr->param.ia_handle = (DAT_IA_HANDLE) ia; + lmr->param.ia = (struct dat_ia *)ia; lmr->param.mem_type = mem_type; lmr->param.region_desc = region_desc; lmr->param.length = length; - lmr->param.pz_handle = pz_handle; + lmr->param.pz = pz; lmr->param.mem_priv = mem_priv; atomic_set(&lmr->lmr_ref_count, 0); @@ -78,9 +73,7 @@ static struct dapl_lmr *dapl_lmr_alloc(s static void dapl_lmr_dealloc(struct dapl_lmr *lmr) { - /* reset magic to prevent reuse */ - lmr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); + dapl_ia_unlink_lmr(lmr->common.owner_ia, lmr); kfree(lmr); } @@ -88,49 +81,50 @@ static void dapl_lmr_dealloc(struct dapl static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, u64 length, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; reg_desc.for_va = virt_addr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, + (struct dat_pz *)pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register(ia, lmr, virt_addr, length, privileges); + status = dapl_ib_mr_register(ia, new_lmr, virt_addr, length, + privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64)(unsigned long) virt_addr; if (registered_length) @@ -139,9 +133,9 @@ static inline u32 dapl_lmr_create_virtua return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; @@ -151,26 +145,26 @@ static inline u32 dapl_lmr_create_physic DAT_REGION_DESCRIPTION phys_addr, u64 page_count, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; u64 *array = phys_addr.for_array; u32 status; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, page_count, - (DAT_PZ_HANDLE) pz, privileges); + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, + page_count, (struct dat_pz *) pz, privileges); - if (NULL == lmr) { + if (NULL == new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, + status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr.for_array, page_count, privileges); if (DAT_SUCCESS != status) @@ -178,23 +172,23 @@ static inline u32 dapl_lmr_create_physic /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = array[0]; if (registered_length) @@ -203,9 +197,9 @@ static inline u32 dapl_lmr_create_physic return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } @@ -214,13 +208,13 @@ static inline u32 dapl_lmr_create_lmr(st struct dapl_lmr *original_lmr, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; @@ -230,39 +224,39 @@ static inline u32 dapl_lmr_create_lmr(st if (status != DAT_SUCCESS) goto error1; - reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; + reg_desc.for_lmr = (struct dat_lmr *) original_lmr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, + (struct dat_pz *) pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_shared(ia, lmr, privileges); + status = dapl_ib_mr_register_shared(ia, new_lmr, privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64) (unsigned long) original_lmr->param.region_desc.for_va; @@ -272,73 +266,72 @@ static inline u32 dapl_lmr_create_lmr(st return DAT_SUCCESS; error3: - dapl_ib_mr_deregister(lmr); + dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } -u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags privileges, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - u64 *registered_length, u64 *registered_address) + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, + u64 *registered_address) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", - ia_handle, mem_type); + ia, mem_type); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - ia = (struct dapl_ia *)ia_handle; - pz = (struct dapl_pz *)pz_handle; + dapl_ia = (struct dapl_ia *)ia; + dapl_pz = (struct dapl_pz *)pz; switch (mem_type) { case DAT_MEM_TYPE_VIRTUAL: - status = dapl_lmr_create_virtual(ia, region_description.for_va, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_virtual(dapl_ia, + region_description.for_va, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_PHYSICAL: - status = dapl_lmr_create_physical(ia, region_description, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_physical(dapl_ia, region_description, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_LMR: { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; - if (DAPL_BAD_HANDLE(region_description.for_lmr_handle, - DAPL_MAGIC_LMR)) { + if (!region_description.for_lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto bail; } - lmr = (struct dapl_lmr *)region_description.for_lmr_handle; + dapl_lmr = (struct dapl_lmr *)region_description.for_lmr; - status = dapl_lmr_create_lmr(ia, lmr, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_lmr(dapl_ia, dapl_lmr, dapl_pz, + privileges, lmr, lmr_context, rmr_context, registered_length, registered_address); break; @@ -357,45 +350,45 @@ bail: return status; } -u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) +u32 dapl_lmr_free(struct dat_lmr *lmr) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } - lmr = (struct dapl_lmr *)lmr_handle; + dapl_lmr = (struct dapl_lmr *)lmr; - switch (lmr->param.mem_type) { + switch (dapl_lmr->param.mem_type) { case DAT_MEM_TYPE_PHYSICAL: case DAT_MEM_TYPE_VIRTUAL: case DAT_MEM_TYPE_LMR: { struct dapl_pz *pz; - if (0 != atomic_read(&lmr->lmr_ref_count)) + if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) return DAT_INVALID_STATE; status = dapl_hash_remove( - lmr->header.owner_ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + dapl_lmr->common.owner_ia->hca_ptr->lmr_hash_table, + dapl_lmr->param.lmr_context, NULL); if (status != DAT_SUCCESS) goto error; - status = dapl_ib_mr_deregister(lmr); + status = dapl_ib_mr_deregister(dapl_lmr); if (status == DAT_SUCCESS) { - pz = (struct dapl_pz *)lmr->param.pz_handle; + pz = (struct dapl_pz *)dapl_lmr->param.pz; atomic_dec(&pz->pz_ref_count); - dapl_lmr_dealloc(lmr); - } else /* failure; put lmr back in hash table */ - dapl_hash_insert(lmr->header.owner_ia-> + dapl_lmr_dealloc(dapl_lmr); + } else /* failure; put dapl_lmr back in hash table */ + dapl_hash_insert(dapl_lmr->common.owner_ia-> hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + dapl_lmr->param.lmr_context, dapl_lmr); break; } @@ -412,15 +405,15 @@ error: return status; } -u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) +u32 dapl_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *lmr_param) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", - lmr_handle, lmr_param); + lmr, lmr_param); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } @@ -429,8 +422,8 @@ u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_ha goto error; } - lmr = (struct dapl_lmr *)lmr_handle; - memcpy(lmr_param, &lmr->param, sizeof *lmr_param); + dapl_lmr = (struct dapl_lmr *)lmr; + memcpy(lmr_param, &dapl_lmr->param, sizeof *lmr_param); status = DAT_SUCCESS; error: return status; @@ -441,7 +434,7 @@ error: * will sync the cache with memory. */ -u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -449,11 +442,11 @@ u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDL u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_read(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } @@ -462,7 +455,7 @@ error: return status; } -u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -470,11 +463,11 @@ u32 dapl_lmr_sync_rdma_write(DAT_IA_HAND u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_write(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } Index: linux-kernel/dat-provider/dapl_get_handle_type.c =================================================================== --- linux-kernel/dat-provider/dapl_get_handle_type.c (revision 2564) +++ linux-kernel/dat-provider/dapl_get_handle_type.c (working copy) @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_get_handle_type - * - * Gets the handle type for the given dat_handle - * - * Input: - * dat_handle - * - * Output: - * handle_type - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ - -u32 dapl_get_handle_type(DAT_HANDLE dat_handle, enum dat_handle_type *type) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - *type = header->handle_type; - -bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_get_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_get_consumer_context.c (revision 2564) +++ linux-kernel/dat-provider/dapl_get_consumer_context.c (working copy) @@ -1,80 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_get_consumer_context - * - * DAPL Requirements Version xxx, 6.2.2.2 - * - * Gets the consumer context from the specified dat_object - * - * Input: - * dat_handle - * - * Output: - * context - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ -u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, union dat_context *context) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - if (context == NULL || ((unsigned long)(header) & 3)) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - *context = header->user_context; - - bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_rmr.c =================================================================== --- linux-kernel/dat-provider/dapl_rmr.c (revision 2564) +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) @@ -61,24 +61,19 @@ static struct dapl_rmr *dapl_rmr_alloc(s /* zero the structure */ memset(rmr, 0, sizeof *rmr); - /* - * initialize the header - */ - rmr->header.provider = pz->header.provider; - rmr->header.magic = DAPL_MAGIC_RMR; - rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; - rmr->header.owner_ia = pz->header.owner_ia; - rmr->header.user_context.as_64 = 0; - rmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&rmr->header.ia_list_entry); - dapl_ia_link_rmr(rmr->header.owner_ia, rmr); - spin_lock_init(&rmr->header.lock); + rmr->rmr.provider = pz->pz.provider; + rmr->common.owner_ia = pz->common.owner_ia; + rmr->common.user_context.as_64 = 0; + rmr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&rmr->common.ia_list_entry); + dapl_ia_link_rmr(rmr->common.owner_ia, rmr); + spin_lock_init(&rmr->common.lock); /* * initialize the body */ - rmr->param.ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; - rmr->param.pz_handle = (DAT_PZ_HANDLE)pz; + rmr->param.ia = (struct dat_ia *)pz->common.owner_ia; + rmr->param.pz = (struct dat_pz *)pz; rmr->param.lmr_triplet.lmr_context = 0; rmr->param.lmr_triplet.pad = 0; rmr->param.lmr_triplet.virtual_address = 0; @@ -93,11 +88,8 @@ static struct dapl_rmr *dapl_rmr_alloc(s static void dapl_rmr_dealloc(struct dapl_rmr *rmr) { - /* reset magic to prevent reuse */ - rmr->header.magic = DAPL_MAGIC_INVALID; - - dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); - /* no need to destroy rmr->header.lock */ + dapl_ia_unlink_rmr(rmr->common.owner_ia, rmr); + /* no need to destroy rmr->common.lock */ kfree(rmr); } @@ -115,7 +107,7 @@ static inline u32 dapl_rmr_bind_fuse(str u32 status; boolean_t is_signaled; - status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, + status = dapl_hash_search(rmr->common.owner_ia->hca_ptr->lmr_hash_table, lmr_triplet->lmr_context, (DAPL_HASH_DATA *) &lmr); if (DAT_SUCCESS != status) { @@ -150,8 +142,8 @@ static inline u32 dapl_rmr_bind_fuse(str } /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ - if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || - (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { + if ((ep_ptr->param.pz != lmr->param.pz) || + (ep_ptr->param.pz != rmr->param.pz)) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -240,7 +232,7 @@ static inline u32 dapl_rmr_bind_unfuse(s } /* If the RMR and EP are not in the same PZ, there is an error */ - if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { + if (ep_ptr->param.pz != rmr->param.pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail1; } @@ -306,9 +298,9 @@ bail1: * Input: * Output: */ -u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, const struct dat_lmr_triplet *lmr_triplet, - enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, + enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) @@ -316,14 +308,14 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han struct dapl_rmr *rmr; struct dapl_ep *ep_ptr; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) + if (!rmr_handle) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) + if (!ep) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); rmr = (struct dapl_rmr *)rmr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* if the rmr should be bound */ if (0 != lmr_triplet->segment_length) @@ -341,7 +333,7 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * Create a remote memory region for the specified protection zone * * Input: - * pz_handle + * pz * * Output: * rmr_handle @@ -351,20 +343,20 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) +u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr) { - struct dapl_pz *pz; - struct dapl_rmr *rmr; + struct dapl_pz *dapl_pz; + struct dapl_rmr *dapl_rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - rmr = dapl_rmr_alloc(pz); + dapl_rmr = dapl_rmr_alloc(dapl_pz); if (rmr == NULL) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, @@ -372,18 +364,18 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han goto bail; } - status = dapl_ib_mw_alloc(rmr); + status = dapl_ib_mw_alloc(dapl_rmr); if (status != DAT_SUCCESS) { - dapl_rmr_dealloc(rmr); + dapl_rmr_dealloc(dapl_rmr); status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY_REGION); goto bail; } - atomic_inc(&pz->pz_ref_count); + atomic_inc(&dapl_pz->pz_ref_count); - *rmr_handle = rmr; + *rmr = (struct dat_rmr *)dapl_rmr; bail: return status; @@ -404,12 +396,12 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) +u32 dapl_rmr_free(struct dat_rmr *rmr_handle) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } @@ -438,12 +430,12 @@ bail: return status; } -u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) +u32 dapl_rmr_query(struct dat_rmr *rmr_handle, struct dat_rmr_param *rmr_param) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } Index: linux-kernel/dat-provider/dapl_cr_util.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_util.c (revision 2564) +++ linux-kernel/dat-provider/dapl_cr_util.c (working copy) @@ -49,17 +49,12 @@ struct dapl_cr *dapl_cr_alloc(struct dap /* zero the structure */ memset(cr_ptr, 0, sizeof *cr_ptr); - /* - * initialize the header - */ - cr_ptr->header.provider = ia_ptr->header.provider; - cr_ptr->header.magic = DAPL_MAGIC_CR; - cr_ptr->header.handle_type = DAT_HANDLE_TYPE_CR; - cr_ptr->header.owner_ia = ia_ptr; - cr_ptr->header.user_context.as_64 = 0; - cr_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&cr_ptr->header.ia_list_entry); - spin_lock_init(&cr_ptr->header.lock); + cr_ptr->cr.provider = ia_ptr->ia.provider; + cr_ptr->common.owner_ia = ia_ptr; + cr_ptr->common.user_context.as_64 = 0; + cr_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&cr_ptr->common.ia_list_entry); + spin_lock_init(&cr_ptr->common.lock); return cr_ptr; } @@ -69,9 +64,5 @@ struct dapl_cr *dapl_cr_alloc(struct dap */ void dapl_cr_free(struct dapl_cr *cr_ptr) { - dapl_os_assert(cr_ptr->header.magic == DAPL_MAGIC_CR || - cr_ptr->header.magic == DAPL_MAGIC_CR_DESTROYED); - /* reset magic to prevent reuse */ - cr_ptr->header.magic = DAPL_MAGIC_INVALID; kfree(cr_ptr); } Index: linux-kernel/dat-provider/dapl_psp.c =================================================================== --- linux-kernel/dat-provider/dapl_psp.c (revision 2564) +++ linux-kernel/dat-provider/dapl_psp.c (working copy) @@ -45,13 +45,13 @@ * the user. * * Input: - * ia_handle - * evd_handle + * ia + * evd * psp_flags * * Output: * conn_qual - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -61,9 +61,9 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ struct dapl_ia *ia_ptr; @@ -72,19 +72,19 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia u32 status = DAT_SUCCESS; int i; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } @@ -93,7 +93,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -117,10 +117,10 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; - sp_ptr->evd_handle = evd_handle; + sp_ptr->ia = ia; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -153,7 +153,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia if (status != DAT_SUCCESS) { atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -168,7 +168,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia * Return handle to the user */ *conn_qual = sp_ptr->conn_qual; - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -183,13 +183,13 @@ bail: * in a notification event. * * Input: - * ia_handle + * ia * conn_qual - * evd_handle + * evd * psp_flags * * Output: - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -198,9 +198,9 @@ bail: * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -208,24 +208,24 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -261,11 +261,11 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -298,7 +298,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han * wrong! Decrements the EVD refcount & release it. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -313,7 +313,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -325,7 +325,7 @@ bail: * Destroy a specific instance of a Service Point. * * Input: - * psp_handle + * psp * * Output: * none @@ -334,26 +334,25 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) +u32 dapl_psp_free(struct dat_sp *psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; DAPL_SP_STATE save_state; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if it has been established * and there are no current connections in progress. @@ -361,18 +360,18 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * container until the last connection is disconnected, after * which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); sp_ptr->listening = FALSE; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -389,8 +388,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han sp_ptr->cr_list_count == 0) { save_state = sp_ptr->state; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -407,7 +406,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free: PSP PENDING\n"); } @@ -416,13 +416,12 @@ bail: return status; } -u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) +u32 dapl_psp_query(struct dat_sp *psp, struct dat_psp_param *psp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || - ((struct dapl_sp *)psp_handle)->listening != TRUE) { + if (!psp || ((struct dapl_sp *)psp)->listening != TRUE) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } @@ -432,11 +431,11 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha goto bail; } - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; - psp_param->ia_handle = sp_ptr->ia_handle; + psp_param->ia = sp_ptr->ia; psp_param->conn_qual = sp_ptr->conn_qual; - psp_param->evd_handle = sp_ptr->evd_handle; + psp_param->evd = sp_ptr->evd; psp_param->psp_flags = sp_ptr->psp_flags; status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2564) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -43,11 +43,6 @@ static void dapl_ep_dealloc(struct dapl_ep *ep_ptr) { - dapl_os_assert(ep_ptr->header.magic == DAPL_MAGIC_EP); - - /* reset magic to prevent reuse */ - ep_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_cb_free(&ep_ptr->req_buffer); dapl_cb_free(&ep_ptr->recv_buffer); @@ -73,18 +68,12 @@ struct dapl_ep *dapl_ep_alloc(struct dap /* zero the structure */ memset(ep_ptr, 0, sizeof *ep_ptr + sizeof (struct sockaddr)); - /* - * initialize the header - */ - ep_ptr->header.provider = ia_ptr->header.provider; - ep_ptr->header.magic = DAPL_MAGIC_EP; - ep_ptr->header.handle_type = DAT_HANDLE_TYPE_EP; - ep_ptr->header.owner_ia = ia_ptr; - ep_ptr->header.user_context.as_64 = 0; - ep_ptr->header.user_context.as_ptr = NULL; - - dapl_llist_init_entry(&ep_ptr->header.ia_list_entry); - spin_lock_init(&ep_ptr->header.lock); + ep_ptr->ep.provider = ia_ptr->ia.provider; + ep_ptr->common.owner_ia = ia_ptr; + ep_ptr->common.user_context.as_64 = 0; + ep_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&ep_ptr->common.ia_list_entry); + spin_lock_init(&ep_ptr->common.lock); /* * Initialize the body @@ -105,7 +94,7 @@ struct dapl_ep *dapl_ep_alloc(struct dap ep_ptr->qp = NULL; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - ep_ptr->cm_handle = NULL; + ep_ptr->cm = NULL; atomic_set(&ep_ptr->req_count, 0); atomic_set(&ep_ptr->recv_count, 0); @@ -165,7 +154,7 @@ static u32 dapl_ep_check_recv_completion return DAT_SUCCESS; } -static u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, +static u32 dapl_ep_post_send_req(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -177,13 +166,13 @@ static u32 dapl_ep_post_send_req(DAT_EP_ struct dapl_cookie *cookie; u32 dat_status; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Synchronization ok since this buffer is only used for send @@ -281,11 +270,11 @@ enum dat_return_subtype dapl_ep_state_su return dat_status; } -u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) +u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + const struct dat_ep_attr *ep_attr, struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -294,18 +283,18 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create (%p, %p, %p, %p, %p, %p, %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -317,44 +306,34 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != DAT_HANDLE_NULL && - DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -370,12 +349,12 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -413,25 +392,25 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (recv_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_recv_dtos = 0; - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (request_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_request_dtos = 0; /* @@ -440,9 +419,9 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -450,7 +429,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -460,36 +439,36 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free - * evd_handles: + * pz: dat_pz_free + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = (struct dat_ep *)ep_ptr; bail: return dat_status; } -u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +u32 dapl_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *ep_attr, - DAT_EP_HANDLE *ep_handle) + struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -498,18 +477,18 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create_with_srq (%p, %p, %p, %p, %p, %p, %p %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, srq_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, srq, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -521,35 +500,27 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)-> + evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); @@ -560,13 +531,13 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * Verify the SRQ handle. It is an error to invoke this call with * a NULL handle */ - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -582,12 +553,12 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -636,26 +607,26 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (recv_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_recv_dtos = 0; } - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (request_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_request_dtos = 0; } @@ -665,9 +636,9 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -675,7 +646,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -685,24 +656,24 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free, - * evd_handles: + * pz: dat_pz_free, + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = ep_ptr; #endif /* notdef */ @@ -710,7 +681,7 @@ bail: return dat_status; } -u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_free(struct dat_ep *ep) { struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; @@ -718,15 +689,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle enum ib_qp_state save_qp_state; u32 dat_status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; param = &ep_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -744,7 +715,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle goto bail; } - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* If we are connected, issue a disconnect. If we are in the * disconnect_pending state, disconnect with the ABRUPT flag @@ -754,14 +725,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle /* * Invoke ep_disconnect to clean up outstanding connections */ - (void)dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + (void)dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); dapl_os_assert(ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_UNCONNECTED); /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); @@ -771,24 +743,24 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle * to set handles to NULL. Set handles to NULL so this routine * is idempotent. */ - if (param->pz_handle != NULL) { - atomic_dec(&((struct dapl_pz *)param->pz_handle)->pz_ref_count); - param->pz_handle = NULL; + if (param->pz != NULL) { + atomic_dec(&((struct dapl_pz *)param->pz)->pz_ref_count); + param->pz = NULL; } - if (param->recv_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->recv_evd_handle)-> + if (param->recv_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->recv_evd)-> evd_ref_count); - param->recv_evd_handle = NULL; + param->recv_evd = NULL; } - if (param->request_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->request_evd_handle)-> + if (param->request_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->request_evd)-> evd_ref_count); - param->request_evd_handle = NULL; + param->request_evd = NULL; } - if (param->connect_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->connect_evd_handle)-> + if (param->connect_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->connect_evd)-> evd_ref_count); - param->connect_evd_handle = NULL; + param->connect_evd = NULL; } /* @@ -804,7 +776,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ save_qp_state = ep_ptr->qp_state; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Free the QP. If the EP has never been used, the QP is invalid */ if (save_qp_state != DAPL_QP_STATE_UNATTACHED) { @@ -828,7 +800,7 @@ bail: } -u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, @@ -839,7 +811,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_connect(%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", - ep_handle, + ep, remote_ia_address->sa_data[2], remote_ia_address->sa_data[3], remote_ia_address->sa_data[4], @@ -847,19 +819,19 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han remote_conn_qual, timeout, private_data_size, private_data, qos, connect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state. The connection handle must be good * at this point. */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD)) { + if (!ep_ptr->param.connect_evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; @@ -889,9 +861,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * * Assign the QP and release the lock. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ)) - { + if (ep_ptr->param.pz == NULL) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); @@ -899,7 +869,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } alloc_ep = *ep_ptr; - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, &alloc_ep, ep_ptr); if (dat_status != DAT_SUCCESS) { dat_status = @@ -908,15 +878,15 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han goto bail; } - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * PZ shouldn't have changed since we're only racing with * dapl_cr_accept() */ if (ep_ptr->qp_state != DAPL_QP_STATE_UNATTACHED) { /* Bail, cleaning up. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = dapl_ib_qp_free(&alloc_ep); if (dat_status != DAT_SUCCESS) { dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -932,42 +902,30 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->qp = alloc_ep.qp; ep_ptr->qp_state = alloc_ep.qp_state; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); } /* * We do state checks and transitions under lock. * The only code we're racing against is dapl_cr_accept. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required - */ - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL - && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required + */ + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); goto bail; @@ -977,12 +935,12 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * we need to attach one now. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_TEP); @@ -991,8 +949,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1003,8 +961,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han /* * At this point we only support one QOS level */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_MODEL_NOT_SUPPORTED, 0); goto bail; } @@ -1018,7 +976,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * At this point we're committed, and done with the endpoint * except for the connect, so we can drop the lock. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Copy the connection qualifiers */ memcpy(ep_ptr->param.remote_ia_address_ptr, remote_ia_address, @@ -1026,7 +984,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->param.remote_port_qual = remote_conn_qual; memcpy(ep_ptr->private_data,private_data,private_data_size); - dat_status = dapl_ib_connect(ep_ptr, remote_ia_address, + dat_status = dapl_ib_connect((struct dat_ep *)ep_ptr, remote_ia_address, remote_conn_qual, timeout, private_data_size, ep_ptr->private_data); @@ -1043,10 +1001,10 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han if (dat_status == DAT_ERROR(DAT_INTERNAL_ERROR, 1)) { dapl_evd_post_connection_event((struct dapl_evd *) ep_ptr->param. - connect_evd_handle, + connect_evd, DAT_CONNECTION_EVENT_UNREACHABLE, - (DAT_HANDLE) ep_ptr, 0, - NULL); + (struct dat_ep *)ep_ptr, + 0, NULL); dat_status = DAT_SUCCESS; } } @@ -1058,7 +1016,7 @@ bail: return dat_status; } -u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, +u32 dapl_ep_dup_connect(struct dat_ep *ep, struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { @@ -1067,13 +1025,13 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep DAT_CONN_QUAL remote_conn_qual; u32 dat_status; - ep_dup_ptr = (struct dapl_ep *)ep_dup_handle; + ep_dup_ptr = (struct dapl_ep *)ep_dup; /* * Verify the dup handle, which must be connected. All other * parameters will be verified by dapl_ep_connect */ - if (DAPL_BAD_HANDLE(ep_dup_handle, DAPL_MAGIC_EP)) { + if (!ep_dup) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1089,10 +1047,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep * there is a problem. Do this under a lock and pull out * the connection parameters for atomicity. */ - spin_lock_irqsave(&ep_dup_ptr->header.lock, ep_dup_ptr->header.flags); + spin_lock_irqsave(&ep_dup_ptr->common.lock, ep_dup_ptr->common.flags); if (ep_dup_ptr->param.ep_state != DAT_EP_STATE_CONNECTED) { - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_dup_ptr)); @@ -1100,10 +1058,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep } remote_ia_address_ptr = ep_dup_ptr->param.remote_ia_address_ptr; remote_conn_qual = ep_dup_ptr->param.remote_port_qual; - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); - dat_status = dapl_ep_connect(ep_handle, + dat_status = dapl_ep_connect(ep, remote_ia_address_ptr, remote_conn_qual, timeout, @@ -1114,7 +1072,7 @@ bail: return dat_status; } -u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags disconnect_flags) { struct dapl_ep *ep_ptr; @@ -1124,14 +1082,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_disconnect (%p, %x)\n", - ep_handle, disconnect_flags); + ep, disconnect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1141,12 +1099,12 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * Do the verification of parameters and the state change * atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Disconnecting a disconnected EP is a no-op. */ if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; goto bail; } @@ -1160,8 +1118,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ ep_ptr->param.ep_state != DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECT_PENDING) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1173,8 +1131,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * If in state DISCONNECT_PENDING then this must be an * ABRUPT disconnect */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } @@ -1187,18 +1145,19 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ */ ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); /* disconnect and make sure we get no callbacks */ (void)dapl_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); /* clean up connection state */ dapl_sp_remove_ep(ep_ptr); - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; dapl_evd_post_connection_event(evd_ptr, DAT_CONNECTION_EVENT_DISCONNECTED, - (DAT_HANDLE) ep_ptr, 0, NULL); + (struct dat_ep *)ep_ptr, 0, + NULL); dat_status = DAT_SUCCESS; goto bail; } @@ -1216,14 +1175,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECT_PENDING; } - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); dat_status = dapl_ib_disconnect(ep_ptr, disconnect_flags); /* * Reacquire the lock and make sure we didn't get a callback * that cleaned up. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (disconnect_flags == DAT_CLOSE_ABRUPT_FLAG && ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { /* @@ -1234,8 +1193,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * on the way out. */ cr_ptr = ep_ptr->cr_ptr; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); if (cr_ptr != NULL) { dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, @@ -1246,13 +1205,13 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ DAT_CONNECTION_EVENT_DISCONNECTED, NULL, cr_ptr->sp_ptr); } else { - dapl_evd_connection_callback(ep_ptr->cm_handle, + dapl_evd_connection_callback(ep_ptr->cm, DAT_CONNECTION_EVENT_DISCONNECTED, NULL, ep_ptr); } } else - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, @@ -1261,7 +1220,7 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ return dat_status; } -u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, enum dat_ep_state *ep_state, +u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle) { struct dapl_ep *ep_ptr; @@ -1269,14 +1228,14 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_get_status (%p, %p, %p, %p)\n", - ep_handle, ep_state, in_dto_idle, out_dto_idle); + ep, ep_state, in_dto_idle, out_dto_idle); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1304,15 +1263,15 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ } static inline u32 -dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, +dapl_ep_modify_validate_parameters(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param, struct dapl_ia **ia_ptr, struct dapl_ep **ep_ptr, struct dat_ep_attr *ep_attr_ptr) { - struct dapl_ia *ia; - struct dapl_ep *ep; + struct dapl_ia *dapl_ia; + struct dapl_ep *dapl_ep; struct dat_ep_attr ep_attr; struct dat_ep_attr ep_attr_limit; struct dat_ep_attr ep_attr_request; @@ -1321,14 +1280,14 @@ dapl_ep_modify_validate_parameters(DAT_E *ia_ptr = NULL; *ep_ptr = NULL; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep = (struct dapl_ep *)ep_handle; - ia = ep->header.owner_ia; + dapl_ep = (struct dapl_ep *)ep; + dapl_ia = dapl_ep->common.owner_ia; /* * Verify parameters valid in current EP state @@ -1348,10 +1307,10 @@ dapl_ep_modify_validate_parameters(DAT_E * TENTATIVE_CONNECTION_PENDING (psp PROVIDER allocated EP) */ if ((mask & DAT_EP_FIELD_PZ_HANDLE) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } @@ -1367,41 +1326,21 @@ dapl_ep_modify_validate_parameters(DAT_E DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS | DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV | DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV)) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_RESERVED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_RESERVED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } /* * Validate handles being modified */ - if (mask & DAT_EP_FIELD_PZ_HANDLE) { - if (ep_param->pz_handle != NULL && - DAPL_BAD_HANDLE(ep_param->pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { - if (ep_param->recv_evd_handle != NULL && - (DAPL_BAD_HANDLE(ep_param->recv_evd_handle, DAPL_MAGIC_EVD) - || !((struct dapl_evd *)ep_param->recv_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - - if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) { - if (ep_param->request_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->request_evd_handle, - DAPL_MAGIC_EVD)) { + if (ep_param->recv_evd != NULL && + !((struct dapl_evd *)ep_param->recv_evd)-> + evd_flags & DAT_EVD_DTO_FLAG) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; @@ -1409,10 +1348,8 @@ dapl_ep_modify_validate_parameters(DAT_E } if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) { - if (ep_param->connect_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->connect_evd_handle, - DAPL_MAGIC_EVD) - && !(((struct dapl_evd *)ep_param->connect_evd_handle)-> + if (ep_param->connect_evd != NULL && + !(((struct dapl_evd *)ep_param->connect_evd)-> evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1423,8 +1360,8 @@ dapl_ep_modify_validate_parameters(DAT_E /* * Validate the attributes against the HCA limits */ - ep_attr = ep->param.ep_attr; - ep_attr_limit = ia->hca_ptr->ep_attr; + ep_attr = dapl_ep->param.ep_attr; + ep_attr_limit = dapl_ia->hca_ptr->ep_attr; ep_attr_request = ep_param->ep_attr; if (mask & DAT_EP_FIELD_EP_ATTR_MAX_MESSAGE_SIZE) { @@ -1471,7 +1408,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS) { if (ep_attr_request.max_recv_dtos > ep_attr_limit.max_recv_dtos - || (ep_param->recv_evd_handle == DAT_HANDLE_NULL + || (ep_param->recv_evd == NULL && (ep_attr_request.max_recv_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1484,7 +1421,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS) { if (ep_attr_request.max_request_dtos > ep_attr_limit.max_request_dtos - || (ep_param->request_evd_handle == DAT_HANDLE_NULL + || (ep_param->request_evd == NULL && (ep_attr_request.max_request_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1541,15 +1478,15 @@ dapl_ep_modify_validate_parameters(DAT_E } } - *ia_ptr = ia; - *ep_ptr = ep; + *ia_ptr = dapl_ia; + *ep_ptr = dapl_ep; *ep_attr_ptr = ep_attr; bail: return dat_status; } -u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, +u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param) { struct dapl_ia *ia; @@ -1575,7 +1512,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand boolean_t rqst_iov_used = FALSE; boolean_t recv_iov_used = FALSE; - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep1, &ep_attr1); @@ -1589,16 +1526,16 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep = *ep1; alloc_ep.param.ep_attr = ep_attr1; if (mask & DAT_EP_FIELD_PZ_HANDLE) - alloc_ep.param.pz_handle = ep_param->pz_handle; + alloc_ep.param.pz = ep_param->pz; if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) - alloc_ep.param.recv_evd_handle = ep_param->recv_evd_handle; + alloc_ep.param.recv_evd = ep_param->recv_evd; if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) - alloc_ep.param.request_evd_handle = ep_param->request_evd_handle; + alloc_ep.param.request_evd = ep_param->request_evd; if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) - alloc_ep.param.connect_evd_handle = ep_param->connect_evd_handle; + alloc_ep.param.connect_evd = ep_param->connect_evd; /* * Allocate everything that might be needed. @@ -1651,18 +1588,18 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * There's a lot of work done here, but there should be no * allocation or blocking. */ - spin_lock_irqsave(&ep1->header.lock, ep1->header.flags); + spin_lock_irqsave(&ep1->common.lock, ep1->common.flags); /* * Revalidate parameters; make sure that races haven't * changed anything important. */ - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep2, &ep_attr2); if (DAT_SUCCESS != dat_status) { - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); goto bail; } @@ -1691,10 +1628,10 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * is stuff changed by dapl_cr_accept, and neither PZ nor EVD is in that * list. */ - new_ep.param.pz_handle = alloc_ep.param.pz_handle; - new_ep.param.recv_evd_handle = alloc_ep.param.recv_evd_handle; - new_ep.param.request_evd_handle = alloc_ep.param.request_evd_handle; - new_ep.param.connect_evd_handle = alloc_ep.param.connect_evd_handle; + new_ep.param.pz = alloc_ep.param.pz; + new_ep.param.recv_evd = alloc_ep.param.recv_evd; + new_ep.param.request_evd = alloc_ep.param.request_evd; + new_ep.param.connect_evd = alloc_ep.param.connect_evd; /* Deal with each of the allocation fields. */ if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS @@ -1756,50 +1693,50 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand if (DAPL_QP_STATE_UNATTACHED != new_ep.qp_state) { dat_status = dapl_ib_qp_modify(ia, ep2, &ep_attr2); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep2->header.lock, - ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, + ep2->common.flags); goto bail; } } *ep2 = new_ep; - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); /* * Modify reference counts, incrementing new ones * and then decrementing old ones (so if they're the same * the refcount never drops to zero). */ - tmp_pz = (struct dapl_pz *)new_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)new_ep.param.pz; if (tmp_pz) atomic_inc(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.request_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); /* decreament the old reference counts */ - tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz; if (tmp_pz) atomic_dec(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); @@ -1847,7 +1784,7 @@ bail: return dat_status; } -u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1858,12 +1795,12 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_read (%p, %d, %p, %p, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1878,7 +1815,7 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE return dat_status; } -u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1888,12 +1825,12 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_write (%p, %d, %p, %P, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1908,7 +1845,7 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL return dat_status; } -u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1919,15 +1856,15 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_recv (%p, %d, %p, %P, %x)\n", - ep_handle, num_segments, local_iov, user_cookie.as_64, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *) ep_handle; + ep_ptr = (struct dapl_ep *) ep; /* * Synchronization ok since this buffer is only used for receive @@ -1957,7 +1894,7 @@ bail: return status; } -u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1967,11 +1904,11 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_send (%p, %d, %p, %P, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1985,21 +1922,21 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h return dat_status; } -u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) +u32 dapl_ep_query(struct dat_ep *ep, struct dat_ep_param *ep_param) { struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_query (%p, %p)\n", - ep_handle, ep_param); + ep, ep_param); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } @@ -2015,12 +1952,10 @@ u32 dapl_ep_query(DAT_EP_HANDLE ep_handl * N.B. Just slam all values into the user structure, there * is nothing to be gained by checking for each bit. */ - if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { + if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) /* obtain the remote IP address */ - status = - dapl_ib_cm_remote_addr((DAT_HANDLE) ep_handle, - &ep_ptr->remote_ia_address); - } + status = dapl_ib_cm_remote_addr(ep, &ep_ptr->remote_ia_address); + ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *)&ep_ptr->remote_ia_address; *ep_param = ep_ptr->param; @@ -2029,21 +1964,21 @@ bail: return status; } -u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, int *nbufs_allocate, +u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocate, int *bufs_alloc_span) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", - ep_handle, nbufs_allocate, bufs_alloc_span); + ep, nbufs_allocate, bufs_alloc_span); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2056,17 +1991,17 @@ bail: } -u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_reset(struct dat_ep *ep) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2088,7 +2023,7 @@ bail: return dat_status; } -u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -2096,14 +2031,14 @@ u32 dapl_ep_set_watermark(DAT_EP_HANDLE u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", - ep_handle, soft_high_watermark, hard_high_watermark); + ep, soft_high_watermark, hard_high_watermark); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; Index: linux-kernel/dat-provider/dapl_rsp.c =================================================================== --- linux-kernel/dat-provider/dapl_rsp.c (revision 2564) +++ linux-kernel/dat-provider/dapl_rsp.c (working copy) @@ -44,13 +44,13 @@ * event * * Input: - * ia_handle + * ia * conn_qual - * ep_handle - * evd_handle + * ep + * evd * * Output: - * rsp_handle + * rsp * * Returns: * DAT_SUCCESS @@ -59,9 +59,9 @@ * DAT_INVALID_STATE * DAT_CONN_QUAL_IN_USE */ -u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle) +u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -70,39 +70,39 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free conn_qual: %x EP: %p\n", - conn_qual, ep_handle); + conn_qual, ep); - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (rsp_handle == NULL) { + if (rsp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -126,11 +126,11 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Fill out the RSP args */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = 0; - sp_ptr->ep_handle = ep_handle; + sp_ptr->ep = ep; /* * Take a reference on the EVD handle @@ -162,7 +162,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han * the state to FREE, so we know the call failed. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; sp_ptr->state = DAPL_SP_STATE_FREE; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -178,7 +178,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *rsp_handle = (DAT_RSP_HANDLE)sp_ptr; + *rsp = (struct dat_sp *)sp_ptr; bail: return status; @@ -190,7 +190,7 @@ bail: * Destroy a specific instance of a Reserved Service Point. * * Input: - * rsp_handle + * rsp * * Output: * none @@ -199,50 +199,49 @@ bail: * DAT_SUCCESS * DAT_INVALID_HANDLE */ -u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) +u32 dapl_rsp_free(struct dat_sp *rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp_handle); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_RSP)) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp); + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if there are no connections. If * we defer removing the sp it becomes something of a zombie * container until disconnection, after which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * Make sure we don't leave a dangling EP. If the state is still * RESERVED then the RSP still owns it. */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -258,8 +257,8 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han /* This RSP has never been used. Clean it up */ sp_ptr->listening = FALSE; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -275,20 +274,20 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); } bail: return status; } -u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) +u32 dapl_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *rsp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { + if (!rsp) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } @@ -298,15 +297,15 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha goto bail; } - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Fill in the RSP params */ - rsp_param->ia_handle = sp_ptr->ia_handle; + rsp_param->ia = sp_ptr->ia; rsp_param->conn_qual = sp_ptr->conn_qual; - rsp_param->evd_handle = sp_ptr->evd_handle; - rsp_param->ep_handle = sp_ptr->ep_handle; + rsp_param->evd = sp_ptr->evd; + rsp_param->ep = sp_ptr->ep; status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_set_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_set_consumer_context.c (revision 2564) +++ linux-kernel/dat-provider/dapl_set_consumer_context.c (working copy) @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_set_consumer_context - * - * Set a consumer context in the provided dat_handle - * - * Input: - * dat_handle - * context - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - */ -u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, union dat_context context) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - header->user_context = context; - - bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_ia_open.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_open.c (revision 2564) +++ linux-kernel/dat-provider/dapl_ia_open.c (working copy) @@ -45,18 +45,18 @@ * and calls this function. */ u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr) + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr) { u32 dat_status = DAT_SUCCESS; struct dat_provider *provider; struct dapl_hca *hca_ptr = NULL; - struct dapl_ia *ia_ptr = NULL; - struct dapl_evd *evd_ptr; + struct dapl_ia *ia = NULL; + struct dapl_evd *evd; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_open (%s, %d, %p, %p)\n", - name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); + name, async_evd_qlen, async_evd_ptr, ia_ptr); dat_status = dapl_provider_list_search(name, &provider); if (DAT_SUCCESS != dat_status) { @@ -64,25 +64,25 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - /* ia_handle_ptr and async_evd_handle_ptr cannot be NULL */ - if (ia_handle_ptr == NULL) { + /* ia_ptr and async_evd_ptr cannot be NULL */ + if (ia_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } - if (async_evd_handle_ptr == NULL) { + if (async_evd_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; } /* initialize the caller's param */ - *ia_handle_ptr = DAT_HANDLE_NULL; + *ia_ptr = NULL; hca_ptr = (struct dapl_hca *)provider->extension; atomic_inc(&hca_ptr->handle_ref_count); /* Allocate and initialize ia structure */ - ia_ptr = dapl_ia_alloc(provider, hca_ptr); - if (!ia_ptr) { + ia = dapl_ia_alloc(provider, hca_ptr); + if (!ia) { atomic_dec(&hca_ptr->handle_ref_count); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -93,17 +93,16 @@ u32 dapl_ia_open(const char *name, int a * use the one passed in (if non-NULL) or create one */ - evd_ptr = (struct dapl_evd *)*async_evd_handle_ptr; - if (evd_ptr) { - if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || - !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { + evd = (struct dapl_evd *)*async_evd_ptr; + if (evd) { + if (!(evd->evd_flags & DAT_EVD_ASYNC_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - if (evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle != + if (evd->common.owner_ia->hca_ptr->ib_hca_handle != hca_ptr->ib_hca_handle) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, @@ -111,8 +110,8 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - ia_ptr->cleanup_async_error_evd = FALSE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = FALSE; + ia->async_error_evd = evd; } else { /* Verify we have >0 length, and let the provider check the size */ if (async_evd_qlen <= 0) { @@ -120,32 +119,33 @@ u32 dapl_ia_open(const char *name, int a DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } - dat_status = dapl_evd_internal_create(ia_ptr, async_evd_qlen, + dat_status = dapl_evd_internal_create(ia, + async_evd_qlen, DAT_UPCALL_DISABLE, NULL, DAT_EVD_ASYNC_FLAG, - &evd_ptr); + &evd); if (dat_status != DAT_SUCCESS) goto bail; - atomic_inc(&evd_ptr->evd_ref_count); + atomic_inc(&evd->evd_ref_count); /* Register the handlers associated with the async EVD. */ - dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); + dat_status = dapl_ia_setup_callbacks(ia, evd); /* Assign the EVD so it gets cleaned up */ - ia_ptr->cleanup_async_error_evd = TRUE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = TRUE; + ia->async_error_evd = evd; if (dat_status != DAT_SUCCESS) goto bail; } dat_status = DAT_SUCCESS; - *ia_handle_ptr = ia_ptr; - *async_evd_handle_ptr = evd_ptr; + *ia_ptr = (struct dat_ia *)ia; + *async_evd_ptr = (struct dat_evd *)evd; - bail: - if (dat_status != DAT_SUCCESS) { - if (ia_ptr) - dapl_ia_close(ia_ptr, DAT_CLOSE_ABRUPT_FLAG); - } +bail: + if (dat_status != DAT_SUCCESS) + if (ia) + dapl_ia_close((struct dat_ia *)ia, + DAT_CLOSE_ABRUPT_FLAG); dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_ia_open () returns 0x%x\n", dat_status); Index: linux-kernel/dat-provider/dapl_mr_util.c =================================================================== --- linux-kernel/dat-provider/dapl_mr_util.c (revision 2564) +++ linux-kernel/dat-provider/dapl_mr_util.c (working copy) @@ -59,7 +59,7 @@ u64 dapl_mr_get_address(DAT_REGION_DESCR case DAT_MEM_TYPE_VIRTUAL: return (u64) (unsigned long) desc.for_va; case DAT_MEM_TYPE_LMR: - lmr = (struct dapl_lmr *)desc.for_lmr_handle; + lmr = (struct dapl_lmr *)desc.for_lmr; /* Since this function is recoursive we cannot inline it */ return dapl_mr_get_address(lmr->param.region_desc, Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2564) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -51,7 +51,7 @@ static void dapl_evd_upcall_trigger(stru } for (;;) { - status = dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); + status = dapl_evd_dequeue((struct dat_evd *)evd, &event); if (DAT_SUCCESS != status) return; @@ -157,17 +157,12 @@ struct dapl_evd *dapl_evd_alloc(struct d /* zero the structure */ memset(evd, 0, sizeof *evd); - /* - * initialize the header - */ - evd->header.provider = ia->header.provider; - evd->header.magic = DAPL_MAGIC_EVD; - evd->header.handle_type = DAT_HANDLE_TYPE_EVD; - evd->header.owner_ia = ia; - evd->header.user_context.as_64 = 0; - evd->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&evd->header.ia_list_entry); - spin_lock_init(&evd->header.lock); + evd->evd.provider = ia->ia.provider; + evd->common.owner_ia = ia; + evd->common.user_context.as_64 = 0; + evd->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&evd->common.ia_list_entry); + spin_lock_init(&evd->common.lock); /* * Initialize the body @@ -249,7 +244,6 @@ static u32 dapl_evd_dealloc(struct dapl_ u32 status = DAT_SUCCESS; struct dapl_ia *ia; - dapl_os_assert(evd->header.magic == DAPL_MAGIC_EVD); dapl_os_assert(atomic_read(&evd->evd_ref_count) == 0); /* @@ -257,7 +251,7 @@ static u32 dapl_evd_dealloc(struct dapl_ * up from it. */ if (evd->cq != NULL) { - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; status = dapl_ib_cq_free(evd); if (status != DAT_SUCCESS) { @@ -265,12 +259,6 @@ static u32 dapl_evd_dealloc(struct dapl_ } } - /* - * We should now be safe to invalidate the EVD; reset the - * magic to prevent reuse. - */ - evd->header.magic = DAPL_MAGIC_INVALID; - /* If the ring buffer allocation failed, then the dapl_rbuf_destroy */ /* function will detect that the ring buffer's internal data (ex. base */ /* pointer) are invalid and will handle the situation appropriately */ @@ -314,14 +302,14 @@ static struct dat_event *dapl_evd_get_ev struct dat_event *event; if (evd->evd_producer_locking_needed) { - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); } event = (struct dat_event *) dapl_rbuf_remove(&evd->free_event_queue); /* Release the lock if it was taken and the call failed. */ if (!event && evd->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); return event; } @@ -348,8 +336,8 @@ static void dapl_evd_post_event(struct d dapl_os_assert(evd->evd_state == DAPL_EVD_STATE_OPEN); if (evd->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd->header.lock, - evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, + evd->common.flags); dapl_evd_upcall_trigger(evd); } @@ -363,12 +351,12 @@ static void dapl_evd_format_overflow_eve { struct dapl_ia *ia; - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; event->event_number = DAT_ASYNC_ERROR_EVD_OVERFLOW; event->event_data.asynch_error_event_data.dat_handle = - (DAT_HANDLE) ia; + (struct dat_ia *)ia; } /* @@ -409,9 +397,9 @@ static struct dat_event *dapl_evd_get_an event = dapl_evd_get_event(evd); if (!event) dapl_evd_post_overflow_event( - evd->header.owner_ia->async_error_evd, evd); + evd->common.owner_ia->async_error_evd, evd); else { - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; event->event_number = evno; } @@ -419,10 +407,10 @@ static struct dat_event *dapl_evd_get_an } u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp, struct sockaddr *ia_address, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle) + struct dat_cr *cr) { struct dat_event *event; event = dapl_evd_get_and_init_event(evd, DAT_CONNECTION_REQUEST_EVENT); @@ -436,11 +424,11 @@ u32 dapl_evd_post_cr_arrival_event(struc return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event->event_data.cr_arrival_event_data.sp_handle = sp_handle; + event->event_data.cr_arrival_event_data.sp = sp; event->event_data.cr_arrival_event_data.local_ia_address_ptr = ia_address; event->event_data.cr_arrival_event_data.conn_qual = conn_qual; - event->event_data.cr_arrival_event_data.cr_handle = cr_handle; + event->event_data.cr_arrival_event_data.cr = cr; dapl_evd_post_event(evd, event); @@ -449,7 +437,7 @@ u32 dapl_evd_post_cr_arrival_event(struc u32 dapl_evd_post_connection_event(struct dapl_evd *evd, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, int private_data_size, void *private_data) { @@ -466,7 +454,7 @@ u32 dapl_evd_post_connection_event(struc DAT_RESOURCE_MEMORY); } - event->event_data.connect_event_data.ep_handle = ep_handle; + event->event_data.connect_event_data.ep = ep; event->event_data.connect_event_data.private_data_size = private_data_size; event->event_data.connect_event_data.private_data = private_data; @@ -492,7 +480,7 @@ static u32 dapl_evd_post_async_error_eve return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event->event_data.asynch_error_event_data.dat_handle = (DAT_HANDLE) ia; + event->event_data.asynch_error_event_data.dat_handle = ia; dapl_evd_post_event(evd, event); @@ -539,12 +527,8 @@ static void dapl_evd_wc_to_event(struct ep = cookie->ep; dapl_os_assert(NULL != ep); - if (ep->header.magic != DAPL_MAGIC_EP) { - /* ep may have been freed, just return */ - return; - } - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; switch (cookie->type) { case DAPL_COOKIE_TYPE_DTO: @@ -563,7 +547,7 @@ static void dapl_evd_wc_to_event(struct } event->event_number = DAT_DTO_COMPLETION_EVENT; - event_data->ep_handle = cookie->ep; + event_data->ep = (struct dat_ep *)cookie->ep; event_data->user_cookie = cookie->val.dto.cookie; event_data->status = dto_status; @@ -606,7 +590,7 @@ static void dapl_evd_wc_to_event(struct event->event_number = DAT_RMR_BIND_COMPLETION_EVENT; event_data = &event->event_data.rmr_completion_event_data; - event_data->rmr_handle = cookie->val.rmr.rmr; + event_data->rmr = (struct dat_rmr *)cookie->val.rmr.rmr; event_data->user_cookie = cookie->val.rmr.cookie; if (dto_status == DAT_DTO_SUCCESS) { @@ -654,7 +638,7 @@ void dapl_evd_qp_async_error_callback(st u32 status; ep = (struct dapl_ep *)context; - async_evd = (struct dapl_evd *)ep->header.owner_ia->async_error_evd; + async_evd = (struct dapl_evd *)ep->common.owner_ia->async_error_evd; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback: " @@ -679,7 +663,7 @@ void dapl_evd_qp_async_error_callback(st * we are not interested in. */ (void)dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback () returns\n"); @@ -698,11 +682,11 @@ void dapl_evd_cq_async_error_callback(st if (!evd) dapl_os_panic("NULL == context\n"); - async_evd = evd->header.owner_ia->async_error_evd; + async_evd = evd->common.owner_ia->async_error_evd; status = dapl_evd_post_async_error_event(async_evd, DAT_ASYNC_ERROR_EVD_OVERFLOW, - async_evd->header.owner_ia); + async_evd->common.owner_ia); if (status != DAT_SUCCESS) { dapl_os_panic("async EVD overflow\n"); @@ -738,7 +722,7 @@ void dapl_evd_un_async_error_callback(st */ dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_un_async_error_callback () returns\n"); @@ -764,14 +748,14 @@ void dapl_evd_connection_callback(struct "ep: %p\n", cm_handle, event, ep); - evd = (struct dapl_evd *)ep->param.connect_evd_handle; + evd = (struct dapl_evd *)ep->param.connect_evd; private_data_size = 0; /* * All operations effect the EP, so lock it once and unlock * when necessary */ - spin_lock_irqsave(&ep->header.lock, ep->header.flags); + spin_lock_irqsave(&ep->common.lock, ep->common.flags); switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: @@ -784,13 +768,13 @@ void dapl_evd_connection_callback(struct /* If someone pulled the plug on the connection, just * exit */ - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); status = DAT_SUCCESS; break; } ep->param.ep_state = DAT_EP_STATE_CONNECTED; - ep->cm_handle = cm_handle; + ep->cm = cm_handle; if (private_data == NULL) { private_data_size = 0; } else { @@ -798,7 +782,7 @@ void dapl_evd_connection_callback(struct memcpy(ep->private_data, private_data, private_data_size); } - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -811,8 +795,8 @@ void dapl_evd_connection_callback(struct */ ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); break; } @@ -820,7 +804,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -828,7 +812,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -836,7 +820,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -844,14 +828,14 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, FALSE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } case DAT_CONNECTION_REQUEST_EVENT: default: { - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); evd = NULL; dapl_os_assert(0); /* shouldn't happen */ @@ -865,7 +849,7 @@ void dapl_evd_connection_callback(struct */ if (evd != NULL) { status = dapl_evd_post_connection_event(evd, event, - (DAT_HANDLE) ep, + (struct dat_ep *)ep, private_data_size, /* 0 except on CONNECT */ ep->private_data); @@ -877,10 +861,10 @@ void dapl_evd_connection_callback(struct * expect a callback on an ABRUPT disconnect. */ dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); - spin_lock_irqsave(&ep->header.lock, ep->header.flags); + spin_lock_irqsave(&ep->common.lock, ep->common.flags); ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); } } @@ -901,7 +885,6 @@ void dapl_evd_dto_callback(struct ib_cq evd = (struct dapl_evd *)user_context; dapl_os_assert(evd->cq == cq); - dapl_os_assert(evd->header.magic == DAPL_MAGIC_EVD); /* Read once. */ state = *(volatile DAPL_EVD_STATE *)&evd->evd_state; @@ -927,9 +910,9 @@ void dapl_evd_dto_callback(struct ib_cq status = dapl_ib_completion_notify(evd); if (DAT_SUCCESS != status) (void)dapl_evd_post_async_error_event( - evd->header.owner_ia->async_error_evd, + evd->common.owner_ia->async_error_evd, DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR, - evd->header.owner_ia); + evd->common.owner_ia); dapl_evd_upcall_trigger(evd); } @@ -1008,10 +991,10 @@ bail: return status; } -u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int min_qlen, +u32 dapl_evd_kcreate(struct dat_ia *ia_handle, int min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object *upcall, - enum dat_evd_flags flags, DAT_EVD_HANDLE *evd_handle) + enum dat_evd_flags flags, struct dat_evd **evd_handle) { struct dapl_ia *ia; struct dapl_evd *evd; @@ -1025,7 +1008,7 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha evd = NULL; *evd_handle = NULL; - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -1053,17 +1036,17 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha if (status != DAT_SUCCESS) goto bail; - *evd_handle = (DAT_EVD_HANDLE) evd; + *evd_handle = (struct dat_evd *)evd; bail: if (status != DAT_SUCCESS) if (evd) - dapl_evd_free(evd); + dapl_evd_free((struct dat_evd *)evd); return status; } -u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) +u32 dapl_evd_free(struct dat_evd *evd_handle) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; @@ -1072,7 +1055,7 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1083,11 +1066,11 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han goto bail; } - dapl_ia_unlink_evd(evd->header.owner_ia, evd); + dapl_ia_unlink_evd(evd->common.owner_ia, evd); status = dapl_evd_dealloc(evd); if (status != DAT_SUCCESS) { - dapl_ia_link_evd(evd->header.owner_ia, evd); + dapl_ia_link_evd(evd->common.owner_ia, evd); } bail: @@ -1097,7 +1080,7 @@ bail: return status; } -u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) +u32 dapl_evd_kquery(struct dat_evd *evd_handle, struct dat_evd_param *evd_param) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; @@ -1108,11 +1091,11 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h } /* Note: the spec. allows for events to be directed to a NULL EVD */ - /* with handle of type DAT_HANDLE_NULL. See 6.3.1 */ - if (DAT_HANDLE_NULL == evd_handle) { + /* with handle of type NULL. See 6.3.1 */ + if (NULL == evd_handle) { memset(evd_param, 0, sizeof *evd_param); } else { - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1128,7 +1111,7 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h * about reading the state variable atomically when we add * in waitable/unwaitable. */ - evd_param->ia_handle = evd->header.owner_ia; + evd_param->ia = (struct dat_ia *)evd->common.owner_ia; evd_param->evd_qlen = evd->qlen; evd_param->evd_flags = evd->evd_flags; evd_param->upcall_policy = evd->upcall_policy; @@ -1139,7 +1122,7 @@ bail: return status; } -u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +u32 dapl_evd_modify_upcall(struct dat_evd *evd_handle, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object *upcall) { @@ -1150,7 +1133,7 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1162,14 +1145,14 @@ bail: return status; } -u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) +u32 dapl_evd_post_se(struct dat_evd *evd_handle, const struct dat_event *event) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1196,7 +1179,7 @@ u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_ return status; } -u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) +u32 dapl_evd_dequeue(struct dat_evd *evd_handle, struct dat_event *event) { struct dapl_evd *evd; struct dat_event *local_event; @@ -1207,7 +1190,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1222,7 +1205,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ * restricts us from having multiple threads in CQ poll, and the * DAT API allows multiple threads in dat_evd_dequeue() */ - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); /* * Make sure there are no other waiters and the evd is active. @@ -1230,7 +1213,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ */ if (evd->evd_state != DAPL_EVD_STATE_OPEN || evd->catastrophic_overflow) { - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); status = DAT_ERROR(DAT_INVALID_STATE, 0); goto bail; } @@ -1249,7 +1232,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ status = DAT_ERROR(DAT_QUEUE_EMPTY, 0); } - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_dequeue () returns 0x%x\n", status); @@ -1257,7 +1240,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ return status; } -u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) +u32 dapl_evd_resize(struct dat_evd *evd_handle, int evd_qlen) { struct dapl_ia *ia; struct dapl_evd *evd; @@ -1267,13 +1250,13 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", evd_handle, evd_qlen); - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } evd = (struct dapl_evd *)evd_handle; - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; if (evd_qlen == evd->qlen) { status = DAT_SUCCESS; @@ -1285,7 +1268,7 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; } - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); pend_cnt = dapl_rbuf_count(&evd->pending_event_queue); if (pend_cnt > evd_qlen) { @@ -1302,6 +1285,6 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; bail: - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); return status; } Index: linux-kernel/dat-provider/dapl_ia_query.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_query.c (revision 2564) +++ linux-kernel/dat-provider/dapl_ia_query.c (working copy) @@ -38,12 +38,12 @@ * Provide the consumer with Interface Adapter and Provider parameters. * * Input: - * ia_handle + * ia * ia_mask * provider_mask * * Output: - * async_evd_handle + * async_evd * ia_parameters * provider_parameters * @@ -51,7 +51,7 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE *async_evd_handle, +u32 dapl_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -66,21 +66,21 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_query (%p, %p, %p, %p)\n", - ia_handle, - async_evd_handle, + ia, + async_evd, ia_attr, provider_attr); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (NULL != async_evd_handle) { - *async_evd_handle = ia_ptr->async_error_evd; + if (NULL != async_evd) { + *async_evd = (struct dat_evd *)ia_ptr->async_error_evd; } if (NULL != ia_attr) @@ -88,7 +88,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl if (NULL != provider_attr) { strncpy(provider_attr->provider_name, - ia_ptr->header.provider->device_name, + ia_ptr->ia.provider->device_name, DAT_NAME_MAX_LENGTH); provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; Index: linux-kernel/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel/dat-provider/dapl_sp_util.c (revision 2564) +++ linux-kernel/dat-provider/dapl_sp_util.c (working copy) @@ -65,22 +65,16 @@ struct dapl_sp *dapl_sp_alloc(struct dap /* zero the structure */ memset(sp_ptr, 0, sizeof *sp_ptr); - /* - * initialize the header - */ - sp_ptr->header.provider = ia_ptr->header.provider; - if (is_psp) { - sp_ptr->header.magic = DAPL_MAGIC_PSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_PSP; - } else { - sp_ptr->header.magic = DAPL_MAGIC_RSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_RSP; - } - sp_ptr->header.owner_ia = ia_ptr; - sp_ptr->header.user_context.as_64 = 0; - sp_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&sp_ptr->header.ia_list_entry); - spin_lock_init(&sp_ptr->header.lock); + sp_ptr->sp.provider = ia_ptr->ia.provider; + if (is_psp) + sp_ptr->sp.type = DAT_SP_TYPE_PSP; + else + sp_ptr->sp.type = DAT_SP_TYPE_RSP; + sp_ptr->common.owner_ia = ia_ptr; + sp_ptr->common.user_context.as_64 = 0; + sp_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&sp_ptr->common.ia_list_entry); + spin_lock_init(&sp_ptr->common.lock); /* * Initialize the Body (set to NULL above) @@ -107,12 +101,8 @@ struct dapl_sp *dapl_sp_alloc(struct dap */ void dapl_sp_dealloc(struct dapl_sp *sp_ptr) { - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); dapl_os_assert(dapl_llist_is_empty(&sp_ptr->cr_list_head)); - sp_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - kfree(sp_ptr); } @@ -134,11 +124,11 @@ void dapl_sp_dealloc(struct dapl_sp *sp_ */ void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) { - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_llist_add_tail(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry, cr_ptr); + &cr_ptr->common.ia_list_entry, cr_ptr); sp_ptr->cr_list_count++; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, sp_ptr->common.flags); } /* @@ -174,7 +164,7 @@ struct dapl_cr *dapl_sp_search_cr(struct if (cr_ptr->ib_cm_handle == ib_cm_handle) return cr_ptr; - cr_ptr = cr_ptr->header.ia_list_entry.flink->data; + cr_ptr = cr_ptr->common.ia_list_entry.flink->data; } while ((void *)cr_ptr != (void *)sp_ptr->cr_list_head->data); return NULL; @@ -208,7 +198,7 @@ void dapl_sp_remove_cr(struct dapl_sp *s } dapl_llist_remove_entry(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); sp_ptr->cr_list_count--; } @@ -238,13 +228,13 @@ void dapl_sp_remove_ep(struct dapl_ep *e if (cr_ptr != NULL) { sp_ptr = cr_ptr->sp_ptr; - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* Remove the CR from the queue */ dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); Index: linux-kernel/dat-provider/dapl_evd.h =================================================================== --- linux-kernel/dat-provider/dapl_evd.h (revision 2564) +++ linux-kernel/dat-provider/dapl_evd.h (working copy) @@ -60,14 +60,14 @@ extern void dapl_evd_un_async_error_call */ extern u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp_handle, struct sockaddr *ia_address, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle); + struct dat_cr *cr_handle); extern u32 dapl_evd_post_connection_event(struct dapl_evd *evd, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep_handle, int private_data_size, void *private_data); Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2564) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -46,23 +46,6 @@ * * *********************************************************************/ -typedef enum dapl_magic { - /* magic number values for verification & debug */ - DAPL_MAGIC_IA = 0xCafeF00d, - DAPL_MAGIC_EVD = 0xFeedFace, - DAPL_MAGIC_EP = 0xDeadBabe, - DAPL_MAGIC_LMR = 0xBeefCafe, - DAPL_MAGIC_RMR = 0xABadCafe, - DAPL_MAGIC_PZ = 0xDeafBeef, - DAPL_MAGIC_PSP = 0xBeadeD0c, - DAPL_MAGIC_RSP = 0xFab4Feed, - DAPL_MAGIC_SRQ = 0xC001Babe, - DAPL_MAGIC_CR = 0xBe12Cee1, - DAPL_MAGIC_CR_DESTROYED = 0xB12bDead, - DAPL_MAGIC_CNO = 0xDeadF00d, - DAPL_MAGIC_INVALID = 0xFFFFFFFF -} DAPL_MAGIC; - typedef enum dapl_evd_state { DAPL_EVD_STATE_OPEN, DAPL_EVD_STATE_DEAD = 0xDEAD @@ -92,23 +75,6 @@ typedef enum dapl_evd_completion { /********************************************************************* * * - * Macros * - * * - *********************************************************************/ - -/* - * Simple macro to verify a handle is bad. Conditions: - * - pointer is NULL - * - pointer is not word aligned - * - pointer's magic number is wrong - */ -#define DAPL_BAD_HANDLE(h, magicNum) ( \ - ((h) == NULL) || \ - ((unsigned long)(h) & 3) || \ - (((struct dapl_header *)(h))->magic != (magicNum))) - -/********************************************************************* - * * * Typedefs * * * *********************************************************************/ @@ -170,11 +136,8 @@ struct dapl_hca { struct dat_ep_attr ep_attr; }; -/* DAPL Objects always have the following header */ -struct dapl_header { - struct dat_provider *provider; /* required by DAT - must be first */ - DAPL_MAGIC magic; /* magic number for verification */ - enum dat_handle_type handle_type; +/* DAPL Objects always have the following items */ +struct dapl_common { struct dapl_ia *owner_ia; struct dapl_llist_entry ia_list_entry; union dat_context user_context; /* user context - opaque to DAPL */ @@ -182,9 +145,9 @@ struct dapl_header { unsigned long flags; /* saved lock flag values */ }; -/* struct dapl_ia maps to DAT_IA_HANDLE */ struct dapl_ia { - struct dapl_header header; /* WARNING: must be first */ + struct dat_ia ia; + struct dapl_common common; struct dapl_hca *hca_ptr; struct dapl_evd *async_error_evd; boolean_t cleanup_async_error_evd; @@ -200,9 +163,9 @@ struct dapl_ia { struct dapl_llist_entry *srq_list_head; /* SRQ queue */ }; -/* struct dapl_evd maps to DAT_EVD_HANDLE */ struct dapl_evd { - struct dapl_header header; /* WARNING: must be first */ + struct dat_evd evd; + struct dapl_common common; DAPL_EVD_STATE evd_state; enum dat_evd_flags evd_flags; @@ -230,9 +193,9 @@ struct dapl_evd { struct dat_upcall_object upcall; }; -/* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { - struct dapl_header header; + struct dat_ep ep; + struct dapl_common common; /* What the DAT Consumer asked for */ struct dat_ep_param param; @@ -241,7 +204,7 @@ struct dapl_ep { enum ib_qp_state qp_state; /* communications manager handle */ - struct dapl_cm_id *cm_handle; + struct dapl_cm_id *cm; /* store the remote IA address here, reference from the param * struct which only has a pointer, no storage */ @@ -267,33 +230,33 @@ struct dapl_ep { int send_iov_num; }; -/* struct dapl_srq maps to DAT_SRQ_HANDLE */ struct dapl_srq { - struct dapl_header header; /* WARNING: must be first */ + struct dat_srq srq; + struct dapl_common common; struct dat_srq_param param; atomic_t srq_ref_count; struct dapl_cookie_buffer recv_buffer; atomic_t recv_count; }; -/* struct dapl_pz maps to DAT_PZ_HANDLE */ struct dapl_pz { - struct dapl_header header; /* WARNING: must be first */ + struct dat_pz pz; + struct dapl_common common; struct ib_pd *pd; atomic_t pz_ref_count; }; -/* struct dapl_lmr maps to DAT_LMR_HANDLE */ struct dapl_lmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_lmr lmr; + struct dapl_common common; struct dat_lmr_param param; struct ib_mr *mr; atomic_t lmr_ref_count; }; -/* struct dapl_rmr maps to DAT_RMR_HANDLE */ struct dapl_rmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_rmr rmr; + struct dapl_common common; struct dat_rmr_param param; struct dapl_ep *ep; struct dapl_pz *pz; @@ -310,17 +273,17 @@ typedef enum dapl_sp_state { DAPL_SP_STATE_RSP_PENDING } DAPL_SP_STATE; -/* struct dapl_sp maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ struct dapl_sp { - struct dapl_header header; /* WARNING: must be first */ + struct dat_sp sp; + struct dapl_common common; DAPL_SP_STATE state; /* type and queue of the SP */ /* PSP/RSP PARAM fields */ - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; /* maintenance fields */ boolean_t listening; /* PSP is registered & active */ @@ -329,9 +292,9 @@ struct dapl_sp { int cr_list_count; /* count of CRs on queue */ }; -/* struct dapl_cr maps to DAT_CR_HANDLE */ struct dapl_cr { - struct dapl_header header; + struct dat_cr cr; + struct dapl_common common; /* for convenience the data is kept as a struct dat_cr_param. * however, the "local_endpoint" field is always NULL * so this wastes a pointer. This is probably ok to @@ -396,91 +359,88 @@ struct dapl_cookie { */ extern u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr); + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr); -extern u32 dapl_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dapl_ia_close(struct dat_ia *, enum dat_close_flags); -extern u32 dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, +extern u32 dapl_ia_query(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); /* helper functions */ -extern u32 dapl_set_consumer_context(DAT_HANDLE handle, +extern u32 dapl_set_consumer_context(void *handle, union dat_context context); -extern u32 dapl_get_consumer_context(DAT_HANDLE handle, +extern u32 dapl_get_consumer_context(void *handle, union dat_context *context); -extern u32 dapl_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type); - /* CR Functions */ -extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); +extern u32 dapl_cr_query(struct dat_cr *, struct dat_cr_param *); -extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +extern u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data); -extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); +extern u32 dapl_cr_reject(struct dat_cr *cr); -extern u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL handoff); +extern u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff); /* EVD Functions */ -extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, +extern u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, u64 *suggested_alignment); -extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, +extern u32 dapl_evd_kcreate(struct dat_ia *ia, int evd_min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE * evd_handle); + struct dat_evd ** evd); -extern u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_kquery(struct dat_evd *evd, struct dat_evd_param *evd_args); -extern u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall); -extern u32 dapl_evd_enable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_enable(struct dat_evd *evd); -extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_disable(struct dat_evd *evd); -extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen); +extern u32 dapl_evd_resize(struct dat_evd *evd, int evd_qlen); -extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_post_se(struct dat_evd *evd, const struct dat_event *event); -extern u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event); +extern u32 dapl_evd_dequeue(struct dat_evd *evd, struct dat_event *event); -extern u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_free(struct dat_evd *evd); -extern u32 dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_set_unwaitable(struct dat_evd *evd); -extern u32 dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_clear_unwaitable(struct dat_evd *evd); /* EP functions */ -extern u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE in_dto_completion_evd_handle, - DAT_EVD_HANDLE out_dto_completion_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, +extern u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_dto_completion_evd, + struct dat_evd *out_dto_completion_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *ep_parameters, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); +extern u32 dapl_ep_query(struct dat_ep *, struct dat_ep_param *); -extern u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask ep_args_mask, const struct dat_ep_param *ep_args); -extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, @@ -489,161 +449,161 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); -extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, +extern u32 dapl_ep_dup_connect(struct dat_ep *ep, + struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); -extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags close_flags); -extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle); -extern u32 dapl_ep_free(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_free(struct dat_ep *ep); -extern u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_reset(struct dat_ep *ep); -extern u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_ep_create_with_srq(struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq_handle, const struct dat_ep_attr *ep_attributes, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocated, int *bufs_alloc_span); -extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark); /* LMR functions */ -extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +extern u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address); -extern u32 dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); +extern u32 dapl_lmr_query(struct dat_lmr *, struct dat_lmr_param *); -extern u32 dapl_lmr_free(DAT_LMR_HANDLE); +extern u32 dapl_lmr_free(struct dat_lmr *); -extern u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); -extern u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); /* RMR Functions */ -extern u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle); +extern u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr); -extern u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_query(struct dat_rmr *rmr, struct dat_rmr_param *rmr_args); -extern u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_bind(struct dat_rmr *rmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *context); -extern u32 dapl_rmr_free(DAT_RMR_HANDLE); +extern u32 dapl_rmr_free(struct dat_rmr *rmr); /* PSP Functions */ -extern u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, +extern u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, +extern u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); +extern u32 dapl_psp_query(struct dat_sp *, struct dat_psp_param *); -extern u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle); +extern u32 dapl_psp_free(struct dat_sp *psp); /* RSP Functions */ -extern u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle); +extern u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp); -extern u32 dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); +extern u32 dapl_rsp_query(struct dat_sp *, struct dat_rsp_param *); -extern u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle); +extern u32 dapl_rsp_free(struct dat_sp *rsp); /* PZ Functions */ -extern u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle); +extern u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz); -extern u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_args); +extern u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_args); -extern u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle); +extern u32 dapl_pz_free(struct dat_pz *pz); /* SRQ functions */ -extern u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, +extern u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *srq_attr, - DAT_SRQ_HANDLE *srq_handle); + struct dat_srq **srq); -extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); +extern u32 dapl_srq_free(struct dat_srq *srq); -extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +extern u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie); -extern u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param); -extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto); -extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark); +extern u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark); /* * DAPL internal utility function prototypes Index: linux-kernel/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_util.c (revision 2564) +++ linux-kernel/dat-provider/dapl_ia_util.c (working copy) @@ -65,17 +65,12 @@ struct dapl_ia *dapl_ia_alloc(struct dat /* zero the structure */ memset(ia_ptr, 0, sizeof *ia_ptr); - /* - * initialize the header - */ - ia_ptr->header.provider = provider; - ia_ptr->header.magic = DAPL_MAGIC_IA; - ia_ptr->header.handle_type = DAT_HANDLE_TYPE_IA; - ia_ptr->header.owner_ia = ia_ptr; - ia_ptr->header.user_context.as_64 = 0; - ia_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&ia_ptr->header.ia_list_entry); - spin_lock_init(&ia_ptr->header.lock); + ia_ptr->ia.provider = provider; + ia_ptr->common.owner_ia = ia_ptr; + ia_ptr->common.user_context.as_64 = 0; + ia_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&ia_ptr->common.ia_list_entry); + spin_lock_init(&ia_ptr->common.lock); /* * initialize the body @@ -127,9 +122,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rmr_list_head)); while (rmr_ptr != NULL) { next_rmr_ptr = dapl_llist_next_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header. + &rmr_ptr->common. ia_list_entry); - dat_status = dapl_rmr_free(rmr_ptr); + dat_status = dapl_rmr_free((struct dat_rmr *)rmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rmr_free(%p) returns %x\n", @@ -141,9 +136,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rsp_list_head)); while (sp_ptr != NULL) { next_sp_ptr = dapl_llist_next_entry(&ia_ptr->rsp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); - dat_status = dapl_rsp_free(sp_ptr); + dat_status = dapl_rsp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rsp_free(%p) returns %x\n", @@ -155,7 +150,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->ep_list_head)); while (ep_ptr != NULL) { next_ep_ptr = dapl_llist_next_entry(&ia_ptr->ep_list_head, - &ep_ptr->header. + &ep_ptr->common. ia_list_entry); /* * Issue a disconnect if the EP needs it @@ -163,21 +158,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - || ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING - || ep_ptr->param.ep_state == - DAT_EP_STATE_DISCONNECT_PENDING) { - dat_status = - dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + DAT_EP_STATE_ACTIVE_CONNECTION_PENDING || + ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { + dat_status = dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_disconnect(%p) returns %x\n", ep_ptr, dat_status); } /* force the EP into error state to force flush all posted DTOs. */ - (void) dapl_modify_qp_state_to_error(ep_ptr->qp); + (void)dapl_modify_qp_state_to_error(ep_ptr->qp); - dat_status = dapl_ep_free(ep_ptr); + dat_status = dapl_ep_free((struct dat_ep *)ep_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_free(%p) returns %x\n", @@ -189,9 +182,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->lmr_list_head)); while (lmr_ptr != NULL) { next_lmr_ptr = dapl_llist_next_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header. + &lmr_ptr->common. ia_list_entry); - dat_status = dapl_lmr_free(lmr_ptr); + dat_status = dapl_lmr_free((struct dat_lmr *)lmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): lmr_free(%p) returns %x\n", @@ -213,7 +206,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status); next_sp_ptr = dapl_llist_next_entry(&ia_ptr->psp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); /* Remove CR's from this PSP and clean them up */ @@ -222,18 +215,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia while (cr_ptr != NULL) { next_cr_ptr = dapl_llist_next_entry( &sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); /* Remove the CR from the queue & cleanup */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); cr_ptr = next_cr_ptr; } - dat_status = dapl_psp_free(sp_ptr); + dat_status = dapl_psp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): psp_free(%p) returns %x\n", @@ -246,9 +240,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->pz_list_head)); while (pz_ptr != NULL) { next_pz_ptr = dapl_llist_next_entry(&ia_ptr->pz_list_head, - &pz_ptr->header. + &pz_ptr->common. ia_list_entry); - dat_status = dapl_pz_free(pz_ptr); + dat_status = dapl_pz_free((struct dat_pz *)pz_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): pz_free(%p) returns %x\n", @@ -266,11 +260,11 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->evd_list_head)); while (evd_ptr != NULL) { next_evd_ptr = dapl_llist_next_entry(&ia_ptr->evd_list_head, - &evd_ptr->header. + &evd_ptr->common. ia_list_entry); if (evd_ptr != ia_ptr->async_error_evd) { /* it isn't the async EVD; delete it. */ - dat_status = dapl_evd_free(evd_ptr); + dat_status = dapl_evd_free((struct dat_evd *)evd_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): evd_free(%p) returns %x\n", @@ -289,7 +283,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status = dapl_ia_teardown_callbacks(ia_ptr); atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - dat_status = dapl_evd_free(ia_ptr->async_error_evd); + dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != dat_status) dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -404,7 +398,7 @@ u32 dapl_ia_graceful_close(struct dapl_i if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - cur_dat_status = dapl_evd_free(ia_ptr->async_error_evd); + cur_dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; @@ -424,8 +418,6 @@ bail: */ void dapl_ia_free(struct dapl_ia *ia_ptr) { - dapl_os_assert(ia_ptr->header.magic == DAPL_MAGIC_IA); - dapl_os_assert(ia_ptr->async_error_evd == NULL); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->lmr_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rmr_list_head)); @@ -434,12 +426,8 @@ void dapl_ia_free(struct dapl_ia *ia_ptr dapl_os_assert(dapl_llist_is_empty(&ia_ptr->psp_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rsp_list_head)); - /* - * deinitialize the header - */ dapl_hca_unlink_ia(ia_ptr->hca_ptr, ia_ptr); - ia_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - /* no need to destroy ia_ptr->header.lock */ + /* no need to destroy ia_ptr->common.lock */ kfree(ia_ptr); } @@ -449,10 +437,10 @@ void dapl_ia_free(struct dapl_ia *ia_ptr */ void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry, ep_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry, ep_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -460,10 +448,10 @@ void dapl_ia_link_ep(struct dapl_ia *ia_ */ void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -471,10 +459,10 @@ void dapl_ia_unlink_ep(struct dapl_ia *i */ void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry, srq_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry, srq_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -482,10 +470,10 @@ void dapl_ia_link_srq(struct dapl_ia *ia */ void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -493,10 +481,10 @@ void dapl_ia_unlink_srq(struct dapl_ia * */ void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry, lmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry, lmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -504,10 +492,10 @@ void dapl_ia_link_lmr(struct dapl_ia *ia */ void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -515,10 +503,10 @@ void dapl_ia_unlink_lmr(struct dapl_ia * */ void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry, rmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry, rmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -526,10 +514,10 @@ void dapl_ia_link_rmr(struct dapl_ia *ia */ void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -537,10 +525,10 @@ void dapl_ia_unlink_rmr(struct dapl_ia * */ void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry, pz_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry, pz_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -548,10 +536,10 @@ void dapl_ia_link_pz(struct dapl_ia *ia_ */ void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -559,10 +547,10 @@ void dapl_ia_unlink_pz(struct dapl_ia *i */ void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry, evd_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry, evd_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -570,10 +558,10 @@ void dapl_ia_link_evd(struct dapl_ia *ia */ void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -581,10 +569,10 @@ void dapl_ia_unlink_evd(struct dapl_ia * */ void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->psp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -594,17 +582,17 @@ void dapl_ia_unlink_sp(struct dapl_ia *i { struct dapl_llist_entry **list_head; - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_PSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_PSP) { list_head = &ia_ptr->psp_list_head; } else { - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == + DAT_SP_TYPE_RSP); list_head = &ia_ptr->rsp_list_head; } - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); - dapl_llist_remove_entry(list_head, &sp_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); + dapl_llist_remove_entry(list_head, &sp_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -634,7 +622,7 @@ struct dapl_sp *dapl_ia_sp_search(struct else list_head = &ia_ptr->rsp_list_head; - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); sp_ptr = (dapl_llist_is_empty(list_head) ? NULL : dapl_llist_peek_head(list_head)); @@ -644,10 +632,10 @@ struct dapl_sp *dapl_ia_sp_search(struct break; sp_ptr = dapl_llist_next_entry(list_head, - &sp_ptr->header.ia_list_entry); + &sp_ptr->common.ia_list_entry); } - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); return sp_ptr; } @@ -670,10 +658,10 @@ struct dapl_sp *dapl_ia_sp_search(struct */ void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rsp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, Index: linux-kernel/dat-provider/dapl_cr_query.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_query.c (revision 2564) +++ linux-kernel/dat-provider/dapl_cr_query.c (working copy) @@ -34,16 +34,16 @@ #include "dapl.h" #include "dapl_openib_util.h" -u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) +u32 dapl_cr_query(struct dat_cr *cr, struct dat_cr_param *cr_param) { struct dapl_cr *cr_ptr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", - cr_handle, cr_param); + cr, cr_param); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (cr == NULL) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } @@ -53,10 +53,10 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* obtain the remote IP address */ - (void) dapl_ib_cm_remote_addr((DAT_HANDLE) cr_handle, + (void) dapl_ib_cm_remote_addr((void *) cr, &cr_ptr->remote_ia_address); /* since the arguments are easily accessible, ignore the mask */ Index: linux-kernel/dat-provider/dapl_provider.c =================================================================== --- linux-kernel/dat-provider/dapl_provider.c (revision 2564) +++ linux-kernel/dat-provider/dapl_provider.c (working copy) @@ -71,9 +71,8 @@ static struct dat_provider g_dapl_provid .ia_close_func = &dapl_ia_close, .ia_memtype_hint_func = &dapl_ia_memtype_hint, - .set_consumer_context_func = &dapl_set_consumer_context, - .get_consumer_context_func = &dapl_get_consumer_context, - .get_handle_type_func = &dapl_get_handle_type, + .set_consumer_context_func = NULL, + .get_consumer_context_func = NULL, .cr_query_func = &dapl_cr_query, .cr_accept_func = &dapl_cr_accept, Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2564) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -70,7 +70,7 @@ static void dapl_destroy_cm_id(struct da conn, conn->cm_id); ib_destroy_cm_id(conn->cm_id); if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } } @@ -172,7 +172,7 @@ static int dapl_cm_active_cb_handler(str spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -285,7 +285,7 @@ static int dapl_cm_passive_cb_handler(st spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -413,7 +413,7 @@ error: * Initiate a connection with the passive listener on another node * * Input: - * ep_handle + * ep * remote_ia_address * remote_conn_qual * private_data_size size of private data and structure @@ -428,7 +428,7 @@ error: * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data) @@ -438,7 +438,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han struct dapl_cm_id *conn; int status; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->qp == NULL) { printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); return DAT_INTERNAL_ERROR; @@ -460,9 +460,9 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han kfree(conn); return DAT_INTERNAL_ERROR; } - conn->ep->cm_handle = conn; + conn->ep->cm = conn; - ia_ptr = conn->ep->header.owner_ia; + ia_ptr = conn->ep->common.owner_ia; /* Setup QP/CM parameters */ memset(&conn->param, 0, sizeof conn->param); @@ -507,7 +507,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han * Disconnect an EP * * Input: - * ep_handle + * ep * close_flags * * Output: @@ -521,7 +521,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han */ u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags) { - struct dapl_cm_id *conn = ep_ptr->cm_handle; + struct dapl_cm_id *conn = ep_ptr->cm; int status; dapl_dbg_log(DAPL_DBG_TYPE_CM, @@ -537,7 +537,7 @@ u32 dapl_ib_disconnect(struct dapl_ep *e status = ib_send_cm_dreq(conn->cm_id, NULL, 0); if (status) printk(KERN_ERR "dapl_ib_disconnect: CM ID 0x%p " - "status %d\n", ep_ptr->cm_handle, status); + "status %d\n", ep_ptr->cm, status); } return DAT_SUCCESS; @@ -607,14 +607,14 @@ void dapl_ib_reinit_ep(struct dapl_ep *e dapl_dbg_log(DAPL_DBG_TYPE_CM, " >>> dapl_ib_reinit_ep: EP %p\n", ep_ptr); - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* * What if QP is connected? Do we need to destroy the cm_id * here as well, send a DREQ, REJ? What cm_id state are we in? * we may not care if we just destroy the cm_id - * ib_destroy_cm_id(ep_ptr->cm_handle); - * supporting this call may require setting the cm_handle to + * ib_destroy_cm_id(ep_ptr->cm); + * supporting this call may require setting the cm to * something like NULL wherever it is destroyed */ @@ -685,7 +685,7 @@ u32 dapl_ib_setup_conn_listener(struct d * Have the CM remove a connection listener. * * Input: - * ia_handle IA handle + * ia IA handle * ServiceID IB Channel Service ID * * Output: @@ -719,7 +719,7 @@ u32 dapl_ib_remove_conn_listener(struct * Perform necessary steps to reject a connection * * Input: - * cm_handle + * cm * * Output: * none @@ -730,24 +730,24 @@ u32 dapl_ib_remove_conn_listener(struct * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle) +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm) { int status; - if (cm_handle == NULL) { + if (cm == NULL) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_reject_connection: invalid cm handle\n"); return DAT_SUCCESS; } - status = ib_send_cm_rej(cm_handle->cm_id, IB_CM_REJ_CONSUMER_DEFINED, + status = ib_send_cm_rej(cm->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_reject_connection: " "ib_send_cm_rej failed: %d\n", status); return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); } - dapl_destroy_cm_id(cm_handle); + dapl_destroy_cm_id(cm); return DAT_SUCCESS; } @@ -757,8 +757,8 @@ u32 dapl_ib_reject_connection(struct dap * Perform necessary steps to accept a connection * * Input: - * cr_handle - * ep_handle + * cr + * ep * private_data_size * private_data * @@ -771,7 +771,7 @@ u32 dapl_ib_reject_connection(struct dap * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *prd_ptr) { @@ -783,9 +783,9 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN struct ib_cm_rep_param passive_params; struct dapl_cm_id *conn; - cr_ptr = (struct dapl_cr *)cr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; - ia_ptr = ep_ptr->header.owner_ia; + cr_ptr = (struct dapl_cr *)cr; + ep_ptr = (struct dapl_ep *)ep; + ia_ptr = ep_ptr->common.owner_ia; conn = cr_ptr->ib_cm_handle; /* Obtain size of private data structure & contents */ @@ -810,8 +810,8 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN } } - cr_ptr->param.local_ep_handle = ep_handle; - ep_ptr->cm_handle = conn; + cr_ptr->param.local_ep = ep; + ep_ptr->cm = conn; conn->ep = ep_ptr; memset(&passive_params, 0, sizeof passive_params); @@ -851,7 +851,7 @@ reject: /* * ib_cm_get_remote_gid */ -static int ib_cm_get_remote_gid(struct dapl_cm_id *cm_handle, u8 *remote_gid) +static int ib_cm_get_remote_gid(struct dapl_cm_id *cm, u8 *remote_gid) { return DAT_INTERNAL_ERROR; /* for now!!! */ } @@ -862,7 +862,7 @@ static int ib_cm_get_remote_gid(struct d * Obtain the remote IP address given a connection * * Input: - * cr_handle + * cr * private data structure handle (only if not using ATS) * * Output: @@ -874,7 +874,7 @@ static int ib_cm_get_remote_gid(struct d * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address) { struct dapl_ia *ia_ptr; @@ -883,7 +883,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da struct dapl_cr *cr_ptr = (struct dapl_cr *)dat_handle; - ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia_handle; + ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia; rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); if (rc < 0) Index: linux-kernel/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel/patches/alt_dat_provider_makefile (revision 2564) +++ linux-kernel/patches/alt_dat_provider_makefile (working copy) @@ -21,8 +21,6 @@ PROVIDER_MODULES := \ dapl_cr_util \ dapl_ep \ dapl_evd \ - dapl_get_consumer_context \ - dapl_get_handle_type \ dapl_hash \ dapl_hca_util \ dapl_ia_close \ @@ -39,7 +37,6 @@ PROVIDER_MODULES := \ dapl_ring_buffer_util \ dapl_rmr \ dapl_rsp \ - dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ dapl_util Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2564) +++ linux-kernel/dat/api.c (working copy) @@ -350,7 +350,7 @@ static u32 dat_strerror_minor(u32 value, } u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, + struct dat_evd **async_event_handle, struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; @@ -387,13 +387,12 @@ u32 dat_ia_openv(const char *name, int a return status; } - return ia_open_func(name, async_event_qlen, async_event_handle, - ia_handle); + return ia_open_func(name, async_event_qlen, async_event_handle, ia); } EXPORT_SYMBOL(dat_ia_openv); -u32 dat_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags flags) +u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; @@ -401,17 +400,16 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle const char *ia_name; dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "%s(ia:%x, flags:%x)\n", __func__, ia_handle, flags); + "%s(ia:%x, flags:%x)\n", __func__, ia, flags); - provider = DAT_HANDLE_TO_PROVIDER(ia_handle); + provider = DAT_HANDLE_TO_PROVIDER(ia); ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia_handle, NULL, NULL, + if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); - else if (DAT_SUCCESS != (status = provider->ia_close_func(ia_handle, - flags))) + else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "close function for IA %s failed\n", ia_name); else { Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2564) +++ linux-kernel/dat/dat.h (working copy) @@ -333,20 +333,6 @@ enum dat_return_subtype { #define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) #define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) -/* dat handle types */ -enum dat_handle_type { - DAT_HANDLE_TYPE_CR, - DAT_HANDLE_TYPE_EP, - DAT_HANDLE_TYPE_EVD, - DAT_HANDLE_TYPE_IA, - DAT_HANDLE_TYPE_LMR, - DAT_HANDLE_TYPE_PSP, - DAT_HANDLE_TYPE_PZ, - DAT_HANDLE_TYPE_RMR, - DAT_HANDLE_TYPE_RSP, - DAT_HANDLE_TYPE_SRQ -}; - #define DAT_NAME_MAX_LENGTH 256 struct dat_named_attr { @@ -385,21 +371,6 @@ enum dat_completion_flags { #define DAT_TIMEOUT_MAX ULONG_MAX -/* dat handles */ -typedef void *DAT_HANDLE; -typedef DAT_HANDLE DAT_CR_HANDLE; -typedef DAT_HANDLE DAT_EP_HANDLE; -typedef DAT_HANDLE DAT_EVD_HANDLE; -typedef DAT_HANDLE DAT_IA_HANDLE; -typedef DAT_HANDLE DAT_LMR_HANDLE; -typedef DAT_HANDLE DAT_PSP_HANDLE; -typedef DAT_HANDLE DAT_PZ_HANDLE; -typedef DAT_HANDLE DAT_RMR_HANDLE; -typedef DAT_HANDLE DAT_RSP_HANDLE; -typedef DAT_HANDLE DAT_SRQ_HANDLE; - -#define DAT_HANDLE_NULL ((DAT_HANDLE)NULL) - typedef u64 DAT_CONN_QUAL; typedef u64 DAT_PORT_QUAL; @@ -541,8 +512,8 @@ enum dat_mem_priv_flags { /* RMR Arguments */ struct dat_rmr_param { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; struct dat_lmr_triplet lmr_triplet; enum dat_mem_priv_flags mem_priv; DAT_RMR_CONTEXT rmr_context; @@ -588,17 +559,17 @@ enum dat_ep_state { }; struct dat_ep_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_ep_state ep_state; struct sockaddr *local_ia_address_ptr; DAT_PORT_QUAL local_port_qual; struct sockaddr *remote_ia_address_ptr; DAT_PORT_QUAL remote_port_qual; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EVD_HANDLE request_evd_handle; - DAT_EVD_HANDLE connect_evd_handle; - DAT_SRQ_HANDLE srq_handle; + struct dat_pz *pz; + struct dat_evd *recv_evd; + struct dat_evd *request_evd; + struct dat_evd *connect_evd; + struct dat_srq *srq; struct dat_ep_attr ep_attr; }; @@ -654,9 +625,9 @@ struct dat_srq_attr { }; struct dat_srq_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_srq_state srq_state; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; int max_recv_dtos; int max_recv_iov; int low_watermark; @@ -667,25 +638,25 @@ struct dat_srq_param { /* PZ Parameters */ struct dat_pz_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; }; /* PSP Parameters */ struct dat_psp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; }; /* RSP Parameters */ struct dat_rsp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_ep *ep; }; /* Connection Request Parameters. @@ -712,11 +683,11 @@ struct dat_cr_param { /* The local Endpoint provided by the Service Point for the requested * connection. It is the only Endpoint that can accept a Connection - * Request on this Service Point. The value DAT_HANDLE_NULL + * Request on this Service Point. The value NULL * represents that there is no associated local Endpoint for the * requested connection. */ - DAT_EP_HANDLE local_ep_handle; + struct dat_ep *local_ep; }; /************************** Events ******************************/ @@ -753,7 +724,7 @@ enum dat_dto_completion_status { /* DTO completion event data */ /* transfered_length is not defined if status is not DAT_SUCCESS */ struct dat_dto_completion_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; DAT_DTO_COOKIE user_cookie; enum dat_dto_completion_status status; u64 transfered_length; @@ -761,25 +732,20 @@ struct dat_dto_completion_event_data { /* RMR bind completion event data */ struct dat_rmr_bind_completion_event_data { - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr *rmr; DAT_RMR_COOKIE user_cookie; enum dat_dto_completion_status status; }; -typedef union dat_sp_handle { - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; -} DAT_SP_HANDLE; - /* Connection Request Arrival event data */ struct dat_cr_arrival_event_data { /* Handle to the Service Point that received the Connection Request * from the remote side. If the Service Point was Reserved, sp is - * DAT_HANDLE_NULL because the reserved Service Point is + * NULL because the reserved Service Point is * automatically destroyed upon generating this event. Can be PSP * or RSP. */ - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; /* Address of the IA on which the Connection Request arrived. */ struct sockaddr *local_ia_address_ptr; @@ -791,30 +757,30 @@ struct dat_cr_arrival_event_data { /* The Connection Request instance created by a Provider for the * arrived Connection Request. Consumers can find out private_data - * passed by a remote Consumer from cr_handle. It is up to a Consumer + * passed by a remote Consumer from cr. It is up to a Consumer * to dat_cr_accept or dat_cr_reject of the Connection Request. */ - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; }; /* Connection event data */ struct dat_connection_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; int private_data_size; void *private_data; }; /* Async Error event data */ -/* For unaffiliated asynchronous event dat_handle is ia_handle. For - * Endpoint affiliated asynchronous event dat_handle is ep_handle. For - * EVD affiliated asynchronous event dat_handle is evd_handle. For SRQ +/* For unaffiliated asynchronous event dat_handle is ia. For + * Endpoint affiliated asynchronous event dat_handle is ep. For + * EVD affiliated asynchronous event dat_handle is evd. For SRQ * affiliated asynchronous event dat_handle is srq_handle. For Memory - * affiliated asynchronous event dat_handle is either lmr_handle, - * rmr_handle or pz_handle. + * affiliated asynchronous event dat_handle is either lmr, + * rmr_handle or pz. */ struct dat_asynch_error_event_data { - DAT_HANDLE dat_handle; /* either IA, EP, EVD, SRQ, */ + void *dat_handle; /* either IA, EP, EVD, SRQ, */ /* LMR, RMR, or PZ handle */ int reason; /* object specific */ }; @@ -894,7 +860,7 @@ typedef union dat_event_data { struct dat_event { enum dat_event_number event_number; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; DAT_EVENT_DATA event_data; }; @@ -934,7 +900,7 @@ struct dat_provider_info { }; struct dat_evd_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; int evd_qlen; enum dat_upcall_policy upcall_policy; struct dat_upcall_object upcall; @@ -960,7 +926,7 @@ enum dat_mem_optimize_flags { typedef union dat_region_description { void *for_va; - DAT_LMR_HANDLE for_lmr_handle; + struct dat_lmr *for_lmr; void *for_pointer; /* For kdapl only */ void *for_array; /* For kdapl only */ u64 for_pa; /* For kdapl only */ @@ -968,11 +934,11 @@ typedef union dat_region_description { /* LMR Arguments */ struct dat_lmr_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_mem_type mem_type; DAT_REGION_DESCRIPTION region_desc; u64 length; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; enum dat_mem_priv_flags mem_priv; DAT_LMR_CONTEXT lmr_context; DAT_RMR_CONTEXT rmr_context; @@ -1083,178 +1049,176 @@ struct dat_provider_attr { * ****************************************************************/ -typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *); +typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **); -typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *, u32, u32, boolean_t); +typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **, u32, u32, boolean_t); -typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum dat_close_flags); -typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_HANDLE, DAT_EVD_HANDLE *, +typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); - -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(void *, union dat_context); -typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, enum dat_handle_type *); +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(void *, union dat_context *); -typedef u32 (*DAT_CR_QUERY_FUNC)(DAT_CR_HANDLE, struct dat_cr_param *); +typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct dat_cr_param *); -typedef u32 (*DAT_CR_ACCEPT_FUNC)(DAT_CR_HANDLE, DAT_EP_HANDLE, int, +typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct dat_ep *, int, const void*); -typedef u32 (*DAT_CR_REJECT_FUNC)(DAT_CR_HANDLE); +typedef u32 (*DAT_CR_REJECT_FUNC)(struct dat_cr *); -typedef u32 (*DAT_CR_HANDOFF_FUNC)(DAT_CR_HANDLE, DAT_CONN_QUAL); +typedef u32 (*DAT_CR_HANDOFF_FUNC)(struct dat_cr *, DAT_CONN_QUAL); -typedef u32 (*DAT_EVD_RESIZE_FUNC)(DAT_EVD_HANDLE, int); +typedef u32 (*DAT_EVD_RESIZE_FUNC)(struct dat_evd *, int); -typedef u32 (*DAT_EVD_POST_SE_FUNC)(DAT_EVD_HANDLE, const struct dat_event *); +typedef u32 (*DAT_EVD_POST_SE_FUNC)(struct dat_evd *, const struct dat_event *); -typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(DAT_EVD_HANDLE, struct dat_event *); +typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(struct dat_evd *, struct dat_event *); -typedef u32 (*DAT_EVD_FREE_FUNC)(DAT_EVD_HANDLE); +typedef u32 (*DAT_EVD_FREE_FUNC)(struct dat_evd *); -typedef u32 (*DAT_EVD_QUERY_FUNC)(DAT_EVD_HANDLE, struct dat_evd_param *); +typedef u32 (*DAT_EVD_QUERY_FUNC)(struct dat_evd *, struct dat_evd_param *); -typedef u32 (*DAT_EP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - const struct dat_ep_attr *, DAT_EP_HANDLE *); +typedef u32 (*DAT_EP_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, struct dat_evd *, + struct dat_evd *, struct dat_evd *, + const struct dat_ep_attr *, struct dat_ep **); -typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_SRQ_HANDLE, +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_evd *, struct dat_evd *, + struct dat_evd *, struct dat_srq *, const struct dat_ep_attr *, - DAT_EP_HANDLE *); + struct dat_ep **); -typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_HANDLE, struct dat_ep_param *); +typedef u32 (*DAT_EP_QUERY_FUNC)(struct dat_ep *, struct dat_ep_param *); -typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, enum dat_ep_param_mask, +typedef u32 (*DAT_EP_MODIFY_FUNC)(struct dat_ep *, enum dat_ep_param_mask, const struct dat_ep_param *); -typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, +typedef u32 (*DAT_EP_CONNECT_FUNC)(struct dat_ep *, struct sockaddr *, DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); -typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(struct dat_ep *, struct dat_ep *, unsigned long timeout, int, const void *, enum dat_qos); -typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_EP_DISCONNECT_FUNC)(struct dat_ep *, enum dat_close_flags); -typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_SEND_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RECV_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RECV_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, enum dat_ep_state *, +typedef u32 (*DAT_EP_GET_STATUS_FUNC)(struct dat_ep *, enum dat_ep_state *, boolean_t *, boolean_t *); -typedef u32 (*DAT_EP_FREE_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_FREE_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RESET_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_RESET_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(DAT_EP_HANDLE, int *, int *); +typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(struct dat_ep *, int *, int *); -typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(DAT_EP_HANDLE, int, int); +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(struct dat_ep *, int, int); -typedef u32 (*DAT_LMR_FREE_FUNC)(DAT_LMR_HANDLE); +typedef u32 (*DAT_LMR_FREE_FUNC)(struct dat_lmr *); -typedef u32 (*DAT_LMR_QUERY_FUNC)(DAT_LMR_HANDLE, struct dat_lmr_param *); +typedef u32 (*DAT_LMR_QUERY_FUNC)(struct dat_lmr *, struct dat_lmr_param *); -typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_PZ_HANDLE, DAT_RMR_HANDLE *); +typedef u32 (*DAT_RMR_CREATE_FUNC)(struct dat_pz *, struct dat_rmr * *); -typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RMR_HANDLE, struct dat_rmr_param *); +typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); -typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, - enum dat_mem_priv_flags, DAT_EP_HANDLE, +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, + enum dat_mem_priv_flags, struct dat_ep *, DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); -typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); +typedef u32 (*DAT_RMR_FREE_FUNC)(struct dat_rmr *); -typedef u32 (*DAT_PSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, - enum dat_psp_flags, DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_evd *, + enum dat_psp_flags, struct dat_sp **); -typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL *, - DAT_EVD_HANDLE, enum dat_psp_flags, - DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(struct dat_ia *, DAT_CONN_QUAL *, + struct dat_evd *, enum dat_psp_flags, + struct dat_sp **); -typedef u32 (*DAT_PSP_QUERY_FUNC)(DAT_PSP_HANDLE, struct dat_psp_param *); +typedef u32 (*DAT_PSP_QUERY_FUNC)(struct dat_sp *, struct dat_psp_param *); -typedef u32 (*DAT_PSP_FREE_FUNC)(DAT_PSP_HANDLE); +typedef u32 (*DAT_PSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_RSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, - DAT_EVD_HANDLE, DAT_RSP_HANDLE *); +typedef u32 (*DAT_RSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_ep *, + struct dat_evd *, struct dat_sp **); -typedef u32 (*DAT_RSP_QUERY_FUNC)(DAT_RSP_HANDLE, struct dat_rsp_param *); +typedef u32 (*DAT_RSP_QUERY_FUNC)(struct dat_sp *, struct dat_rsp_param *); -typedef u32 (*DAT_RSP_FREE_FUNC)(DAT_RSP_HANDLE); +typedef u32 (*DAT_RSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_PZ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE *); +typedef u32 (*DAT_PZ_CREATE_FUNC)(struct dat_ia *, struct dat_pz **); -typedef u32 (*DAT_PZ_QUERY_FUNC)(DAT_PZ_HANDLE, struct dat_pz_param *); +typedef u32 (*DAT_PZ_QUERY_FUNC)(struct dat_pz *, struct dat_pz_param *); -typedef u32 (*DAT_PZ_FREE_FUNC)(DAT_PZ_HANDLE); +typedef u32 (*DAT_PZ_FREE_FUNC)(struct dat_pz *); -typedef u32 (*DAT_SRQ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - struct dat_srq_attr *, DAT_SRQ_HANDLE *); +typedef u32 (*DAT_SRQ_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_srq_attr *, struct dat_srq **); -typedef u32 (*DAT_SRQ_SET_LW_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_SET_LW_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_FREE_FUNC)(DAT_SRQ_HANDLE); +typedef u32 (*DAT_SRQ_FREE_FUNC)(struct dat_srq *); -typedef u32 (*DAT_SRQ_QUERY_FUNC)(DAT_SRQ_HANDLE, struct dat_srq_param *); +typedef u32 (*DAT_SRQ_QUERY_FUNC)(struct dat_srq *, struct dat_srq_param *); -typedef u32 (*DAT_SRQ_RESIZE_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_RESIZE_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DAT_SRQ_HANDLE, int, +typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(struct dat_srq *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, +typedef u32 (*DAT_LMR_KCREATE_FUNC)(struct dat_ia *, enum dat_mem_type, DAT_REGION_DESCRIPTION, u64, - DAT_PZ_HANDLE, enum dat_mem_priv_flags, + struct dat_pz *, enum dat_mem_priv_flags, enum dat_mem_optimize_flags, - DAT_LMR_HANDLE *, + struct dat_lmr **, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, u64 *, u64 *); -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, u64, +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(struct dat_ia *, enum dat_mem_type, u64, enum dat_mem_optimize_flags, u64 *, u64 *); -typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, enum dat_upcall_policy, +typedef u32 (*DAT_EVD_KCREATE_FUNC) (struct dat_ia *, int, enum dat_upcall_policy, const struct dat_upcall_object *, - enum dat_evd_flags, DAT_EVD_HANDLE *); + enum dat_evd_flags, struct dat_evd **); -typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(struct dat_evd *, enum dat_upcall_policy, const struct dat_upcall_object *); @@ -1269,7 +1233,6 @@ struct dat_provider { DAT_SET_CONSUMER_CONTEXT_FUNC set_consumer_context_func; DAT_GET_CONSUMER_CONTEXT_FUNC get_consumer_context_func; - DAT_GET_HANDLE_TYPE_FUNC get_handle_type_func; DAT_CR_QUERY_FUNC cr_query_func; DAT_CR_ACCEPT_FUNC cr_accept_func; @@ -1338,6 +1301,52 @@ struct dat_provider { DAT_SRQ_SET_LW_FUNC srq_set_lw_func; }; +struct dat_ep { + struct dat_provider *provider; +}; + +struct dat_ia { + struct dat_provider *provider; +}; + +struct dat_evd { + struct dat_provider *provider; +}; + +struct dat_cno { + struct dat_provider *provider; +}; + +struct dat_srq { + struct dat_provider *provider; +}; + +struct dat_pz { + struct dat_provider *provider; +}; + +struct dat_lmr { + struct dat_provider *provider; +}; + +struct dat_rmr { + struct dat_provider *provider; +}; + +enum dat_sp_type { + DAT_SP_TYPE_RSP, + DAT_SP_TYPE_PSP +}; + +struct dat_sp { + struct dat_provider *provider; + enum dat_sp_type type; +}; + +struct dat_cr { + struct dat_provider *provider; +}; + /*************************************************************** * * FUNCTION PROTOTYPES @@ -1359,8 +1368,8 @@ extern u32 dat_registry_remove_provider( * DAT registry functions for consumers */ extern u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, - DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, + struct dat_evd **async_event_handle, + struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety); #define dat_ia_open(name, qlen, async_evd, ia) \ @@ -1368,7 +1377,7 @@ extern u32 dat_ia_openv(const char *name DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ DAT_THREADSAFE) -extern u32 dat_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); extern u32 dat_strerror(u32, const char **, const char **); @@ -1383,27 +1392,21 @@ extern u32 dat_registry_list_providers(i #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) -static inline u32 dat_set_consumer_context(DAT_HANDLE handle, +static inline u32 dat_set_consumer_context(void *handle, union dat_context context) { return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, context); } -static inline u32 dat_get_consumer_context(DAT_HANDLE handle, +static inline u32 dat_get_consumer_context(void *handle, union dat_context *context) { return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, context); } -static inline u32 dat_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type) -{ - return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); -} - -static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, +static inline u32 dat_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimize, @@ -1415,7 +1418,7 @@ static inline u32 dat_ia_memtype_hint(DA preferred_alignment); } -static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE *async_evd, +static inline u32 dat_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -1423,49 +1426,49 @@ static inline u32 dat_ia_query(DAT_IA_HA ia_query_func, ia, async_evd, ia_attr, provider_attr); } -static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, +static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data) { return DAT_CALL_PROVIDER_FUNC( cr_accept_func, cr, ep, private_data_size, private_data); } -static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) +static inline u32 dat_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff) { return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); } -static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) +static inline u32 dat_cr_query(struct dat_cr *cr, struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } -static inline u32 dat_cr_reject(DAT_CR_HANDLE cr) +static inline u32 dat_cr_reject(struct dat_cr *cr) { return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); } -static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) +static inline u32 dat_evd_dequeue(struct dat_evd *evd, struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } -static inline u32 dat_evd_free(DAT_EVD_HANDLE evd) +static inline u32 dat_evd_free(struct dat_evd *evd) { return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); } -static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, +static inline u32 dat_evd_kcreate(struct dat_ia *ia, int qlen, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall, enum dat_evd_flags flags, - DAT_EVD_HANDLE * evd) + struct dat_evd ** evd) { return DAT_CALL_PROVIDER_FUNC(evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); } -static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall) { @@ -1473,23 +1476,23 @@ static inline u32 dat_evd_modify_upcall( upcall); } -static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_post_se(struct dat_evd *evd, const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } -static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) +static inline u32 dat_evd_query(struct dat_evd *evd, struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } -static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, int qlen) +static inline u32 dat_evd_resize(struct dat_evd *evd, int qlen) { return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, +static inline u32 dat_ep_connect(struct dat_ep *ep, struct sockaddr *ia_addr, DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, @@ -1500,36 +1503,36 @@ static inline u32 dat_ep_connect(DAT_EP_ qos, flags); } -static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, +static inline u32 dat_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, struct dat_evd *out_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_func, ia, pz, in_evd, out_evd, connect_evd, attr, ep); } -static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, - DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, - DAT_SRQ_HANDLE srq, +static inline u32 dat_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, + struct dat_evd *out_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_with_srq_func, ia, pz, in_evd, out_evd, connect_evd, srq, attr, ep); } -static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_disconnect(struct dat_ep *ep, enum dat_close_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } -static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, +static inline u32 dat_ep_dup_connect(struct dat_ep *ep, struct dat_ep *dup_ep, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) @@ -1538,25 +1541,25 @@ static inline u32 dat_ep_dup_connect(DAT private_data_size, private_data, qos); } -static inline u32 dat_ep_free(DAT_EP_HANDLE ep) +static inline u32 dat_ep_free(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, enum dat_ep_state *state, +static inline u32 dat_ep_get_status(struct dat_ep *ep, enum dat_ep_state *state, boolean_t *recv_idle, boolean_t *req_idle) { return DAT_CALL_PROVIDER_FUNC(ep_get_status_func, ep, state, recv_idle, req_idle); } -static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, enum dat_ep_param_mask mask, +static inline u32 dat_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } -static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_read(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1566,7 +1569,7 @@ static inline u32 dat_ep_post_rdma_read( local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_write(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1576,7 +1579,7 @@ static inline u32 dat_ep_post_rdma_write local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_recv(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, enum dat_completion_flags flags) @@ -1585,7 +1588,7 @@ static inline u32 dat_ep_post_recv(DAT_E cookie, flags); } -static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_send(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1595,24 +1598,24 @@ static inline u32 dat_ep_post_send(DAT_E ep_post_send_func, ep, size, local_iov, cookie, flags); } -static inline u32 dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) +static inline u32 dat_ep_query(struct dat_ep *ep, struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } -static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, int *bufs_alloc, +static inline u32 dat_ep_recv_query(struct dat_ep *ep, int *bufs_alloc, int *bufs_avail) { return DAT_CALL_PROVIDER_FUNC(ep_recv_query_func, ep, bufs_alloc, bufs_avail); } -static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) +static inline u32 dat_ep_reset(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); } -static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, +static inline u32 dat_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -1620,17 +1623,17 @@ static inline u32 dat_ep_set_watermark(D soft_high_watermark, hard_high_watermark); } -static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) +static inline u32 dat_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } -static inline u32 dat_lmr_free(DAT_LMR_HANDLE lmr) +static inline u32 dat_lmr_free(struct dat_lmr *lmr) { return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); } -static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1638,7 +1641,7 @@ static inline u32 dat_lmr_sync_rdma_read num_iovs); } -static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1646,20 +1649,20 @@ static inline u32 dat_lmr_sync_rdma_writ num_iovs); } -static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE *rmr) +static inline u32 dat_rmr_create(struct dat_pz *pz, struct dat_rmr * *rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline u32 dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) +static inline u32 dat_rmr_query(struct dat_rmr * rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, +static inline u32 dat_rmr_bind(struct dat_rmr * rmr, const struct dat_lmr_triplet *iov, enum dat_mem_priv_flags mem_flags, - DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, + struct dat_ep *ep, DAT_RMR_COOKIE cookie, enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) { @@ -1667,84 +1670,84 @@ static inline u32 dat_rmr_bind(DAT_RMR_H cookie, comp_flags, context); } -static inline u32 dat_rmr_free(DAT_RMR_HANDLE rmr) +static inline u32 dat_rmr_free(struct dat_rmr * rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) +static inline u32 dat_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd, +static inline u32 dat_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_any_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) +static inline u32 dat_psp_query(struct dat_sp *psp, struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } -static inline u32 dat_psp_free(DAT_PSP_HANDLE psp) +static inline u32 dat_psp_free(struct dat_sp *psp) { return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); } -static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, - DAT_RSP_HANDLE *rsp) +static inline u32 dat_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_create_func, ia, conn_qual, ep, evd, rsp); } -static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) +static inline u32 dat_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } -static inline u32 dat_rsp_free(DAT_RSP_HANDLE rsp) +static inline u32 dat_rsp_free(struct dat_sp *rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); } -static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE *pz) +static inline u32 dat_pz_create(struct dat_ia *ia, struct dat_pz **pz) { return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) +static inline u32 dat_pz_query(struct dat_pz *pz, struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } -static inline u32 dat_pz_free(DAT_PZ_HANDLE pz) +static inline u32 dat_pz_free(struct dat_pz *pz) { return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); } -static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, +static inline u32 dat_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *attr, - DAT_SRQ_HANDLE *srq) + struct dat_srq **srq) { return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); } -static inline u32 dat_srq_free(DAT_SRQ_HANDLE srq) +static inline u32 dat_srq_free(struct dat_srq *srq) { return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); } -static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, int num_iovs, +static inline u32 dat_srq_post_recv(struct dat_srq *srq, int num_iovs, struct dat_lmr_triplet *iovs, DAT_DTO_COOKIE cookie) { @@ -1752,34 +1755,34 @@ static inline u32 dat_srq_post_recv(DAT_ cookie); } -static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) +static inline u32 dat_srq_query(struct dat_srq *srq, struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } -static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, int max_recv_dtos) +static inline u32 dat_srq_resize(struct dat_srq *srq, int max_recv_dtos) { return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); } -static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, int low_watermark) +static inline u32 dat_srq_set_lw(struct dat_srq *srq, int low_watermark) { return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, enum dat_mem_type type, +static inline u32 dat_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type type, DAT_REGION_DESCRIPTION region, u64 len, - DAT_PZ_HANDLE pz, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { return DAT_CALL_PROVIDER_FUNC(lmr_kcreate_func, ia, type, region, len, - pz, privileges, optimization, lmr_handle, + pz, privileges, optimization, lmr, lmr_context, rmr_context, registered_length, registered_address); } From roland at topspin.com Mon Jun 6 12:36:38 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 12:36:38 -0700 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: (Kevin Reilly's message of "Mon, 6 Jun 2005 14:21:45 -0400") References: Message-ID: <52psuzl1kp.fsf@topspin.com> I just checked in support for a new ibv_query_device() verb, which will allow you to query (among many other things) the number of ports for a device. examples/devinfo.c in libibverbs should show how to use the new function. - R. From roland at topspin.com Mon Jun 6 13:50:52 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 13:50:52 -0700 Subject: [openib-general] Re: user space verbs examples In-Reply-To: <1118090378.4493.28.camel@hal.voltaire.com> (Hal Rosenstock's message of "06 Jun 2005 16:39:38 -0400") References: <1117667806.4490.160.camel@hal.voltaire.com> <52fyw14pp0.fsf@topspin.com> <1118090378.4493.28.camel@hal.voltaire.com> Message-ID: <52vf4rjjkj.fsf@topspin.com> >> If so, then send the output of 'strace -ewrite=all >> -etrace=write ibv_asyncwatch' Looks like libibverbs can't open /dev/infiniband/uverbs0. Does that file exist and if so are the permissions such that your processes can open it? - R. From roland at topspin.com Mon Jun 6 13:44:57 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 13:44:57 -0700 Subject: [openib-general] Re: [PATCH] user_mad: support retries in ib_umad_write In-Reply-To: <1118068362.1622.304.camel@hal.voltaire.com> (Hal Rosenstock's message of "06 Jun 2005 10:32:43 -0400") References: <1118068362.1622.304.camel@hal.voltaire.com> Message-ID: <523brvkyeu.fsf@topspin.com> This looks good to commit to me. Will there need to be another ABI bump to support RMPP receives in userspace? - R. From roland at topspin.com Mon Jun 6 12:28:50 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 12:28:50 -0700 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: (Shirley Ma's message of "Mon, 6 Jun 2005 12:20:34 -0700") References: Message-ID: <52zmu3l1xp.fsf@topspin.com> Shirley> A REDME file for TODO list would be helpful. OK, I've heard a few requests for this lately. I'll start on a README for libibverbs next... - R. From roland at topspin.com Mon Jun 6 12:38:24 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 12:38:24 -0700 Subject: [openib-general] error messages for debugging In-Reply-To: <20050606160222.GA3849@cse.ohio-state.edu> (Sayantan Sur's message of "Mon, 6 Jun 2005 12:02:23 -0400") References: <20050606022357.GA28138@cse.ohio-state.edu> <20050606025235.GB28138@cse.ohio-state.edu> <52d5qzpkjm.fsf@topspin.com> <20050606160222.GA3849@cse.ohio-state.edu> Message-ID: <52k6l7l1hr.fsf@topspin.com> Sayantan> I seem to have gotten around this issue. When I was Sayantan> creating the QP, I didn't specify any `max_inline_data' Sayantan> (i.e. left attr.qp.max_inline_data completely Sayantan> untouched). After I specify the `max_inline_data' and Sayantan> use IBV_SEND_INLINE in the `send_flags', it works. I'm a little confused as to what happened. I guess you left max_inline_data uninitialized and didn't look at the value returned by ibv_create_qp(), and then tried to post a too-large work request with inline data? - R. From roland at topspin.com Mon Jun 6 12:31:08 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 06 Jun 2005 12:31:08 -0700 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: (Kevin Reilly's message of "Mon, 6 Jun 2005 14:21:45 -0400") References: Message-ID: <52vf4rl1tv.fsf@topspin.com> Kevin> Roland, Do you know other things that are missing that you Kevin> can rattle off the top of your head? Else we keep looking Kevin> for what we need then ask questions if we don't find them. There are a couple of fairly large features that have not been implemented at all, even in the kernel. For example, shared receive queues (SRQ) and memory windows. Other than that I think libibverbs support is pretty complete right now. Of course the best thing is for application developers to ask for the features they need. That way I don't waste time implementing something that happens to be in the spec but no one cares about. - R. From mst at mellanox.co.il Mon Jun 6 23:44:31 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 7 Jun 2005 09:44:31 +0300 Subject: [openib-general] Re: What is the correct method of determining the number of ports In-Reply-To: <1118091818.4493.55.camel@hal.voltaire.com> References: <1118088017.4493.2.camel@hal.voltaire.com> <20050606204652.GB30710@mellanox.co.il> <1118091818.4493.55.camel@hal.voltaire.com> Message-ID: <20050607064431.GB25990@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: Re: What is the correct method of determining?the?number of ports > > On Mon, 2005-06-06 at 16:46, Michael S. Tsirkin wrote: > > Quoting r. Hal Rosenstock : > > > Subject: Re: What is the correct method of determining?the?number of ports > > > > > > On Mon, 2005-06-06 at 01:07, Shirley Ma wrote: > > > > I was thinking to use ibv_get_devices() to get the list of device > > > > names, then use umad_get_ca() to get ca info. (it actually gets the > > > > info from directory /sys/class/infiniband.) Which might not what you > > > > want. > > > > > > umad was done long before uverbs was available and using > > > /sys/class/infiniband was the only way to obtain the needed information > > > in userspace. If the community thinks uverbs should be used to obtain > > > this information, this can be converted over. Patches accepted :-) > > > > > > -- Hal > > > > Off the top of my head, the right approach seems to be to use > > /dev/infiniband/umad%d for umad. > > Right? > > It does that... > > The info it gets from /sys/class/infiniband/ pertains to a variety of > driver (verbs) queries. > > -- Hal > I wander why does umad need any verbs queries? -- MST From halr at voltaire.com Tue Jun 7 05:05:05 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 08:05:05 -0400 Subject: [openib-general] Re: [PATCH] user_mad: support retries in ib_umad_write In-Reply-To: <523brvkyeu.fsf@topspin.com> References: <1118068362.1622.304.camel@hal.voltaire.com> <523brvkyeu.fsf@topspin.com> Message-ID: <1118145904.4477.3.camel@localhost.localdomain> On Mon, 2005-06-06 at 16:44, Roland Dreier wrote: > This looks good to commit to me. Will there need to be another ABI > bump to support RMPP receives in userspace? I'm hoping not: There is a outstanding question about read (length returned when supplied buffer is too small) and failing doing it that way, there would be an additional ioctl but it would be backwards compatible (use of this ioctl from userspace would result in an error being returned). -- Hal From halr at voltaire.com Tue Jun 7 05:39:26 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 08:39:26 -0400 Subject: [openib-general] Re: user space verbs examples In-Reply-To: <52vf4rjjkj.fsf@topspin.com> References: <1117667806.4490.160.camel@hal.voltaire.com> <52fyw14pp0.fsf@topspin.com> <1118090378.4493.28.camel@hal.voltaire.com> <52vf4rjjkj.fsf@topspin.com> Message-ID: <1118147965.4477.20.camel@localhost.localdomain> On Mon, 2005-06-06 at 16:50, Roland Dreier wrote: > >> If so, then send the output of 'strace -ewrite=all > >> -etrace=write ibv_asyncwatch' > > Looks like libibverbs can't open /dev/infiniband/uverbs0. Does that > file exist and if so are the permissions such that your processes can > open it? What's supposed to create /dev/infiniband ? On the machine where this works, /dev has: 4 d-- 3 root 4096 Jun 7 08:17 infiniband On the machine which ddoesn't have this. (mthca is started and running). -- Hal From surs at cse.ohio-state.edu Tue Jun 7 06:58:25 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Tue, 7 Jun 2005 09:58:25 -0400 Subject: [openib-general] error messages for debugging In-Reply-To: <52k6l7l1hr.fsf@topspin.com> References: <20050606022357.GA28138@cse.ohio-state.edu> <20050606025235.GB28138@cse.ohio-state.edu> <52d5qzpkjm.fsf@topspin.com> <20050606160222.GA3849@cse.ohio-state.edu> <52k6l7l1hr.fsf@topspin.com> Message-ID: <20050607135824.GA15582@cse.ohio-state.edu> * On Jun,6 Roland Dreier wrote : > Sayantan> I seem to have gotten around this issue. When I was > Sayantan> creating the QP, I didn't specify any `max_inline_data' > Sayantan> (i.e. left attr.qp.max_inline_data completely > Sayantan> untouched). After I specify the `max_inline_data' and > Sayantan> use IBV_SEND_INLINE in the `send_flags', it works. > > I'm a little confused as to what happened. I guess you left > max_inline_data uninitialized and didn't look at the value returned by > ibv_create_qp(), and then tried to post a too-large work request with > inline data? I also guess that's what happened. If I hit this error again, I will post a full dump of the condition. > > - R. -- http://www.cse.ohio-state.edu/~surs From krause at cup.hp.com Tue Jun 7 07:10:44 2005 From: krause at cup.hp.com (Michael Krause) Date: Tue, 07 Jun 2005 07:10:44 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <42A0ABC3.30203@ichips.intel.com> References: <001401c56083$b27dfa70$9b5aa8c0@infiniconsys.com> <42A0ABC3.30203@ichips.intel.com> Message-ID: <6.2.0.14.2.20050607071004.0284e938@esmail.cup.hp.com> At 12:13 PM 6/3/2005, Sean Hefty wrote: >Fab Tillier wrote: >>Ok, so this question is from a noob, but here goes anyway. Why can't IPoIB >>advertise a larger MTU than the UD MTU, and then just fragment large IP >>packets up if they need to go over the IB UD transport? Is there any reason >>this couldn't work? If it does, it allows IPoIB to expose a single MTU to >>the OS, and take care of the rest under the covers. >>Just a thought. > >I don't remember seeing a response to this. Something like this could >work. I guess one disadvantage is that it can be less efficient if you >lose a lot of packets. TCP would resend an entire MTU (as seen by TCP) of >data if only a single IB packet were lost. Why not just use the IETF draft for RC / UC based IP over IB and not worry about creating something new? Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Tue Jun 7 08:36:07 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 07 Jun 2005 08:36:07 -0700 Subject: [openib-general] Re: user space verbs examples In-Reply-To: <1118147965.4477.20.camel@localhost.localdomain> (Hal Rosenstock's message of "07 Jun 2005 08:39:26 -0400") References: <1117667806.4490.160.camel@hal.voltaire.com> <52fyw14pp0.fsf@topspin.com> <1118090378.4493.28.camel@hal.voltaire.com> <52vf4rjjkj.fsf@topspin.com> <1118147965.4477.20.camel@localhost.localdomain> Message-ID: <52ekbei3h4.fsf@topspin.com> Hal> What's supposed to create /dev/infiniband ? Either the administrator (on systems with static /dev) or udev (on systems with dynamic /dev). You'll need to add a udev rule like KERNEL="uverbs*", NAME="infiniband/%k", MODE="0666" to get the uverbs files created in the correct place. - R. From itamar at mellanox.co.il Tue Jun 7 08:56:47 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 7 Jun 2005 18:56:47 +0300 Subject: [openib-general] RE: [ib_at] oops in ib_at when running kdapltest Message-ID: <91DB792C7985D411BEC300B40080D29CC35C43@mtvex01.mtv.mtl.com> > > Can you turn on the DEBUG printing in at.c by changing the following > lines: > #define DEBUG(fmt, ...) while (0) {} > //#define DEBUG(fmt, arg ...) printk("ib_at: %s: " fmt > "\n", __FUNCTION__, ## arg); > > to: > //#define DEBUG(fmt, ...) while (0) {} > #define DEBUG(fmt, arg ...) printk("ib_at: %s: " fmt "\n", > __FUNCTION__, ## arg); > > rebuild ib_at, and send the output of /var/log/messages. Thanks. > > -- Hal > attached is the output of /var/log/messages that contain ib_at oops with ib_at debug trun on. svn rev 2564 (ib + kdapl + at from users/jlentini) test is : ./kdapltest -T T -s 11.1.4.43 -D mthca0a -d -t 2 -w 8 -i 20 client SR server SR i run the test 4-5 times before the oops occur (In order to get the oops your kernel must be compiled with CONFIG_DEBUG_PAGEALLOC) Itamar -------------- next part -------------- A non-text attachment was scrubbed... Name: log Type: application/octet-stream Size: 115931 bytes Desc: not available URL: From roland at topspin.com Tue Jun 7 08:37:30 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 07 Jun 2005 08:37:30 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <6.2.0.14.2.20050607071004.0284e938@esmail.cup.hp.com> (Michael Krause's message of "Tue, 07 Jun 2005 07:10:44 -0700") References: <001401c56083$b27dfa70$9b5aa8c0@infiniconsys.com> <42A0ABC3.30203@ichips.intel.com> <6.2.0.14.2.20050607071004.0284e938@esmail.cup.hp.com> Message-ID: <52acm2i3et.fsf@topspin.com> Michael> Why not just use the IETF draft for RC / UC based IP over Michael> IB and not worry about creating something new? I think we've come full circle. The original post was a suggestion on how to handle the fact the the connected-mode IPoIB draft requires a network stack to deal with different MTUs for different destinations on the same logical link. - R. From surs at cse.ohio-state.edu Tue Jun 7 08:43:14 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Tue, 7 Jun 2005 11:43:14 -0400 Subject: [openib-general] email archives for March 2004 Message-ID: <20050607154313.GA16326@cse.ohio-state.edu> Hi, I am trying to access the archives for March 2004 ... more specifically this message link: www.openib.org/pipermail/openib-general/2004-March/001513.html However, the archives on this webpage: http://openib.org/pipermail/openib-general/ date back only to July 2004. Is there any way I can access that message link? Thanks, Sayantan. -- http://www.cse.ohio-state.edu/~surs From itamar at mellanox.co.il Tue Jun 7 09:13:53 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 7 Jun 2005 19:13:53 +0300 Subject: [openib-general] [PATCH] initial CM module Message-ID: <91DB792C7985D411BEC300B40080D29CC35C45@mtvex01.mtv.mtl.com> This is a very old mail Thread ,but only now i had time to work on it in kdapl. we have discussed the feature that if the call to ib_cm_listen() comes with service_id == 0 that mean the consumer requste that the cm module will assign a service id. Sean, I dont see that you have added it to the .h file and not implemented it in the .c are you going to add it ? i dont need it soon but at least if you are going to support it can you update the .h file and i will prepare the kdapl file acording the .h file ? Itamar > -----Original Message----- > From: Sean Hefty [mailto:sean.hefty at intel.com] > Sent: Tuesday, December 21, 2004 7:26 PM > To: 'Roland Dreier' > Cc: 'Itamar Rabenstein'; Sean Hefty; openib-general at openib.org > Subject: RE: [openib-general] [PATCH] initial CM module > > > >Yes, some mechanism of allocating unique "local OS administered" > >service IDs (see section A.3.2.3.3 of the spec) is required. > > Thanks, that's what I was looking for in the spec, but > couldn't find it. > > >Or did I misunderstand and you were just worried about possible > >service ID collisions? In that case there is no problem, all service > >IDs starting with 0x02 are reserved for this "local OS" allocation > >method and may be freely assigned by the CM. > > I was worried about possible service ID collisions. I will > update the API > to allow this, with the plan to use the existing listen call. > > - Sean > From ftillier at silverstorm.com Tue Jun 7 09:28:34 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Tue, 7 Jun 2005 09:28:34 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <52acm2i3et.fsf@topspin.com> Message-ID: <000101c56b7d$f5940610$9c5aa8c0@infiniconsys.com> > From: Roland Dreier [mailto:roland at topspin.com] > Sent: Tuesday, June 07, 2005 8:38 AM > > Michael> Why not just use the IETF draft for RC / UC based IP over > Michael> IB and not worry about creating something new? > > I think we've come full circle. The original post was a suggestion on > how to handle the fact the the connected-mode IPoIB draft requires a > network stack to deal with different MTUs for different destinations > on the same logical link. That's right - by implementing IP segmentation in the IPoIB driver when going over UD, the driver could expose a single MTU to the network stack, thereby removing all the issues related to having per-endpoint MTUs. Keeping a 2K MTU for RC mode doesn't really take advantage of IB's RC capabilities. I'd probably target 64K as the MTU. - Fab From mshefty at ichips.intel.com Tue Jun 7 09:31:15 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 07 Jun 2005 09:31:15 -0700 Subject: [openib-general] [PATCH] initial CM module In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C45@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C45@mtvex01.mtv.mtl.com> Message-ID: <42A5CBD3.6020106@ichips.intel.com> Itamar Rabenstein wrote: > This is a very old mail Thread ,but only now i had time to work on it in > kdapl. > > we have discussed the feature that if the call to ib_cm_listen() > comes with service_id == 0 that mean the consumer requste that the cm module > will assign a service id. > > Sean, > I dont see that you have added it to the .h file and not implemented it in > the .c > > are you going to add it ? > > i dont need it soon but at least if you are going to support it > can you update the .h file and i will prepare the kdapl file acording the .h > file ? I can add in this support, but probably not until the end of the month. I will look at at least updating the header file immediately though. (It sounds like it's only a documentation change, but I'll see what make the most sense to do.) - Sean From itamar at mellanox.co.il Tue Jun 7 10:08:27 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 7 Jun 2005 20:08:27 +0300 Subject: [openib-general] [PATCH] initial CM module Message-ID: <91DB792C7985D411BEC300B40080D29CC35C46@mtvex01.mtv.mtl.com> end of the month is good for me. Thanks -Itamar > -----Original Message----- > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > Sent: Tuesday, June 07, 2005 7:31 PM > To: Itamar Rabenstein > Cc: Sean Hefty; openib-general at openib.org > Subject: Re: [openib-general] [PATCH] initial CM module > > > Itamar Rabenstein wrote: > > This is a very old mail Thread ,but only now i had time to > work on it in > > kdapl. > > > > we have discussed the feature that if the call to ib_cm_listen() > > comes with service_id == 0 that mean the consumer requste > that the cm module > > will assign a service id. > > > > Sean, > > I dont see that you have added it to the .h file and not > implemented it in > > the .c > > > > are you going to add it ? > > > > i dont need it soon but at least if you are going to support it > > can you update the .h file and i will prepare the kdapl > file acording the .h > > file ? > > I can add in this support, but probably not until the end of > the month. I > will look at at least updating the header file immediately > though. (It > sounds like it's only a documentation change, but I'll see > what make the > most sense to do.) > > - Sean > From sean.hefty at intel.com Tue Jun 7 10:50:14 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 7 Jun 2005 10:50:14 -0700 Subject: [openib-general] [PATCH] [CM] add OS administered service IDs Message-ID: The following patch adds support for OS administered service IDs. Signed-off-by: Sean Hefty Index: include/ib_cm.h =================================================================== --- include/ib_cm.h (revision 2563) +++ include/ib_cm.h (working copy) @@ -311,16 +311,20 @@ struct ib_cm_id *ib_create_cm_id(ib_cm_h */ void ib_destroy_cm_id(struct ib_cm_id *cm_id); +#define IB_CM_ASSIGN_SERVICE_ID __constant_cpu_to_be64(0x0200000000000000ULL) + /** * ib_cm_listen - Initiates listening on the specified service ID for * connection and service ID resolution requests. * @cm_id: Connection identifier associated with the listen request. * @service_id: Service identifier matched against incoming connection * and service ID resolution requests. The service ID should be specified - * network-byte order. + * network-byte order. If set to IB_CM_ASSIGN_SERVICE_ID, the CM will + * assign a service ID to the caller. * @service_mask: Mask applied to service ID used to listen across a * range of service IDs. If set to 0, the service ID is matched - * exactly. + * exactly. This parameter is ignored if %service_id is set to + * IB_CM_ASSIGN_SERVICE_ID. */ int ib_cm_listen(struct ib_cm_id *cm_id, u64 service_id, Index: core/cm.c =================================================================== --- core/cm.c (revision 2563) +++ core/cm.c (working copy) @@ -64,6 +64,7 @@ static struct ib_cm { struct list_head device_list; rwlock_t device_lock; struct rb_root listen_service_table; + u64 listen_service_id; /* struct rb_root peer_service_table; todo: fix peer to peer */ struct rb_root remote_qp_table; struct rb_root remote_id_table; @@ -718,14 +719,23 @@ int ib_cm_listen(struct ib_cm_id *cm_id, unsigned long flags; int ret = 0; + if ((service_id & IB_CM_ASSIGN_SERVICE_ID) == IB_CM_ASSIGN_SERVICE_ID && + (service_id != IB_CM_ASSIGN_SERVICE_ID)) + return -EINVAL; + cm_id_priv = container_of(cm_id, struct cm_id_private, id); BUG_ON(cm_id->state != IB_CM_IDLE); cm_id->state = IB_CM_LISTEN; - cm_id->service_id = service_id; - cm_id->service_mask = service_mask ? service_mask : ~0ULL; spin_lock_irqsave(&cm.lock, flags); + if (service_id == IB_CM_ASSIGN_SERVICE_ID) { + cm_id->service_id = __cpu_to_be64(cm.listen_service_id++); + cm_id->service_mask = ~0ULL; + } else { + cm_id->service_id = service_id; + cm_id->service_mask = service_mask ? service_mask : ~0ULL; + } cur_cm_id_priv = cm_insert_listen(cm_id_priv); spin_unlock_irqrestore(&cm.lock, flags); @@ -3251,6 +3261,7 @@ static int __init ib_cm_init(void) rwlock_init(&cm.device_lock); spin_lock_init(&cm.lock); cm.listen_service_table = RB_ROOT; + cm.listen_service_id = __constant_be64_to_cpu(IB_CM_ASSIGN_SERVICE_ID); cm.remote_id_table = RB_ROOT; cm.remote_qp_table = RB_ROOT; cm.remote_sidr_table = RB_ROOT; From ftillier at silverstorm.com Tue Jun 7 10:58:23 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Tue, 7 Jun 2005 10:58:23 -0700 Subject: [openib-general] [PATCH] [CM] add OS administered service IDs In-Reply-To: Message-ID: <000201c56b8a$804806b0$9c5aa8c0@infiniconsys.com> > From: Sean Hefty [mailto:sean.hefty at intel.com] > Sent: Tuesday, June 07, 2005 10:50 AM > > Index: core/cm.c > =================================================================== > --- core/cm.c (revision 2563) > +++ core/cm.c (working copy) > @@ -64,6 +64,7 @@ static struct ib_cm { > struct list_head device_list; > rwlock_t device_lock; > struct rb_root listen_service_table; > + u64 listen_service_id; > /* struct rb_root peer_service_table; todo: fix peer to peer */ > struct rb_root remote_qp_table; > struct rb_root remote_id_table; > @@ -718,14 +719,23 @@ int ib_cm_listen(struct ib_cm_id *cm_id, > unsigned long flags; > int ret = 0; > > + if ((service_id & IB_CM_ASSIGN_SERVICE_ID) == IB_CM_ASSIGN_SERVICE_ID && > + (service_id != IB_CM_ASSIGN_SERVICE_ID)) > + return -EINVAL; > + This check only checks that the 2nd bit in the MSB of the SID is set. You need to check that the first byte is 0x02, which means you need a mask. Something like: #define IB_CM_ASSIGN_SID_MASK __constant_cpu_to_be64(0xFF00000000000000ULL) if ((service_id & IB_CM_ASSIGN_SID_MASK) == IB_CM_ASSIGN_SERVICE_ID && (service_id != IB_CM_ASSIGN_SERVICE_ID)) > cm_id_priv = container_of(cm_id, struct cm_id_private, id); > BUG_ON(cm_id->state != IB_CM_IDLE); > > cm_id->state = IB_CM_LISTEN; > - cm_id->service_id = service_id; > - cm_id->service_mask = service_mask ? service_mask : ~0ULL; > > spin_lock_irqsave(&cm.lock, flags); > + if (service_id == IB_CM_ASSIGN_SERVICE_ID) { > + cm_id->service_id = __cpu_to_be64(cm.listen_service_id++); > + cm_id->service_mask = ~0ULL; > + } else { > + cm_id->service_id = service_id; > + cm_id->service_mask = service_mask ? service_mask : ~0ULL; > + } Should there be a check here for potential duplication? I realize that the SID is 64-bits, so it would take a very long time to wrap. Also, just for good measure, you should prevent cm.listen_service_id from exceeding 0x00FFFFFFFFFFFFF so that the upper byte is always 0x02 as required. An alternative is to just use the cm_id as the service_id: cm_id->service_id = (__cpu_to_be64(cm_id) & IB_CM_ASSING_SID_MASK) | IB_CM_ASSIGN_SERVICE_ID; - Fab From mshefty at ichips.intel.com Tue Jun 7 11:08:48 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 07 Jun 2005 11:08:48 -0700 Subject: [openib-general] [PATCH] [CM] add OS administered service IDs In-Reply-To: <000201c56b8a$804806b0$9c5aa8c0@infiniconsys.com> References: <000201c56b8a$804806b0$9c5aa8c0@infiniconsys.com> Message-ID: <42A5E2B0.2040407@ichips.intel.com> Fab Tillier wrote: >>+ if ((service_id & IB_CM_ASSIGN_SERVICE_ID) == IB_CM_ASSIGN_SERVICE_ID && >>+ (service_id != IB_CM_ASSIGN_SERVICE_ID)) >>+ return -EINVAL; >>+ > > This check only checks that the 2nd bit in the MSB of the SID is set. You need > to check that the first byte is 0x02, which means you need a mask. Something > like: > > #define IB_CM_ASSIGN_SID_MASK __constant_cpu_to_be64(0xFF00000000000000ULL) > > if ((service_id & IB_CM_ASSIGN_SID_MASK) == IB_CM_ASSIGN_SERVICE_ID && > (service_id != IB_CM_ASSIGN_SERVICE_ID)) Good catch - I'll fix this. >>+ if (service_id == IB_CM_ASSIGN_SERVICE_ID) { >>+ cm_id->service_id = __cpu_to_be64(cm.listen_service_id++); >>+ cm_id->service_mask = ~0ULL; >>+ } else { >>+ cm_id->service_id = service_id; >>+ cm_id->service_mask = service_mask ? service_mask : ~0ULL; >>+ } > > > Should there be a check here for potential duplication? I realize that the SID > is 64-bits, so it would take a very long time to wrap. Also, just for good > measure, you should prevent cm.listen_service_id from exceeding > 0x00FFFFFFFFFFFFF so that the upper byte is always 0x02 as required. You'd need to call listen something like 1,000,000 times a second for over 2000 years before you'd even exhaust the OS administered IDs. - Sean From sean.hefty at intel.com Tue Jun 7 11:24:21 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 7 Jun 2005 11:24:21 -0700 Subject: [openib-general] [PATCH] [CM] add OS administered service IDs In-Reply-To: <000201c56b8a$804806b0$9c5aa8c0@infiniconsys.com> Message-ID: >> + if ((service_id & IB_CM_ASSIGN_SERVICE_ID) == IB_CM_ASSIGN_SERVICE_ID && >> + (service_id != IB_CM_ASSIGN_SERVICE_ID)) >> + return -EINVAL; >> + > >This check only checks that the 2nd bit in the MSB of the SID is set. You need >to check that the first byte is 0x02, which means you need a mask. Something >like: > >#define IB_CM_ASSIGN_SID_MASK __constant_cpu_to_be64(0xFF00000000000000ULL) > >if ((service_id & IB_CM_ASSIGN_SID_MASK) == IB_CM_ASSIGN_SERVICE_ID && > (service_id != IB_CM_ASSIGN_SERVICE_ID)) Here's a patch to fix this: Signed-off-by: Sean Hefty Index: include/ib_cm.h =================================================================== --- include/ib_cm.h (revision 2567) +++ include/ib_cm.h (working copy) @@ -311,7 +311,8 @@ struct ib_cm_id *ib_create_cm_id(ib_cm_h */ void ib_destroy_cm_id(struct ib_cm_id *cm_id); -#define IB_CM_ASSIGN_SERVICE_ID __constant_cpu_to_be64(0x0200000000000000ULL) +#define IB_SERVICE_ID_AGN_MASK __constant_cpu_to_be64(0xFF00000000000000ULL) +#define IB_CM_ASSIGN_SERVICE_ID __constant_cpu_to_be64(0x0200000000000000ULL) /** * ib_cm_listen - Initiates listening on the specified service ID for Index: core/cm.c =================================================================== --- core/cm.c (revision 2567) +++ core/cm.c (working copy) @@ -719,7 +719,7 @@ int ib_cm_listen(struct ib_cm_id *cm_id, unsigned long flags; int ret = 0; - if ((service_id & IB_CM_ASSIGN_SERVICE_ID) == IB_CM_ASSIGN_SERVICE_ID && + if ((service_id & IB_SERVICE_ID_AGN_MASK) == IB_CM_ASSIGN_SERVICE_ID && (service_id != IB_CM_ASSIGN_SERVICE_ID)) return -EINVAL; From mlleinin at hpcn.ca.sandia.gov Tue Jun 7 11:47:28 2005 From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger) Date: Tue, 07 Jun 2005 11:47:28 -0700 Subject: [openib-general] email archives for March 2004 In-Reply-To: <20050607154313.GA16326@cse.ohio-state.edu> References: <20050607154313.GA16326@cse.ohio-state.edu> Message-ID: <1118170048.14142.524.camel@localhost> On Tue, 2005-06-07 at 11:43 -0400, Sayantan Sur wrote: > Hi, > > I am trying to access the archives for March 2004 ... more specifically > this message link: > > www.openib.org/pipermail/openib-general/2004-March/001513.html > > However, the archives on this webpage: > > http://openib.org/pipermail/openib-general/ > > date back only to July 2004. Is there any way I can access that message > link? Yes, use one of the archives that have the full openib-general history. See http://www.openib.org/contact.html - Matt From mshefty at ichips.intel.com Tue Jun 7 11:50:53 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 07 Jun 2005 11:50:53 -0700 Subject: [openib-general] Re: Send Side RMPP and OpenSM GetTableResp In-Reply-To: <1117480184.4476.110.camel@hal.voltaire.com> References: <1117480184.4476.110.camel@hal.voltaire.com> Message-ID: <42A5EC8D.10600@ichips.intel.com> Hal Rosenstock wrote: > <---------------------------- SA GetTableResp > RMPP flags 0x03 (Data, First) > SegmentNumber 1 > PayloadLength 0x370 > TID 8 > SA GetTable ----------------------------> > RMPP flags 0x02 (ACK) > SegmentNumber 1 > NewWindowLast 6 > TID 8 I'm not clear why RMPP flags is set to 0x02. I would have expected 0x01 to indicate that RMPP is active. If the RMPP active flag is not set, the OpenIB RMPP code will process the packet as if it were a non-RMPP packet. Maybe RMPP flags here indicates the type? (It looks like the flags field on the SA side.) > <---------------------------- SA GetTableResp > RMPP flags 0x01 (Data) > SegmentNumber 2 > PayloadLength 0x370 > TID 8 > <---------------------------- SA GetTableResp > RMPP flags 0x01 (Data) > SegmentNumber 3 > PayloadLength 0x370 > TID 8 > <---------------------------- SA GetTableResp > RMPP flags 0x05 (Data, Last) > SegmentNumber 4 > PayloadLength 0x34 > TID 8 > SA GetTable ----------------------------> > RMPP flags 0x02 (ACK) > SegmentNumber 1 > NewWindowLast 6 > TID 8 Assuming that the RMPP active flag is set, this should be seen as a duplicated ACK and discarded since it doesn't carry any new information. > <---------------------------- SA GetTableResp > RMPP flags 0x01 (Data) > SegmentNumber 5 > PayloadLength 0x34 > TID 8 I'm clueless yet as to how this MAD could have been generated... If I add in some debug checks, can you rerun this test? - Sean From krause at cup.hp.com Tue Jun 7 11:28:48 2005 From: krause at cup.hp.com (Michael Krause) Date: Tue, 07 Jun 2005 11:28:48 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <000101c56b7d$f5940610$9c5aa8c0@infiniconsys.com> References: <52acm2i3et.fsf@topspin.com> <000101c56b7d$f5940610$9c5aa8c0@infiniconsys.com> Message-ID: <6.2.0.14.2.20050607112638.0283e2e0@esmail.cup.hp.com> At 09:28 AM 6/7/2005, Fab Tillier wrote: > > From: Roland Dreier [mailto:roland at topspin.com] > > Sent: Tuesday, June 07, 2005 8:38 AM > > > > Michael> Why not just use the IETF draft for RC / UC based IP over > > Michael> IB and not worry about creating something new? > > > > I think we've come full circle. The original post was a suggestion on > > how to handle the fact the the connected-mode IPoIB draft requires a > > network stack to deal with different MTUs for different destinations > > on the same logical link. > >That's right - by implementing IP segmentation in the IPoIB driver when going >over UD, the driver could expose a single MTU to the network stack, thereby >removing all the issues related to having per-endpoint MTUs. > >Keeping a 2K MTU for RC mode doesn't really take advantage of IB's RC >capabilities. I'd probably target 64K as the MTU. The draft should state a minimum for all RC / UC which should be the TCP MSS. Whether one does SAR over a UD endpoint independent of the underlying physical MTU can be done but it should not require end-to-end understanding of the operation, i.e. the send side tells its local that the TCP MSS is X while the receive side only posts 2-4 KB buffers. This has been done over Ethernet for years. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Tue Jun 7 12:06:43 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 15:06:43 -0400 Subject: [openib-general] Re: Send Side RMPP and OpenSM GetTableResp In-Reply-To: <42A5EC8D.10600@ichips.intel.com> References: <1117480184.4476.110.camel@hal.voltaire.com> <42A5EC8D.10600@ichips.intel.com> Message-ID: <1118171203.4506.15.camel@markf.us.voltaire.com> Hi Sean, On Tue, 2005-06-07 at 14:50, Sean Hefty wrote: > Hal Rosenstock wrote: > > <---------------------------- SA GetTableResp > > RMPP flags 0x03 (Data, First) > > SegmentNumber 1 > > PayloadLength 0x370 > > TID 8 > > SA GetTable ----------------------------> > > RMPP flags 0x02 (ACK) > > SegmentNumber 1 > > NewWindowLast 6 > > TID 8 > > I'm not clear why RMPP flags is set to 0x02. I would have expected 0x01 to > indicate that RMPP is active. If the RMPP active flag is not set, the > OpenIB RMPP code will process the packet as if it were a non-RMPP packet. > Maybe RMPP flags here indicates the type? (It looks like the flags field on > the SA side.) My bad. Type was 0x02 and flags were 0x71 (resp time 0xE and RMPP active). > > <---------------------------- SA GetTableResp > > RMPP flags 0x01 (Data) > > SegmentNumber 2 > > PayloadLength 0x370 > > TID 8 > > <---------------------------- SA GetTableResp > > RMPP flags 0x01 (Data) > > SegmentNumber 3 > > PayloadLength 0x370 > > TID 8 > > <---------------------------- SA GetTableResp > > RMPP flags 0x05 (Data, Last) > > SegmentNumber 4 > > PayloadLength 0x34 > > TID 8 > > SA GetTable ----------------------------> > > RMPP flags 0x02 (ACK) > > SegmentNumber 1 > > NewWindowLast 6 > > TID 8 > > Assuming that the RMPP active flag is set, this should be seen as a > duplicated ACK and discarded since it doesn't carry any new information. Correct. That's what I called a reACK. > > <---------------------------- SA GetTableResp > > RMPP flags 0x01 (Data) > > SegmentNumber 5 > > PayloadLength 0x34 > > TID 8 > > I'm clueless yet as to how this MAD could have been generated... If I add > in some debug checks, can you rerun this test? Yes. I'm working on this right now to limited success so any help is appreciated. Thanks. -- Hal From sean.hefty at intel.com Tue Jun 7 12:21:06 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 7 Jun 2005 12:21:06 -0700 Subject: [openib-general] Re: Send Side RMPP and OpenSM GetTableResp In-Reply-To: <1118171203.4506.15.camel@markf.us.voltaire.com> Message-ID: >> I'm clueless yet as to how this MAD could have been generated... If I add >> in some debug checks, can you rerun this test? > >Yes. I'm working on this right now to limited success so any help is >appreciated. Thanks. Can you apply this patch and send the results when you oops? - Sean Index: mad_rmpp.c =================================================================== --- mad_rmpp.c (revision 2563) +++ mad_rmpp.c (working copy) @@ -588,6 +588,7 @@ mad_send_wr->sg_list[1].lkey = mad_send_wr->sg_list[0].lkey; } + BUG_ON(mad_send_wr->seg_num > mad_send_wr->total_seg); if (mad_send_wr->seg_num == mad_send_wr->total_seg) { rmpp_mad->rmpp_hdr.rmpp_rtime_flags |= IB_MGMT_RMPP_FLAG_LAST; rmpp_mad->rmpp_hdr.paylen_newwin = From mshefty at ichips.intel.com Tue Jun 7 12:27:32 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 07 Jun 2005 12:27:32 -0700 Subject: [openib-general] Re: Send Side RMPP and OpenSM GetTableResp In-Reply-To: <1118171203.4506.15.camel@markf.us.voltaire.com> References: <1117480184.4476.110.camel@hal.voltaire.com> <42A5EC8D.10600@ichips.intel.com> <1118171203.4506.15.camel@markf.us.voltaire.com> Message-ID: <42A5F524.6050403@ichips.intel.com> Hal Rosenstock wrote: >>Assuming that the RMPP active flag is set, this should be seen as a >>duplicated ACK and discarded since it doesn't carry any new information. > > > Correct. That's what I called a reACK. The problem is in process_rmpp_ack(). The check for /* Old ACK */ is incorrect. I'll work on a fix. - Sean From ftillier at silverstorm.com Tue Jun 7 12:59:45 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Tue, 7 Jun 2005 12:59:45 -0700 Subject: [openib-general] CM MRA handling Message-ID: <000301c56b9b$74a1f080$9c5aa8c0@infiniconsys.com> Sean, When setting the delay for a received MRA, the CM doesn't add a packet lifetime as documented in the spec (12.7.32 - Service Timeout). If you do end up adding the packet lifetime, you'll need to revisit the cm_convert_to_ms to prevent overflow of the return value. Overflow happens at IB timeout values > 39. - Fab From libor at topspin.com Tue Jun 7 13:36:16 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 7 Jun 2005 13:36:16 -0700 Subject: [openib-general] [PATCH][SDP] state transition simplification. Message-ID: <20050607133616.A27322@topspin.com> This patch simplifies the state transitions for connection management within SDP. Previously the connection state was maintained using two variables 'istate' and 'state' with a combined total of 34 different values. The patch merges the two into a single variable 'state' and removes redundancy for a total of 16 state values. Also the file sdp_wall.c is removed who's original purpose was to keep the two state variables synchronized. The simplification also paves the way for the next step, which is to use the CM event for transitions instead of the current incorrest use of CM state. The CQ event pending flags were moved from the connection flags field to a new field in the event lock. Finally, I needed to ifdef out one of the send_dreq calls, because I was seeing a problem when two DREQs would cross in flight and both sides responded with a DREP. 13 files changed, 404 insertions(+), 1045 deletions(-) -Libor Index: infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- infiniband/ulp/sdp/sdp_rcvd.c (revision 2564) +++ infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -37,54 +37,27 @@ /* * Specific MID handler functions. (RECV) */ - static int sdp_rcvd_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff) { int result = 0; + int band; sdp_dbg_ctrl(conn, "Disconnect msg received."); switch (conn->state) { case SDP_CONN_ST_ESTABLISHED: - /* - * transition state - */ SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_RECV_1); - /* - * initiate disconnect to framework - */ - result = sdp_wall_recv_close(conn); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> closing connection", - result); - goto error; - } - + band = POLL_IN; break; - case SDP_CONN_ST_DIS_PEND_1: - SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_PEND_R); - break; case SDP_CONN_ST_DIS_SEND_1: SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_RECV_R); + band = POLL_HUP; break; case SDP_CONN_ST_DIS_SENT_1: - /* - * After receiving the final disconnet and posting the DREQ, - * the next step is TIME_WAIT from the CM - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_RECV_R); SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_1); + band = POLL_HUP; +#if 0 /* crossing DREQs appear not to work... */ /* - * acknowledge disconnect to framework (we're in active - * disconnect) - */ - result = sdp_wall_recv_closing(conn); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> confirming conn close", - result); - goto error; - } - /* * Begin IB/CM disconnect */ result = ib_send_cm_dreq(conn->cm_id, NULL, 0); @@ -99,17 +72,28 @@ if (result != -EPROTO) goto error; } - +#endif break; default: - sdp_warn("Unexpected conn state. conn <%d> state <%04x:%04x>", - conn->hashent, conn->istate, conn->state); - - result = -EFAULT; + sdp_warn("Disconnect rcvd, unexpected state. <%d> <%04x>", + conn->hashent, conn->state); + result = -EPROTO; goto error; break; } + conn->shutdown |= RCV_SHUTDOWN; + /* + * cancel all outstanding read AIO's since there will be + * no more data from the peer. + */ + sdp_iocb_q_cancel_all_read(conn, 0); + /* + * async notification. POLL_HUP on full duplex close only. + */ + sdp_inet_wake_generic(conn->sk); + sk_wake_async(conn->sk, 1, band); + return 0; error: return result; @@ -122,23 +106,17 @@ sdp_dbg_ctrl(conn, "Abort msg received."); /* * Connection should be in some post DisConn recveived state. - * Notify gateway interface about abort */ switch (conn->state) { case SDP_CONN_ST_DIS_RECV_1: - case SDP_CONN_ST_DIS_PEND_R: case SDP_CONN_ST_DIS_RECV_R: - case SDP_CONN_ST_DIS_PEND_2: case SDP_CONN_ST_DIS_SEND_2: - case SDP_CONN_ST_DIS_SENT_2: - result = sdp_wall_abort(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> during abort", result); + sdp_conn_abort(conn); break; default: - sdp_warn("Unexpected abort. conn <%d> state <%04x:%04x>", - conn->hashent, conn->istate, conn->state); + sdp_warn("Unexpected abort. conn <%d> state <%04x>", + conn->hashent, conn->state); result = -EPROTO; } @@ -856,8 +834,7 @@ /* * abort connection (send reset) */ - result = sdp_wall_abort(conn); - SDP_EXPECT(result >= 0); + sdp_conn_abort(conn); /* * drop packet */ Index: infiniband/ulp/sdp/sdp_inet.c =================================================================== --- infiniband/ulp/sdp/sdp_inet.c (revision 2564) +++ infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -209,112 +209,78 @@ } /* - * internal socket/handle managment functions - */ - -/* - * sdp_inet_abort - abort an existing connection - */ -static int sdp_inet_abort(struct sdp_opt *conn) -{ - int result; - - conn->send_buf = 0; - - switch (conn->istate) { - case SDP_SOCK_ST_CONNECT: - case SDP_SOCK_ST_ACCEPTING: - case SDP_SOCK_ST_ACCEPTED: - sdp_dbg_warn(conn, "Unexpected abort"); - - case SDP_SOCK_ST_ESTABLISHED: - case SDP_SOCK_ST_CLOSE: - case SDP_SOCK_ST_DISCONNECT: - case SDP_SOCK_ST_CLOSING: - result = sdp_wall_abort(conn); - if (result < 0) { - - result = -ECONNABORTED; - SDP_CONN_SET_ERR(conn, ECONNABORTED); - conn->istate = SDP_SOCK_ST_ERROR; - } - - break; - case SDP_SOCK_ST_LISTEN: - case SDP_SOCK_ST_CLOSED: - case SDP_SOCK_ST_ERROR: - sdp_dbg_warn(conn, "Unhandled abort"); - - conn->istate = SDP_SOCK_ST_ERROR; - result = -EINVAL; - break; - default: - sdp_dbg_warn(conn, "Unknown abort state"); - - conn->istate = SDP_SOCK_ST_ERROR; - result = -EINVAL; - break; - } - - return result; -} - -/* * sdp_inet_disconnect - disconnect a connection */ static int sdp_inet_disconnect(struct sdp_opt *conn) { int result = 0; + /* + * close buffered data transmission space + */ + conn->send_buf = 0; + /* + * Generate a Disconnect message, and mark self as disconnecting. + */ + switch (conn->state) { + case SDP_CONN_ST_REQ_PATH: + case SDP_CONN_ST_REQ_SENT: + /* + * outstanding request. Mark it in error, and + * completions needs to complete the closing. + */ + SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); + SDP_CONN_SET_ERR(conn, ECONNRESET); + break; + case SDP_CONN_ST_REQ_RECV: + case SDP_CONN_ST_REP_RECV: + case SDP_CONN_ST_ESTABLISHED: + /* + * Attempt to send a disconnect message + */ + SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_SEND_1); - switch (conn->istate) { - case SDP_SOCK_ST_CONNECT: - result = sdp_wall_abort(conn); + result = sdp_send_ctrl_disconnect(conn); if (result < 0) { - result = -ECONNABORTED; - SDP_CONN_SET_ERR(conn, ECONNABORTED); - conn->istate = SDP_SOCK_ST_ERROR; + sdp_dbg_warn(conn, + "Error <%d> send disconnect request", + result); + goto error; } - break; - case SDP_SOCK_ST_ESTABLISHED: - case SDP_SOCK_ST_ACCEPTED: - conn->istate = SDP_SOCK_ST_DISCONNECT; - result = sdp_wall_send_close(conn); - if (result < 0) { - result = -ECONNABORTED; - SDP_CONN_SET_ERR(conn, ECONNABORTED); - conn->istate = SDP_SOCK_ST_ERROR; - } - break; - case SDP_SOCK_ST_CLOSE: - conn->istate = SDP_SOCK_ST_CLOSING; - result = sdp_wall_send_closing(conn); - if (result < 0) { - result = -ECONNABORTED; - SDP_CONN_SET_ERR(conn, ECONNABORTED); - conn->istate = SDP_SOCK_ST_ERROR; - } break; - case SDP_SOCK_ST_ACCEPTING: - case SDP_SOCK_ST_DISCONNECT: - case SDP_SOCK_ST_CLOSING: + case SDP_CONN_ST_DIS_RECV_1: /* - * nothing to do, and somewhat unexpected state + * Change state, and send a disconnect request */ - sdp_dbg_warn(conn, "Unexpected disconnect"); + SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_SEND_2); + result = sdp_send_ctrl_disconnect(conn); + if (result < 0) { + sdp_dbg_warn(conn, + "Error <%d> send disconnect request", + result); + goto error; + } break; - case SDP_SOCK_ST_LISTEN: - case SDP_SOCK_ST_CLOSED: - case SDP_SOCK_ST_ERROR: + case SDP_CONN_ST_TIME_WAIT_1: + case SDP_CONN_ST_TIME_WAIT_2: + case SDP_CONN_ST_ERROR: + case SDP_CONN_ST_CLOSED: break; default: - sdp_dbg_warn(conn, "Unknown disconnect state"); - conn->istate = SDP_SOCK_ST_ERROR; - result = -EINVAL; - break; + sdp_dbg_warn(conn, "Incorrect state for disconnect"); + result = -EBADE; + goto error; } + return 0; +error: + /* + * abortive close. + */ + sdp_conn_inet_error(conn, ECONNRESET); + (void)ib_send_cm_dreq(conn->cm_id, NULL, 0); + return result; } @@ -353,7 +319,7 @@ sdp_conn_lock(conn); conn->shutdown = SHUTDOWN_MASK; - if (conn->istate == SDP_SOCK_ST_LISTEN) { + if (conn->state == SDP_CONN_ST_LISTEN) { /* * stop listening */ @@ -384,22 +350,16 @@ /* * abort. */ - result = sdp_inet_abort(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> while aborting socket", - result); - + sdp_conn_abort(conn); goto done; } /* - * disconnect. (state dependant) + * disconnect. (state dependant) On error skip linger, since + * the socket is already out of the normal path. */ result = sdp_inet_disconnect(conn); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> while disconnecting socket", - result); + if (result < 0) goto done; - } /* * Skip lingering/canceling if * non-blocking and not exiting. @@ -419,7 +379,7 @@ set_current_state(TASK_INTERRUPTIBLE); while (timeout > 0 && - !(SDP_ST_MASK_CLOSED & conn->istate)) { + !(SDP_ST_MASK_CLOSED & conn->state)) { sdp_conn_unlock(conn); timeout = schedule_timeout(timeout); sdp_conn_lock(conn); @@ -437,7 +397,7 @@ * Cancel write and close again to force closing the * connection. */ - if (SDP_ST_MASK_DRAIN & conn->istate) { + if (SDP_ST_MASK_DRAIN & conn->state) { sdp_iocb_q_cancel_all_write(conn, -ECANCELED); @@ -448,9 +408,6 @@ } done: - if (SDP_ST_MASK_CLOSED & conn->istate) - do {} while(0); /* pass */ - /* * finally drop socket reference. (socket API reference) */ @@ -513,7 +470,7 @@ */ sdp_conn_lock(conn); - if (conn->istate != SDP_SOCK_ST_CLOSED || conn->src_port > 0) { + if (conn->state != SDP_CONN_ST_CLOSED || conn->src_port > 0) { result = -EINVAL; goto done; } @@ -590,7 +547,7 @@ switch (sock->state) { case SS_UNCONNECTED: - if (!(SDP_ST_MASK_CLOSED & conn->istate)) { + if (conn->state != SDP_CONN_ST_CLOSED) { result = -EISCONN; goto done; } @@ -610,7 +567,6 @@ SDP_CONN_SET_ERR(conn, 0); sock->state = SS_CONNECTING; - conn->istate = SDP_SOCK_ST_CONNECT; conn->dst_addr = ntohl(addr->sin_addr.s_addr); conn->dst_port = ntohs(addr->sin_port); @@ -630,7 +586,7 @@ conn->dst_port = 0; sock->state = SS_UNCONNECTED; - conn->istate = SDP_SOCK_ST_CLOSED; + conn->state = SDP_CONN_ST_CLOSED; goto done; } @@ -660,7 +616,7 @@ add_wait_queue(sk->sk_sleep, &wait); set_current_state(TASK_INTERRUPTIBLE); - while (timeout > 0 && conn->istate == SDP_SOCK_ST_CONNECT) { + while (timeout > 0 && (conn->state & SDP_ST_MASK_CONNECT)) { sdp_conn_unlock(conn); timeout = schedule_timeout(timeout); @@ -675,7 +631,7 @@ set_current_state(TASK_RUNNING); remove_wait_queue(sk->sk_sleep, &wait); - if (conn->istate == SDP_SOCK_ST_CONNECT) { + if (conn->state & SDP_ST_MASK_CONNECT) { if (timeout > 0) { @@ -691,16 +647,18 @@ * point. In this case connect should return normally and allow * the normal mechnaism for detecting these states. */ - switch (conn->istate) { - case SDP_SOCK_ST_CONNECT: + switch (conn->state) { + case SDP_CONN_ST_REQ_PATH: + case SDP_CONN_ST_REQ_SENT: + case SDP_CONN_ST_REP_RECV: break; - case SDP_SOCK_ST_ESTABLISHED: - case SDP_SOCK_ST_CLOSE: + case SDP_CONN_ST_ESTABLISHED: + case SDP_CONN_ST_DIS_RECV_1: sock->state = SS_CONNECTED; result = 0; break; - case SDP_SOCK_ST_CLOSED: - case SDP_SOCK_ST_ERROR: + case SDP_CONN_ST_CLOSED: + case SDP_CONN_ST_ERROR: result = sdp_conn_error(conn) ? : -ECONNABORTED; sock->state = SS_UNCONNECTED; break; @@ -711,7 +669,6 @@ } sdp_dbg_ctrl(conn, "CONNECT complete"); - done: sdp_conn_unlock(conn); return result; @@ -735,13 +692,13 @@ sdp_conn_lock(conn); if (SS_UNCONNECTED != sock->state || - (conn->istate != SDP_SOCK_ST_CLOSED && - conn->istate != SDP_SOCK_ST_LISTEN)) { + (conn->state != SDP_CONN_ST_CLOSED && + conn->state != SDP_CONN_ST_LISTEN)) { result = -EINVAL; goto done; } - if (conn->istate != SDP_SOCK_ST_LISTEN) { + if (conn->state != SDP_CONN_ST_LISTEN) { result = sdp_inet_listen_start(conn); if (result < 0) { sdp_dbg_warn(conn, "Error <%d> starting listen", @@ -795,7 +752,7 @@ sdp_conn_lock(listen_conn); - if (listen_conn->istate != SDP_SOCK_ST_LISTEN) { + if (listen_conn->state != SDP_CONN_ST_LISTEN) { result = -EINVAL; goto listen_done; } @@ -816,7 +773,7 @@ set_current_state(TASK_INTERRUPTIBLE); while (timeout > 0 && - listen_conn->istate == SDP_SOCK_ST_LISTEN && + listen_conn->state == SDP_CONN_ST_LISTEN && !listen_conn->backlog_cnt) { sdp_conn_unlock(listen_conn); timeout = schedule_timeout(timeout); @@ -834,7 +791,7 @@ if (!listen_conn->backlog_cnt) { result = 0; - if (listen_conn->istate != SDP_SOCK_ST_LISTEN) + if (listen_conn->state != SDP_CONN_ST_LISTEN) result = -EINVAL; if (signal_pending(current)) result = sock_intr_errno(timeout); @@ -846,47 +803,19 @@ } else { accept_sk = accept_conn->sk; - switch (accept_conn->istate) { - case SDP_SOCK_ST_ACCEPTED: + switch (accept_conn->state) { + case SDP_CONN_ST_REQ_RECV: + case SDP_CONN_ST_ESTABLISHED: + case SDP_CONN_ST_DIS_RECV_1: sock_graft(accept_sk, accept_sock); accept_conn->pid = current->pid; accept_sock->state = SS_CONNECTED; - accept_conn->istate = - SDP_SOCK_ST_ESTABLISHED; sdp_inet_wake_send(accept_sk); break; - case SDP_SOCK_ST_ACCEPTING: - sock_graft(accept_sk, accept_sock); - - accept_conn->pid = current->pid; - accept_sock->state = SS_CONNECTED; - - accept_conn->istate = SDP_SOCK_ST_ACCEPTED; - /* - * connection completion/establishment will - * open this up - */ - set_bit(SOCK_NOSPACE, &accept_sock->flags); - - break; - case SDP_SOCK_ST_CLOSE: - sock_graft(accept_sk, accept_sock); - - accept_conn->pid = current->pid; - accept_sock->state = SS_CONNECTED; - - sdp_inet_wake_send(accept_sk); - - break; default: - sdp_dbg_warn(accept_conn, "bad accept state"); - - case SDP_SOCK_ST_CLOSED: - case SDP_SOCK_ST_CLOSING: - case SDP_SOCK_ST_ERROR: /* * this accept socket has problems, keep * trying. @@ -950,7 +879,7 @@ addr->sin_family = proto_family; if (peer > 0) if (htons(conn->dst_port) > 0 && - !(SDP_ST_MASK_CLOSED & conn->istate)) { + !(SDP_ST_MASK_CLOSED & conn->state)) { addr->sin_port = htons(conn->dst_port); addr->sin_addr.s_addr = htonl(conn->dst_addr); @@ -996,17 +925,17 @@ /* * no locking, should be safe as is. */ - switch (conn->istate) { - case SDP_SOCK_ST_LISTEN: + switch (conn->state) { + case SDP_CONN_ST_LISTEN: mask |= (conn->backlog_cnt > 0) ? (POLLIN | POLLRDNORM) : 0; break; - case SDP_SOCK_ST_ERROR: + case SDP_CONN_ST_ERROR: mask |= POLLERR; break; - case SDP_SOCK_ST_CLOSED: + case SDP_CONN_ST_CLOSED: mask |= POLLHUP; break; - case SDP_SOCK_ST_ESTABLISHED: + case SDP_CONN_ST_ESTABLISHED: /* * fall through */ @@ -1131,7 +1060,7 @@ case SIOCINQ: sdp_conn_lock(conn); - if (conn->istate != SDP_SOCK_ST_LISTEN) { + if (conn->state != SDP_CONN_ST_LISTEN) { /* * TODO need to subtract/add URG (inline vs. OOB) */ @@ -1145,7 +1074,7 @@ case SIOCOUTQ: sdp_conn_lock(conn); - if (conn->istate != SDP_SOCK_ST_LISTEN) { + if (conn->state != SDP_CONN_ST_LISTEN) { value = conn->send_qud; result = put_user(value, (int __user *) arg); } else @@ -1332,15 +1261,18 @@ conn->shutdown |= flag; - switch (conn->istate) { - case SDP_SOCK_ST_CLOSED: - result = -ENOTCONN; - break; - case SDP_SOCK_ST_LISTEN: + switch (conn->state) { + case SDP_CONN_ST_REQ_PATH: + case SDP_CONN_ST_REQ_SENT: /* - * Send shutdown is benign. + * outstanding request. Mark it in error, and + * completions needs to complete the closing. */ - if (RCV_SHUTDOWN & flag) { + SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); + SDP_CONN_SET_ERR(conn, ECONNRESET); + break; + case SDP_CONN_ST_LISTEN: + if (flag & RCV_SHUTDOWN) { result = sdp_inet_listen_stop(conn); if (result < 0) sdp_dbg_warn(conn, "listen stop error <%d>", @@ -1348,52 +1280,22 @@ } break; - case SDP_SOCK_ST_ERROR: - result = sdp_conn_error(conn); - result = (result < 0) ? result : -ECONNABORTED; + case SDP_CONN_ST_CLOSED: + case SDP_CONN_ST_ERROR: + result = -ENOTCONN; break; - case SDP_SOCK_ST_ACCEPTED: - case SDP_SOCK_ST_CONNECT: - result = sdp_inet_abort(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> aborting connection", - result); - break; - case SDP_SOCK_ST_ESTABLISHED: - case SDP_SOCK_ST_CLOSE: - result = sdp_inet_disconnect(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> disconnecting conn", - result); - break; - case SDP_SOCK_ST_DISCONNECT: - case SDP_SOCK_ST_CLOSING: - /* - * nothing. - */ - break; - case SDP_SOCK_ST_ACCEPTING: - sdp_dbg_warn(conn, "connection state error"); - - conn->istate = SDP_SOCK_ST_ERROR; - SDP_CONN_SET_ERR(conn, EPROTO); - result = -EFAULT; - - break; default: - sdp_warn("Unknown socket state. conn <%d> state <%04x:%04x>", - conn->hashent, conn->istate, conn->state); + if (!(flag & RCV_SHUTDOWN)) { + result = sdp_inet_disconnect(conn); + if (result < 0) + sdp_dbg_warn(conn, "disconnect error <%d>", + result); + } - conn->istate = SDP_SOCK_ST_ERROR; - SDP_CONN_SET_ERR(conn, EPROTO); - result = -EFAULT; + break; } - if (result < 0) - sdp_inet_wake_generic(sock->sk); - else - sdp_inet_wake_error(sock->sk); - + sdp_inet_wake_generic(sock->sk); sdp_conn_unlock(conn); return result; } Index: infiniband/ulp/sdp/sdp_proto.h =================================================================== --- infiniband/ulp/sdp/sdp_proto.h (revision 2564) +++ infiniband/ulp/sdp/sdp_proto.h (working copy) @@ -114,22 +114,10 @@ /* * Wall between userspace protocol and SDP protocol proper */ -int sdp_wall_send_close(struct sdp_opt *conn); +void sdp_conn_abort(struct sdp_opt *conn); -int sdp_wall_send_closing(struct sdp_opt *conn); +void sdp_conn_inet_error(struct sdp_opt *conn, int error); -int sdp_wall_send_abort(struct sdp_opt *conn); - -int sdp_wall_recv_close(struct sdp_opt *conn); - -int sdp_wall_recv_closing(struct sdp_opt *conn); - -int sdp_wall_recv_abort(struct sdp_opt *conn); - -void sdp_wall_recv_drop(struct sdp_opt *conn); - -int sdp_wall_abort(struct sdp_opt *conn); - int sdp_recv_buff(struct sdp_opt *conn, struct sdpc_buff *buff); /* @@ -332,8 +320,6 @@ /* * passive connect functions */ -void sdp_cm_pass_error(struct sdp_opt *conn, int error); - int sdp_cm_pass_establish(struct sdp_opt *conn); int sdp_cm_req_handler(struct ib_cm_id *cm_id, @@ -481,9 +467,9 @@ struct sdp_opt *x = (conn); \ if (x) { \ sdp_dbg_out(level, type, \ - "<%d> <%04x:%04x> " format, \ - x->hashent, x->istate, x->state , \ - ## arg); \ + "<%d> <%04x> " format, \ + x->hashent, x->state , \ + ## arg); \ } \ else { \ sdp_dbg_out(level, type, format, ## arg); \ @@ -555,19 +541,18 @@ static inline int sdp_inet_write_space(struct sdp_opt *conn, int urg) { int size; - /* - * Allow for more space if Urgent data is being considered + * Allow for more space if Urgent data is being considered. + * send_buf may be zero if we are holding data back, state + * transition will open it. */ size = (conn->send_buf - conn->send_qud); /* * write space is determined by amount of outstanding bytes of data * and number of buffers used for transmission by this connection */ - if ((SDP_ST_MASK_OPEN & conn->istate) && - (conn->send_max > - sdp_desc_q_types_size(&conn->send_queue, - SDP_DESC_TYPE_BUFF))) + if (conn->send_max > sdp_desc_q_types_size(&conn->send_queue, + SDP_DESC_TYPE_BUFF)) return ((SDP_INET_SEND_MARK < size || 1 < urg) ? size : 0); else return 0; @@ -578,7 +563,7 @@ */ static inline int sdp_inet_writable(struct sdp_opt *conn) { - if (SDP_ST_MASK_OPEN & conn->istate) + if (conn->send_buf > 0) return (sdp_inet_write_space(conn, 0) < (conn->send_qud / 2)) ? 0 : 1; else Index: infiniband/ulp/sdp/Makefile =================================================================== --- infiniband/ulp/sdp/Makefile (revision 2564) +++ infiniband/ulp/sdp/Makefile (working copy) @@ -12,7 +12,6 @@ sdp_sent.o \ sdp_read.o \ sdp_write.o \ - sdp_wall.o \ sdp_proc.o \ sdp_inet.o \ sdp_iocb.o \ Index: infiniband/ulp/sdp/sdp_send.c =================================================================== --- infiniband/ulp/sdp/sdp_send.c (revision 2564) +++ infiniband/ulp/sdp/sdp_send.c (working copy) @@ -1247,8 +1247,7 @@ buff = sdp_buff_pool_get(); if (!buff) { sdp_dbg_warn(conn, "Failed to allocate buffer for control"); - result = -ENOMEM; - goto error; + return -ENOMEM; } /* * setup header. @@ -1282,7 +1281,6 @@ (void)sdp_buff_pool_put(buff); } -error: return result; } @@ -1301,8 +1299,7 @@ buff = sdp_buff_pool_get(); if (!buff) { sdp_dbg_warn(conn, "Failed to allocate buffer for disconnect"); - result = -ENOMEM; - goto error; + return -ENOMEM; } /* * setup header. @@ -1316,20 +1313,18 @@ SDP_BUFF_F_CLR_SE(buff); SDP_BUFF_F_CLR_UNSIG(buff); /* - * change state to reflect disconnect is queued. - * DIS_PEND_X to DIS_SEND_X - */ - SDP_ST_PEND_2_SEND(conn); - /* * send */ result = sdp_send_ctrl_buff_buffered(conn, buff); if (result < 0) { sdp_dbg_warn(conn, "Error <%d> posting control message", result); + (void)sdp_buff_pool_put(buff); goto error; } - + + conn->flags &= ~SDP_CONN_F_DIS_PEND; + return 0; error: return result; @@ -1340,8 +1335,7 @@ */ int sdp_send_ctrl_disconnect(struct sdp_opt *conn) { - int result; - + int result = 0; /* * Only create/post the message if there is no data in the data queue, * otherwise ignore the call. The flush queue will see to it, that a @@ -1349,20 +1343,13 @@ * clean. The state is now in a disconnect send, the message will be * sent once data is flushed. */ - if (SDP_ST_MASK_DIS_PEND & conn->state) { - if (!(conn->flags & SDP_CONN_F_DIS_HOLD) && - !sdp_desc_q_size(&conn->send_queue) && - !conn->src_sent) - result = do_send_ctrl_disconnect(conn); - else { - sdp_dbg_ctrl(conn, "defer disconnect <%d:%d> <%08x>", - sdp_desc_q_size(&conn->send_queue), - conn->src_sent, conn->flags); + if ((conn->flags & SDP_CONN_F_DIS_HOLD) || + sdp_desc_q_size(&conn->send_queue) || + conn->src_sent) + conn->flags |= SDP_CONN_F_DIS_PEND; + else + result = do_send_ctrl_disconnect(conn); - result = 0; - } - } else - result = -EPROTO; return result; } @@ -1995,8 +1982,8 @@ /* * disconnect flush */ - if (SDP_ST_MASK_DIS_PEND & conn->state) { - result = do_send_ctrl_disconnect(conn); + if (conn->flags & SDP_CONN_F_DIS_PEND) { + result = sdp_send_ctrl_disconnect(conn); if (result < 0) { sdp_dbg_warn(conn, "Error <%d> flushing disconnect", result); @@ -2049,8 +2036,9 @@ * continue being processed, it'll wait below until the send window * is opened on sucessful connect, or error on an unsucessful attempt. */ - if (SDP_ST_MASK_CLOSED & conn->istate) { - result = -EPIPE; + if (conn->state == SDP_CONN_ST_LISTEN || + conn->state == SDP_CONN_ST_CLOSED) { + result = -ENOTCONN; goto done; } /* @@ -2139,7 +2127,7 @@ break; } - if (conn->istate == SDP_SOCK_ST_ERROR) { + if (conn->state == SDP_CONN_ST_ERROR) { result = -EPROTO; /* error should always be set, but just in case */ break; Index: infiniband/ulp/sdp/sdp_conn.c =================================================================== --- infiniband/ulp/sdp/sdp_conn.c (revision 2564) +++ infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -73,6 +73,101 @@ return psn; } +void sdp_conn_inet_error(struct sdp_opt *conn, int error) +{ + sdp_dbg_ctrl(conn, "report connection error <%d>", error); + /* + * the connection has failed, move to error, and notify anyone + * waiting of the state change. remove connection from listen + * queue if possible. + */ + (void)sdp_inet_accept_q_remove(conn); + + SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); + SDP_CONN_SET_ERR(conn, error); + conn->shutdown = SHUTDOWN_MASK; + conn->send_buf = 0; + + if (conn->sk->sk_socket) + conn->sk->sk_socket->state = SS_UNCONNECTED; + + sdp_iocb_q_cancel_all(conn, -error); + sdp_inet_wake_error(conn->sk); +} + +void sdp_conn_abort(struct sdp_opt *conn) +{ + int result; + int error = ECONNRESET; + + sdp_dbg_ctrl(conn, "Abort send. src <%08x:%04x> dst <%08x:%04x>", + conn->src_addr, conn->src_port, + conn->dst_addr, conn->dst_port); + + switch (conn->state) { + case SDP_CONN_ST_DIS_SENT_1: /* IBTA v1.1 spec A4.5.3.2 */ + /* + * clear the pending control buffer. + */ + sdp_desc_q_clear(&conn->send_ctrl); + /* + * fall through + */ + case SDP_CONN_ST_DIS_SEND_2: + case SDP_CONN_ST_DIS_SEND_1: + /* + * don't touch control queue, diconnect message may + * still be queued. + */ + sdp_desc_q_clear(&conn->send_queue); + /* + * post abort + */ + if (conn->flags & SDP_CONN_F_DIS_PEND) + result = -ECONNRESET; + else + result = sdp_send_ctrl_abort(conn); + + if (!result) + break; + + conn->flags &= ~SDP_CONN_F_DIS_PEND; + case SDP_CONN_ST_DIS_RECV_1: + error = EPIPE; + case SDP_CONN_ST_ESTABLISHED: + /* + * abortive close. + */ + result = ib_send_cm_dreq(conn->cm_id, NULL, 0); + if (result) + sdp_dbg_warn(conn, "Error <%d> CM disconnect send", + result); + break; + case SDP_CONN_ST_REQ_PATH: + case SDP_CONN_ST_REQ_SENT: + case SDP_CONN_ST_REQ_RECV: + /* + * outstanding CM request. Mark it in error, and CM + * completion needs to complete the closing. + */ + error = ECONNREFUSED; + break; + case SDP_CONN_ST_ERROR: + case SDP_CONN_ST_CLOSED: + case SDP_CONN_ST_TIME_WAIT_1: + case SDP_CONN_ST_TIME_WAIT_2: + break; + default: + /* + * post abort + */ + sdp_dbg_warn(conn, "Unexpected connection state for abort"); + break; + } + + sdp_conn_inet_error(conn, error); + return; +} /* * sdp_inet_accept_q_put - put a conn into a listen conn's accept Q. */ @@ -175,6 +270,8 @@ accept_conn->accept_prev = NULL; accept_conn->parent = NULL; + sdp_conn_put(accept_conn); /* AcceptQueue INET reference */ + return 0; } @@ -185,12 +282,12 @@ { unsigned long flags; - if (conn->istate != SDP_SOCK_ST_CLOSED) { + if (conn->state != SDP_CONN_ST_CLOSED) { sdp_dbg_warn(conn, "Incorrect connection state to listen."); return -EBADFD; } - conn->istate = SDP_SOCK_ST_LISTEN; + conn->state = SDP_CONN_ST_LISTEN; conn->accept_next = conn; conn->accept_prev = conn; /* @@ -217,15 +314,14 @@ int sdp_inet_listen_stop(struct sdp_opt *listen_conn) { struct sdp_opt *accept_conn; - int result; unsigned long flags; - if (listen_conn->istate != SDP_SOCK_ST_LISTEN) { + if (listen_conn->state != SDP_CONN_ST_LISTEN) { sdp_dbg_warn(listen_conn, "Incorrect state to stop listen."); return -EBADFD; } - listen_conn->istate = SDP_SOCK_ST_CLOSED; + listen_conn->state = SDP_CONN_ST_CLOSED; /* * table lock */ @@ -252,13 +348,7 @@ * Remember to unlock since the Get function will acquire * the lock. */ - accept_conn->istate = SDP_SOCK_ST_CLOSED; - - result = sdp_wall_send_abort(accept_conn); - if (result < 0) - sdp_dbg_warn(accept_conn, "Error <%d> during abort", - result); - + sdp_conn_abort(accept_conn); /* AcceptQueueGet */ sdp_conn_unlock(accept_conn); /* INET reference (AcceptQueuePut). */ @@ -341,7 +431,7 @@ */ if (!sk->sk_reuse || !srch->sk_reuse || - look->istate == SDP_SOCK_ST_LISTEN) { + look->state == SDP_CONN_ST_LISTEN) { /* * 5) neither socket is using a * specific address @@ -848,7 +938,7 @@ break; /* exit CQ handler routine */ } - conn->flags &= ~SDP_CONN_F_MASK_EVENT; + conn->lock.event = 0; spin_unlock_irqrestore(&conn->lock.slock, flags); } @@ -902,7 +992,7 @@ result); } -return calls; + return calls; } /* @@ -914,15 +1004,13 @@ /* * poll CQs for events. */ - if (conn) { - if (conn->flags & SDP_CONN_F_RECV_CQ_PEND) - calls += sdp_conn_cq_drain(conn->recv_cq, conn); + if (conn->lock.event & SDP_LOCK_F_RECV_CQ) + calls += sdp_conn_cq_drain(conn->recv_cq, conn); - if (conn->flags & SDP_CONN_F_SEND_CQ_PEND) - calls += sdp_conn_cq_drain(conn->send_cq, conn); + if (conn->lock.event & SDP_LOCK_F_SEND_CQ) + calls += sdp_conn_cq_drain(conn->send_cq, conn); - conn->flags &= ~SDP_CONN_F_MASK_EVENT; - } + conn->lock.event = 0; } /* @@ -932,6 +1020,7 @@ { spin_lock_init(&(conn->lock.slock)); conn->lock.users = 0; + conn->lock.event = 0; init_waitqueue_head(&(conn->lock.waitq)); } @@ -1176,7 +1265,6 @@ conn->pid = 0; conn->sk = sk; conn->hashent = SDP_DEV_SK_INVALID; - conn->istate = SDP_SOCK_ST_CLOSED; conn->flags = 0; conn->shutdown = 0; conn->recv_mode = SDP_MODE_COMB; @@ -1409,19 +1497,19 @@ return offset; } -#define SDP_CONN_PROC_DATA_SIZE 176 /* output line size. */ +#define SDP_CONN_PROC_DATA_SIZE 171 /* output line size. */ #define SDP_PROC_CONN_DATA_HEAD \ - " ID conn inet r s fl sh send_buf recv_buf send q'd recv q'd " \ + " ID conn r s fl sh send_buf recv_buf send q'd recv q'd " \ "send_seq recv_seq advt_seq smax rmax recv_max lrcv " \ "lavt rrcv sd sc sp rd rp swqs rbuf sbuf " \ "us cu send_oob rurg back maxb\n" #define SDP_PROC_CONN_DATA_SEP \ - "---- ---- ---- - - -- -- -------- -------- -------- -------- " \ + "---- ---- - - -- -- -------- -------- -------- -------- " \ "-------- -------- -------- ---- ---- -------- ---- " \ "---- ---- -- -- -- -- -- ---- ---- ---- " \ "-- -- -------- ---- ---- ----\n" #define SDP_PROC_CONN_DATA_FORM \ - "%04x %04x %04x %01x %01x %02x %02x " \ + "%04x %04x %01x %01x %02x %02x " \ "%08x %08x %08x %08x %08x %08x " \ "%08x %04x %04x %08x %04x %04x %04x %02x %02x " \ "%02x %02x %02x %04x %04x %04x %02x %02x %08x %04x %04x %04x\n" @@ -1470,7 +1558,6 @@ offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_FORM, conn->hashent, conn->state, - conn->istate, conn->recv_mode, conn->send_mode, conn->flags, Index: infiniband/ulp/sdp/sdp_actv.c =================================================================== --- infiniband/ulp/sdp/sdp_actv.c (revision 2564) +++ infiniband/ulp/sdp/sdp_actv.c (working copy) @@ -49,19 +49,17 @@ */ switch (conn->state) { default: - sdp_dbg_warn(conn, "REP error in unknown connection state"); + sdp_dbg_warn(conn, "Error in unknown connection state"); + case SDP_CONN_ST_REQ_SENT: case SDP_CONN_ST_REQ_PATH: /* * CM message was never sent. */ - conn->state = SDP_CONN_ST_CLOSED; + SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); + case SDP_CONN_ST_ERROR: case SDP_CONN_ST_CLOSED: break; - case SDP_CONN_ST_ERROR_STRM: /* socket has been destroyed. */ - error = ECONNRESET; - case SDP_CONN_ST_REQ_SENT: case SDP_CONN_ST_REP_RECV: - case SDP_CONN_ST_RTU_SENT: /* * All four states we have gotten a REP and are now in * one of these states. @@ -74,7 +72,7 @@ sdp_dbg_warn(conn, "Error <%d> sending CM REJ.", result); - conn->state = SDP_CONN_ST_CLOSED; + SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); break; case SDP_CONN_ST_ESTABLISHED: /* @@ -86,12 +84,11 @@ sdp_dbg_warn(NULL, "Error <%d> sending CM DREQ", result); - conn->state = SDP_CONN_ST_TIME_WAIT_1; + SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_1); break; } SDP_CONN_SET_ERR(conn, error); - conn->istate = SDP_SOCK_ST_ERROR; conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; @@ -142,10 +139,6 @@ goto done; } /* - * finalize connection acceptance. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_RTU_SENT); - /* * post receive buffers. */ result = sdp_recv_flush(conn); @@ -183,7 +176,6 @@ SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED); sk->sk_socket->state = SS_CONNECTED; - conn->istate = SDP_SOCK_ST_ESTABLISHED; conn->send_buf = SDP_INET_SEND_SIZE; /* * release disconnects. @@ -271,7 +263,7 @@ struct sdp_opt *conn) { struct sdp_msg_hello_ack *hello_ack; - int result = -EPROTO; + int result = -ECONNRESET; if (cm_id != conn->cm_id) { sdp_dbg_warn(conn, "REP comm ID mismatch. <%08x:%08x>", @@ -286,6 +278,7 @@ if (conn->state != SDP_CONN_ST_REQ_SENT) goto error; + SDP_CONN_ST_SET(conn, SDP_CONN_ST_REP_RECV); /* * check Hello Header Ack, to determine if we want * the connection. @@ -296,7 +289,6 @@ goto error; } - SDP_CONN_ST_SET(conn, SDP_CONN_ST_REP_RECV); /* * read remote information */ @@ -491,7 +483,7 @@ param.local_cm_response_timeout = 20; param.retry_count = SDP_CM_PARAM_RETRY; param.rnr_retry_count = SDP_CM_PARAM_RNR_RETRY; - param.max_cm_retries = 15; + param.max_cm_retries = 7; #if 0 /* XXX set timeout to default value of 14 */ path->packet_life = 13; Index: infiniband/ulp/sdp/sdp_wall.c =================================================================== --- infiniband/ulp/sdp/sdp_wall.c (revision 2365) +++ infiniband/ulp/sdp/sdp_wall.c (working copy) @@ -1,450 +0,0 @@ -/* - * Copyright (c) 2005 Topspin Communications. All rights reserved. - * - * This software is available to you under a choice of one of two - * licenses. You may choose to be licensed under the terms of the GNU - * General Public License (GPL) Version 2, available from the file - * COPYING in the main directory of this source tree, or the - * OpenIB.org BSD license below: - * - * Redistribution and use in source and binary forms, with or - * without modification, are permitted provided that the following - * conditions are met: - * - * - Redistributions of source code must retain the above - * copyright notice, this list of conditions and the following - * disclaimer. - * - * - Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following - * disclaimer in the documentation and/or other materials - * provided with the distribution. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF - * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS - * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN - * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN - * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE - * SOFTWARE. - * - * $Id$ - */ - -#include "sdp_main.h" - -/* - * SDP protocol (public functions) - */ - -/* - * sdp_wall_send_close - callback to accept an active close - */ -int sdp_wall_send_close(struct sdp_opt *conn) -{ - int result; - - sdp_dbg_ctrl(conn, "Close send. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port); - /* - * close buffered data transmission space - */ - conn->send_buf = 0; - /* - * Generate a Disconnect message, and mark self as disconnecting. - */ - switch (conn->state) { - case SDP_CONN_ST_REP_SENT: - /* - * clear out the sent HelloAck message - */ - (void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post)); - /* - * fall through - */ - case SDP_CONN_ST_RTU_SENT: - /* - * wait for established before attempting to send. this - * way we'll reduce the number of state permutations. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_PEND_1); - - break; - case SDP_CONN_ST_ESTABLISHED: - /* - * Send a disconnect message - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_PEND_1); - - result = sdp_send_ctrl_disconnect(conn); - if (result < 0) { - sdp_dbg_warn(conn, - "Error <%d> send disconnect request", - result); - goto error; - } - - break; - default: - sdp_dbg_warn(conn, "Incorrect state for disconnect"); - result = -EBADE; - goto error; - } - - return 0; -error: - /* - * abortive close. - */ - result = sdp_wall_send_abort(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result); - - return result; -} - -/* - * sdp_wall_send_closing - callback to confirm a passive close - */ -int sdp_wall_send_closing(struct sdp_opt *conn) -{ - int result; - - sdp_dbg_ctrl(conn, "Close confirm. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port); - /* - * close buffered data transmission space - */ - conn->send_buf = 0; - /* - * Generate a response Disconnect message, and mark self as such. - */ - switch (conn->state) { - case SDP_CONN_ST_DIS_RECV_1: - /* - * Change state, and send a disconnect request - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_PEND_2); - - result = sdp_send_ctrl_disconnect(conn); - if (result < 0) { - sdp_dbg_warn(conn, - "Error <%d> send disconnect request", - result); - goto error; - } - break; - default: - sdp_dbg_warn(conn, "Incorrect state for disconnect"); - result = -EBADE; - goto error; - } - - return 0; -error: - /* - * abortive close. - */ - result = sdp_wall_send_abort(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result); - - return result; -} - -/* - * sdp_wall_send_abort - callback to accept an active abort - */ -int sdp_wall_send_abort(struct sdp_opt *conn) -{ - int result = 0; - - sdp_dbg_ctrl(conn, "Abort send. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port); - - switch (conn->state) { - case SDP_CONN_ST_DIS_SENT_1: /* IBTA v1.1 spec A4.5.3.2 */ - /* - * clear the pending control buffer. - */ - sdp_desc_q_clear(&conn->send_ctrl); - /* - * fall through - */ - case SDP_CONN_ST_DIS_SEND_1: /* don't touch control queue, diconnect - message may still be queued. */ - sdp_desc_q_clear(&conn->send_queue); - /* - * post abort - */ - result = sdp_send_ctrl_abort(conn); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> send abort request", - result); - goto error; - } - /* - * state change, no more STRM interface calls from here on out. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM); - - break; - case SDP_CONN_ST_CLOSED: - case SDP_CONN_ST_ERROR_CM: - case SDP_CONN_ST_TIME_WAIT_2: - /* - * no more CM callbacks will be made. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_CLOSED); - - break; - case SDP_CONN_ST_TIME_WAIT_1: - /* - * waiting for idle. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_2); - - break; - case SDP_CONN_ST_REQ_PATH: - /* - * cancel address resolution? - * - * instead of canceling allow the path completion to - * determine that the socket has moved to an error state. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_CLOSED); - break; - case SDP_CONN_ST_REQ_SENT: - case SDP_CONN_ST_REP_SENT: - /* - * outstanding CM request. Mark it in error, and CM - * completion needs to complete the closing. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM); - - break; - case SDP_CONN_ST_ESTABLISHED: /* protocol errors and others. */ - case SDP_CONN_ST_DIS_RECV_1: /* Abort msg after disconnect msg */ - case SDP_CONN_ST_DIS_PEND_1: /* data recv after send/recv close */ - case SDP_CONN_ST_ERROR_CQ: /* begin CM level disconnect */ - /* - * post abort - */ - goto abort; - default: - /* - * post abort - */ - sdp_dbg_warn(conn, "Unexpected connection state for abort"); - goto error; - } - - return 0; -abort: -error: - /* - * abortive close. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM); - - result = ib_send_cm_dreq(conn->cm_id, NULL, 0); - if (result) - sdp_dbg_warn(conn, "Error <%d> CM disconnect send", result); - - return 0; -} - -/* - * SDP INET (public functions) - */ - -/* - * sdp_wall_recv_close - callback to accept an active close - */ -int sdp_wall_recv_close(struct sdp_opt *conn) -{ - sdp_dbg_ctrl(conn, "Close recv. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port); - /* - * wake the closing connection, and change state to close received. - */ - switch (conn->istate) { - case SDP_SOCK_ST_ACCEPTING: - case SDP_SOCK_ST_ACCEPTED: - /* - * shift to close. - */ - conn->istate = SDP_SOCK_ST_CLOSE; - conn->shutdown |= RCV_SHUTDOWN; - - break; - default: - conn->istate = SDP_SOCK_ST_CLOSE; - conn->shutdown |= RCV_SHUTDOWN; - /* - * cancel all outstanding read AIO's since there will be - * no more data from the peer. - */ - sdp_iocb_q_cancel_all_read(conn, 0); - /* - * async notification. POLL_HUP on full duplex close only. - */ - sdp_inet_wake_generic(conn->sk); - sk_wake_async(conn->sk, 1, POLL_IN); - - break; - } - - return 0; -} - -/* - * sdp_wall_recv_closing - callback for a close confirmation - */ -int sdp_wall_recv_closing(struct sdp_opt *conn) -{ - sdp_dbg_ctrl(conn, "Closing recv. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port); - /* - * change state, finalize the close, and wake the closer. - */ - SDP_EXPECT((conn->istate == SDP_SOCK_ST_DISCONNECT)); - - conn->send_buf = 0; - conn->istate = SDP_SOCK_ST_CLOSED; - conn->shutdown |= RCV_SHUTDOWN; - /* - * cancel all outstanding read AIO's since there will be no more data - * from the peer. - */ - sdp_iocb_q_cancel_all_read(conn, 0); - /* - * async notification. POLL_HUP on full duplex close only. - */ - sdp_inet_wake_generic(conn->sk); - sk_wake_async(conn->sk, 1, POLL_HUP); - - return 0; -} - -/* - * sdp_wall_recv_abort - abortive close notification - */ -int sdp_wall_recv_abort(struct sdp_opt *conn) -{ - sdp_dbg_ctrl(conn, "Abort recv. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port); - /* - * wake the connection in case it's doing anything, and mark it as - * closed. Leave data in their buffers so the user can blead it - * dry. - */ - switch (conn->istate) { - case SDP_SOCK_ST_CONNECT: - case SDP_SOCK_ST_ACCEPTING: - case SDP_SOCK_ST_ACCEPTED: - SDP_CONN_SET_ERR(conn, ECONNREFUSED); - break; - case SDP_SOCK_ST_CLOSE: - SDP_CONN_SET_ERR(conn, EPIPE); - break; - default: - SDP_CONN_SET_ERR(conn, ECONNRESET); - break; - } - - conn->send_buf = 0; - conn->istate = SDP_SOCK_ST_ERROR; - conn->shutdown = SHUTDOWN_MASK; - /* - * cancel all outstanding IOCBs - */ - sdp_iocb_q_cancel_all(conn, -ECONNRESET); - - sdp_inet_wake_error(conn->sk); - - return 0; -} - -/* - * sdp_wall_recv_drop - drop SDP protocol reference to socket - */ -void sdp_wall_recv_drop(struct sdp_opt *conn) -{ - int result; - - sdp_dbg_ctrl(conn, "Drop. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port); - - conn->send_buf = 0; - - switch (conn->istate) { - case SDP_SOCK_ST_ACCEPTING: - /* - * pull the listen sockets accept queue. - */ - result = sdp_inet_accept_q_remove(conn); - if (result < 0) - sdp_dbg_warn(conn, - "Error <%d> removing from accept queue.", - result); - - conn->istate = SDP_SOCK_ST_CLOSED; - - break; - case SDP_SOCK_ST_CLOSING: - conn->istate = SDP_SOCK_ST_CLOSED; - sdp_inet_wake_generic(conn->sk); - - break; - default: - /* - * wake the connection in case it's doing anything, and mark - * it as closed. Leave data in their buffers so the user can - * bleed it dry. - */ - SDP_CONN_SET_ERR(conn, ECONNRESET); - conn->istate = SDP_SOCK_ST_ERROR; - /* - * cancel all outstanding IOCBs - */ - sdp_iocb_q_cancel_all(conn, -ECONNRESET); - - sdp_inet_wake_error(conn->sk); - - break; - } -} - -/* - * SDP common (public functions) - */ - -/* - * sdp_wall_abort - intiate socket dropping - */ -int sdp_wall_abort(struct sdp_opt *conn) -{ - int result; - - /* - * notify both halves of the wall that the connection is being aborted. - */ - result = sdp_wall_recv_abort(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> recving abort request", result); - /* if */ - result = sdp_wall_send_abort(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> sending abort request", result); - - return 0; -} Index: infiniband/ulp/sdp/sdp_recv.c =================================================================== --- infiniband/ulp/sdp/sdp_recv.c (revision 2564) +++ infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -828,8 +828,7 @@ /* * abort connection (send reset) */ - result = sdp_wall_abort(conn); - SDP_EXPECT(result >= 0); + sdp_conn_abort(conn); /* * drop packet */ @@ -1200,8 +1199,8 @@ sdp_conn_lock(conn); - if (conn->istate == SDP_SOCK_ST_LISTEN || - conn->istate == SDP_SOCK_ST_CLOSED) { + if (conn->state == SDP_CONN_ST_LISTEN || + conn->state == SDP_CONN_ST_CLOSED) { result = -ENOTCONN; goto done; } @@ -1393,7 +1392,7 @@ break; } - if (conn->istate == SDP_SOCK_ST_ERROR) { + if (conn->state == SDP_CONN_ST_ERROR) { result = -EPROTO; /* error should always be set, but just in case */ break; Index: infiniband/ulp/sdp/sdp_conn.h =================================================================== --- infiniband/ulp/sdp/sdp_conn.h (revision 2564) +++ infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -48,18 +48,15 @@ */ #define SDP_CONN_COMM_ID_NULL 0xFFFFFFFF /* when no id is available */ -#define SDP_CONN_F_SRC_CANCEL_L 0x0001 /* source cancel was issued */ -#define SDP_CONN_F_SRC_CANCEL_R 0x0002 /* source cancel was received */ -#define SDP_CONN_F_SRC_CANCEL_C 0x0004 /* source data cancelled */ -#define SDP_CONN_F_SNK_CANCEL 0x0008 /* sink cancel issued */ -#define SDP_CONN_F_DIS_HOLD 0x0010 /* Hold pending disconnects. */ -#define SDP_CONN_F_OOB_SEND 0x0020 /* OOB notification pending. */ -#define SDP_CONN_F_RECV_CQ_PEND 0x0040 /* recv CQ event is pending */ -#define SDP_CONN_F_SEND_CQ_PEND 0x0080 /* send CQ event is pending */ -#define SDP_CONN_F_DEAD 0xFFFF /* connection has been deleted */ +#define SDP_CONN_F_SRC_CANCEL_L 0x01 /* source cancel was issued */ +#define SDP_CONN_F_SRC_CANCEL_R 0x02 /* source cancel was received */ +#define SDP_CONN_F_SRC_CANCEL_C 0x04 /* source data cancelled */ +#define SDP_CONN_F_SNK_CANCEL 0x08 /* sink cancel issued */ +#define SDP_CONN_F_DIS_HOLD 0x10 /* Hold pending disconnects. */ +#define SDP_CONN_F_DIS_PEND 0x20 /* disconnect pending. */ +#define SDP_CONN_F_OOB_SEND 0x40 /* OOB notification pending. */ +#define SDP_CONN_F_DEAD 0xFF /* connection has been deleted */ -#define SDP_CONN_F_MASK_EVENT (SDP_CONN_F_RECV_CQ_PEND| \ - SDP_CONN_F_SEND_CQ_PEND) /* * SDP states. */ @@ -71,77 +68,42 @@ }; /* - * First two bytes are the primary state values. Third byte is a bit - * field used for different mask operations, defined below. Fourth - * byte are the mib values for the different states. + * First two bytes are the primary state values. Third and fourth + * byte are a bit field used for different mask operations, defined + * below. */ #define SDP_CONN_ST_LISTEN 0x0100 /* listening */ -#define SDP_CONN_ST_ESTABLISHED 0x11B1 /* connected */ +#define SDP_CONN_ST_ESTABLISHED 0x1171 /* connected */ #define SDP_CONN_ST_REQ_PATH 0x2100 /* active open, path record lookup */ #define SDP_CONN_ST_REQ_SENT 0x2200 /* active open, Hello msg sent */ -#define SDP_CONN_ST_REQ_RECV 0x2300 /* passive open, Hello msg recv'd */ -#define SDP_CONN_ST_REP_SENT 0x2480 /* passive open, Hello ack sent */ -#define SDP_CONN_ST_REP_RECV 0x2500 /* active open, Hello ack recv'd */ -#define SDP_CONN_ST_RTU_SENT 0x2680 /* active open, Hello ack, acked */ +#define SDP_CONN_ST_REQ_RECV 0x2340 /* passive open, Hello msg recv'd */ +#define SDP_CONN_ST_REP_RECV 0x2440 /* active open, Hello ack recv'd */ -#define SDP_CONN_ST_DIS_RECV_1 0x31B1 /* recv disconnect, passive close */ -#define SDP_CONN_ST_DIS_PEND_1 0x32F1 /* pending disconn, active close */ -#define SDP_CONN_ST_DIS_SEND_1 0x33B1 /* send disconnect, active close */ -#define SDP_CONN_ST_DIS_SENT_1 0x34A1 /* disconnect sent, active close */ -#define SDP_CONN_ST_DIS_PEND_R 0x35F1 /* disconnect recv, active close */ -#define SDP_CONN_ST_DIS_RECV_R 0x36B1 /* disconnect recv, active close */ -#define SDP_CONN_ST_DIS_PEND_2 0x37F1 /* pending disconn, passive close */ -#define SDP_CONN_ST_DIS_SEND_2 0x38B1 /* send disconnect, passive close */ -#define SDP_CONN_ST_DIS_SENT_2 0x3901 /* disconnect sent, passive close */ -#define SDP_CONN_ST_TIME_WAIT_1 0x3A01 /* IB/gateway disconnect */ -#define SDP_CONN_ST_TIME_WAIT_2 0x3B01 /* waiting for idle close */ +#define SDP_CONN_ST_DIS_RECV_1 0x4171 /* recv disconnect, passive close */ +#define SDP_CONN_ST_DIS_SEND_1 0x4271 /* send disconnect, active close */ +#define SDP_CONN_ST_DIS_SENT_1 0x4361 /* disconnect sent, active close */ +#define SDP_CONN_ST_DIS_RECV_R 0x4471 /* disconnect recv, active close */ +#define SDP_CONN_ST_DIS_SEND_2 0x4571 /* send disconnect, passive close */ +#define SDP_CONN_ST_TIME_WAIT_1 0x4701 /* IB/gateway disconnect */ +#define SDP_CONN_ST_TIME_WAIT_2 0x4801 /* waiting for idle close */ -#define SDP_CONN_ST_ERROR_CM 0xFB03 /* CM error, waiting on gateway */ -#define SDP_CONN_ST_ERROR_CQ 0xFC03 /* gateway error, waiting on CM */ -#define SDP_CONN_ST_ERROR_STRM 0xFD01 /* gateway error, waiting on CM */ -#define SDP_CONN_ST_CLOSED 0xFE03 /* not connected */ -#define SDP_CONN_ST_INVALID 0xFF03 /* not connected */ +#define SDP_CONN_ST_CLOSED 0x8E01 /* not connected */ +#define SDP_CONN_ST_ERROR 0x8D01 /* not connected */ +#define SDP_CONN_ST_INVALID 0x8F01 /* not connected */ /* * states masks for SDP */ +#define SDP_ST_MASK_CONNECT 0x2000 /* connection establishment states */ +#define SDP_ST_MASK_CLOSED 0x8000 /* all connection closed states. */ #define SDP_ST_MASK_EVENTS 0x0001 /* event processing is allowed. */ -#define SDP_ST_MASK_ERROR 0x0002 /* protocol in error state. */ #define SDP_ST_MASK_SEND_OK 0x0010 /* posting data for send */ #define SDP_ST_MASK_CTRL_OK 0x0020 /* posting control for send */ -#define SDP_ST_MASK_DIS_PEND 0x0040 /* disconnect transmission pending. */ -#define SDP_ST_MASK_RCV_POST 0x0080 /* posting IB recv's is allowed. */ +#define SDP_ST_MASK_RCV_POST 0x0040 /* posting IB recv's is allowed. */ /* - * transition one of the disconnect pending states to disconnect send - */ -#define SDP_ST_PEND_2_SEND(conn) \ - ((conn)->state = ((conn)->state + 0x0100) & ~SDP_ST_MASK_DIS_PEND) - -/* - * internal connection structure - */ -#define SDP_SOCK_ST_CLOSED 0x0101 -#define SDP_SOCK_ST_CONNECT 0x0204 -#define SDP_SOCK_ST_ACCEPTING 0x0300 -#define SDP_SOCK_ST_ACCEPTED 0x0404 /* writable ? */ -#define SDP_SOCK_ST_ESTABLISHED 0x050E -#define SDP_SOCK_ST_DISCONNECT 0x0602 /* active close request */ -#define SDP_SOCK_ST_CLOSE 0x070C /* passive close request */ -#define SDP_SOCK_ST_CLOSING 0x0800 -#define SDP_SOCK_ST_LISTEN 0x0901 -#define SDP_SOCK_ST_ERROR 0xFF01 - -/* - * state masks. - */ -#define SDP_ST_MASK_CLOSED 0x0001 /* socket is not referenced by the GW. */ -#define SDP_ST_MASK_SEND 0x0004 /* valid state for API send */ -#define SDP_ST_MASK_OPEN 0x0008 /* send window is valid (writeable) */ - -/* * event dispatch table */ #define SDP_MSG_EVENT_TABLE_SIZE 0x20 @@ -205,11 +167,14 @@ * connection lock */ struct sdp_conn_lock { - __u32 users; + u16 users; + u16 event; spinlock_t slock; wait_queue_head_t waitq; }; +#define SDP_LOCK_F_RECV_CQ 0x01 /* recv CQ event is pending */ +#define SDP_LOCK_F_SEND_CQ 0x02 /* send CQ event is pending */ /* * SDP Connection structure. */ @@ -245,7 +210,6 @@ s32 rwin_max; /* maximum recveive window size */ u16 state; /* connection state */ - u16 istate; /* inet connection state */ u8 flags; /* single bit flags. */ u8 shutdown; /* shutdown flag */ @@ -494,8 +458,7 @@ unsigned long flags; spin_lock_irqsave(&conn->lock.slock, flags); - if ((conn->flags & SDP_CONN_F_MASK_EVENT) && - (SDP_ST_MASK_EVENTS & conn->state)) { + if (conn->lock.event && (conn->state & SDP_ST_MASK_EVENTS)) { sdp_conn_internal_unlock(conn); } Index: infiniband/ulp/sdp/sdp_pass.c =================================================================== --- infiniband/ulp/sdp/sdp_pass.c (revision 2564) +++ infiniband/ulp/sdp/sdp_pass.c (working copy) @@ -35,38 +35,6 @@ #include "sdp_main.h" /* - * handle incoming passive connection error. (REJ) - */ -void sdp_cm_pass_error(struct sdp_opt *conn, int error) -{ - int result; - - sdp_dbg_ctrl(conn, - "passive error. src <%08x:%04x> dst <%08x:%04x> <%d>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port, error); - /* - * the connection has failed, move to error, and notify anyone - * waiting of the state change. remove connection from listen - * queue if possible - */ - result = sdp_inet_accept_q_remove(conn); - if (!result) - sdp_conn_put(conn); /* AcceptQueue INET reference */ - - SDP_CONN_SET_ERR(conn, error); - conn->istate = SDP_SOCK_ST_ERROR; - conn->shutdown = SHUTDOWN_MASK; - conn->send_buf = 0; - - if (conn->sk->sk_socket) - conn->sk->sk_socket->state = SS_UNCONNECTED; - - sdp_iocb_q_cancel_all(conn, (0 - error)); - sdp_inet_wake_error(conn->sk); -} - -/* * handle incoming passive connection establishment. (RTU) */ int sdp_cm_pass_establish(struct sdp_opt *conn) @@ -82,8 +50,6 @@ * free hello ack message */ (void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post)); - - SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED); qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL); if (!qp_attr) { @@ -109,15 +75,6 @@ conn->send_buf = SDP_INET_SEND_SIZE; - switch (conn->istate) { - case SDP_SOCK_ST_ACCEPTING: - conn->istate = SDP_SOCK_ST_ACCEPTED; - break; - case SDP_SOCK_ST_ACCEPTED: - conn->istate = SDP_SOCK_ST_ESTABLISHED; - break; - } - result = sdp_send_flush(conn); if (0 > result) { sdp_dbg_warn(conn, "Error <%d> flushing sends.", result); @@ -137,9 +94,7 @@ error: kfree(qp_attr); done: - sdp_cm_pass_error(conn, (0 - result)); - SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM); - + sdp_conn_inet_error(conn, -result); return result; } /* @@ -247,10 +202,6 @@ goto error; } /* - * set connection state to allow recv buffer posts. - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_REP_SENT); - /* * Post receive buffers for this connection */ result = sdp_recv_flush(conn); @@ -342,8 +293,6 @@ * insert accept socket into listen sockets list. * TODO: needs to be a FIFO not a LIFO, as is now. */ - conn->istate = SDP_SOCK_ST_ACCEPTING; - inet_sk(sk)->num = conn->src_port; inet_sk(sk)->sport = htons(conn->src_port); inet_sk(sk)->rcv_saddr = htonl(conn->src_addr); Index: infiniband/ulp/sdp/sdp_sent.c =================================================================== --- infiniband/ulp/sdp/sdp_sent.c (revision 2564) +++ infiniband/ulp/sdp/sdp_sent.c (working copy) @@ -58,21 +58,6 @@ break; case SDP_CONN_ST_DIS_SEND_2: - /* - * passive disconnect message send - */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_DIS_SENT_2); - /* - * Begin IB/CM disconnect - */ - result = ib_send_cm_dreq(conn->cm_id, NULL, 0); - if (result) { - sdp_dbg_warn(conn, "Error <%d> posting CM disconnect", - result); - goto error; - } - - break; case SDP_CONN_ST_DIS_RECV_R: /* * simultaneous disconnect. Received a disconnect, after we @@ -81,15 +66,6 @@ */ SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_1); /* - * acknowledge disconnect to framework - */ - result = sdp_wall_recv_closing(conn); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> closing connection.", - result); - goto error; - } - /* * Begin IB/CM disconnect */ result = ib_send_cm_dreq(conn->cm_id, NULL, 0); @@ -102,10 +78,12 @@ result); break; + case SDP_CONN_ST_ERROR: + break; default: - sdp_warn("Disconnect sent, unexpected state. <%d> <%04x:%04d>", - conn->hashent, conn->istate, conn->state); - result = -EFAULT; + sdp_warn("Disconnect sent, unexpected state. <%d> <%04x>", + conn->hashent, conn->state); + result = -EPROTO; goto error; break; @@ -124,8 +102,7 @@ * The gateway interface should be in error state, initiate CM * disconnect. */ - SDP_EXPECT((conn->state == SDP_CONN_ST_ERROR_STRM)); - SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_STRM); + SDP_CONN_ST_SET(conn, SDP_CONN_ST_CLOSED); result = ib_send_cm_dreq(conn->cm_id, NULL, 0); if (result) Index: infiniband/ulp/sdp/sdp_event.c =================================================================== --- infiniband/ulp/sdp/sdp_event.c (revision 2564) +++ infiniband/ulp/sdp/sdp_event.c (working copy) @@ -45,9 +45,9 @@ { int result = 0; - if (SDP_ST_MASK_ERROR & conn->state) { + if (SDP_ST_MASK_CLOSED & conn->state) { /* - * Ignore events in error state, connection is being + * Ignore events in closed state, connection is being * terminated, connection cleanup will take care of freeing * posted buffers. */ @@ -112,12 +112,7 @@ /* * abort. */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR_CQ); - - result = sdp_wall_abort(conn); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> during abort", result); - + sdp_conn_abort(conn); return -EFAULT; } @@ -132,7 +127,7 @@ { s32 hashent = (unsigned long)arg; struct sdp_opt *conn; - s32 result; + u16 event; unsigned long flags; sdp_dbg_data(NULL, "CQ event. hashent <%d>", hashent); @@ -157,41 +152,42 @@ * has been made, the act of unlocking the connection will * drain the CQ. */ - if (!(SDP_ST_MASK_EVENTS & conn->state)) { - /* - * passive and active connect respectively - */ - if (conn->state == SDP_CONN_ST_REP_SENT || - conn->state == SDP_CONN_ST_RTU_SENT) { - result = ib_cm_establish(conn->cm_id); - SDP_EXPECT(result >= 0); - } - else - sdp_dbg_warn(conn, "Unexpected locked state."); + event = (cq == conn->recv_cq) ? SDP_LOCK_F_RECV_CQ:SDP_LOCK_F_SEND_CQ; - conn->flags |= SDP_CONN_F_MASK_EVENT; - goto unlock; + if (!conn->lock.users) { + if (!(SDP_ST_MASK_EVENTS & conn->state)) { + /* + * passive and active connect respectively + */ + if (conn->state == SDP_CONN_ST_REQ_RECV) + (void)ib_cm_establish(conn->cm_id); + else + sdp_dbg_warn(conn, "Unexpected event state."); + } + else { + /* + * dispatch CQ completions. + */ + (void)sdp_conn_cq_drain(cq, conn); + event = 0; + } } + /* + * Mark the event which was received, for the unlock code to + * process at a later time. + */ + conn->lock.event |= event; - if (!conn->lock.users) - /* - * dispatch CQ completions. - */ - (void)sdp_conn_cq_drain(cq, conn); - else - /* - * Mark the event which was received, for the unlock code to - * process at a later time. - */ - conn->flags |= ((cq == conn->recv_cq) ? - SDP_CONN_F_RECV_CQ_PEND : - SDP_CONN_F_SEND_CQ_PEND); - -unlock: SDP_CONN_UNLOCK_IRQ(conn, flags); sdp_conn_put(conn); } +static void sdp_cm_to_error(struct sdp_opt *conn) +{ + sdp_conn_inet_error(conn, ECONNRESET); + conn->cm_id = NULL; + sdp_conn_put(conn); /* CM reference */ +} /* * Connection establishment IB/CM callback functions */ @@ -206,31 +202,27 @@ */ switch (conn->state) { case SDP_CONN_ST_REQ_SENT: - conn->state = SDP_CONN_ST_CLOSED; - sdp_cm_actv_error(conn, ECONNREFUSED); break; - case SDP_CONN_ST_REP_SENT: - conn->state = SDP_CONN_ST_CLOSED; - - sdp_cm_pass_error(conn, ECONNREFUSED); + case SDP_CONN_ST_REQ_RECV: + case SDP_CONN_ST_ESTABLISHED: + sdp_conn_inet_error(conn, ECONNREFUSED); break; case SDP_CONN_ST_TIME_WAIT_1: - case SDP_CONN_ST_ESTABLISHED: sdp_dbg_warn(conn, "Unexpected connection state"); /* * fall through */ - case SDP_CONN_ST_ERROR_STRM: + case SDP_CONN_ST_CLOSED: + case SDP_CONN_ST_ERROR: case SDP_CONN_ST_TIME_WAIT_2: /* * Connection is finally dead. Drop the CM reference */ - sdp_wall_recv_drop(conn); break; default: - sdp_warn("Unknown conn state. conn <%d> state <%04x:%04x>", - conn->hashent, conn->istate, conn->state); + sdp_warn("Unknown conn state. conn <%d> state <%04x>", + conn->hashent, conn->state); break; } @@ -254,14 +246,20 @@ * check state */ switch (conn->state) { - case SDP_CONN_ST_REP_SENT: + case SDP_CONN_ST_REQ_RECV: + SDP_CONN_ST_SET(conn, SDP_CONN_ST_ESTABLISHED); + case SDP_CONN_ST_DIS_SEND_1: + case SDP_CONN_ST_DIS_RECV_R: + case SDP_CONN_ST_DIS_SEND_2: + /* bring QP to established state, and flush queues. */ result = sdp_cm_pass_establish(conn); if (!result) break; /* * on error fall through to disconnect */ - case SDP_CONN_ST_ERROR_STRM: + case SDP_CONN_ST_CLOSED: + case SDP_CONN_ST_ERROR: /* * Begin abortive disconnect. * Leave state unchanged, time_wait and idle will handle the @@ -269,29 +267,18 @@ */ result = ib_send_cm_dreq(conn->cm_id, NULL, 0); if (result) { - sdp_dbg_warn(NULL, "Error <%d> sending CM DREQ", + sdp_dbg_warn(conn, "Error <%d> sending CM DREQ", result); - goto done; - } - - break; - case SDP_CONN_ST_DIS_PEND_1: - /* active open, and active close */ - case SDP_CONN_ST_DIS_PEND_R: - /* active open, and active close, confirm */ - case SDP_CONN_ST_DIS_PEND_2: - result = sdp_send_flush(conn); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> flushing receives.", - result); goto error; } break; + case SDP_CONN_ST_ESTABLISHED: + break; default: - sdp_warn("Unexpected conn state. conn <%d> state <%04x:%04x>", - conn->hashent, conn->istate, conn->state); + sdp_warn("Unexpected conn state. conn <%d> state <%04x>", + conn->hashent, conn->state); result = -EINVAL; goto error; break; @@ -299,10 +286,7 @@ return 0; error: - sdp_wall_recv_drop(conn); -done: - conn->cm_id = NULL; - sdp_conn_put(conn); /* CM reference */ + sdp_cm_to_error(conn); return result; } @@ -319,11 +303,7 @@ result = ib_send_cm_drep(conn->cm_id, NULL, 0); if (result) { sdp_dbg_warn(conn, "Error <%d> sending CM DREP", result); - - sdp_wall_recv_drop(conn); - - conn->cm_id = NULL; - sdp_conn_put(conn); /* CM reference */ + sdp_cm_to_error(conn); } return result; @@ -351,7 +331,8 @@ * process state changes. */ switch (conn->state) { - case SDP_CONN_ST_ERROR_STRM: + case SDP_CONN_ST_CLOSED: + case SDP_CONN_ST_ERROR: /* * error on stream interface, no more call to/from those * interfaces. @@ -359,7 +340,6 @@ break; case SDP_CONN_ST_DIS_RECV_R: case SDP_CONN_ST_DIS_SEND_2: - case SDP_CONN_ST_DIS_SENT_2: case SDP_CONN_ST_TIME_WAIT_1: /* * SDP disconnect messages have been exchanged, and @@ -367,7 +347,6 @@ */ SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_2); break; - case SDP_CONN_ST_DIS_PEND_1: case SDP_CONN_ST_DIS_SEND_1: case SDP_CONN_ST_DIS_SENT_1: case SDP_CONN_ST_DIS_RECV_1: @@ -380,29 +359,18 @@ * Change state, so we only need to wait for the abort * callback, and idle. Call the abort callback. */ - SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_1); + SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_2); - result = sdp_wall_abort(conn); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> during abort", result); - goto error; - } - + sdp_conn_abort(conn); break; default: - sdp_warn("Unexpected conn state. conn <%d> state <%04x:%04x>", - conn->hashent, conn->istate, conn->state); + sdp_warn("Unexpected conn state. conn <%d> state <%04x>", + conn->hashent, conn->state); + sdp_cm_to_error(conn); result = -EINVAL; - goto error; break; } - return 0; -error: - sdp_wall_recv_drop(conn); - - conn->cm_id = NULL; - sdp_conn_put(conn); /* CM reference */ return result; } @@ -460,7 +428,7 @@ * if a socket was found, release the lock, and put the reference. */ if (conn) { - if (result < 0) { + if (result < 0 && event->event != IB_CM_TIMEWAIT_EXIT) { sdp_dbg_warn(conn, "CM state <%d> event <%d> error <%d>", cm_id->state, event->event, result); From jlentini at netapp.com Tue Jun 7 13:42:46 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 7 Jun 2005 16:42:46 -0400 (EDT) Subject: [openib-general] RE: [patch][kdapl] enable kdapltest -T P In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C24@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C24@mtvex01.mtv.mtl.com> Message-ID: On Fri, 3 Jun 2005, Itamar Rabenstein wrote: > */ >> >> >> Could we initialize the ia_attr.max_evd_qlen value correctly in the >> provider? >> >>> > > Current openib gen2 code is not reporting the max cq size and i dont think > that we should put > a fix number . > if we want to get the number we need Roland to fill this number in mthca but > as Roland said before > "what real App will meed this number?" I see the initialization code you are refering to in mthca_query_device of mthca_provider.c. I think a "real app" would use this number in exactly the same way that dapltest's performance subtest uses it: pipeline_length = min(max_cqe, max_qp_wr) Why would a limit like the one above be unnecessary? >> >> Again, why not initialize these correclty in the provider? >> > > again the same as max_evd_qlen value > >> >> Good catch. 255 was not a good value to have been using. >> >> Could we use pt_ptr->provider_attr.optimal_buffer_alignment instead? > > if it is 256 that it is O.k but i dont think we should fix it the current > code is O.k for me. > >>> Index: test/dapl_bpool.c >>> =================================================================== >>> --- test/dapl_bpool.c (revision 2522) >>> +++ test/dapl_bpool.c (working copy) >>> @@ -388,7 +388,7 @@ >>> u64 >>> DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index) >>> { >>> - return ( bpool_ptr->reg_addr + index * bpool_ptr->seg_size ); >>> + return (virt_to_phys(DT_Bpool_GetBuffer(bpool_ptr,index))); >> >> Was this also a problem in the transaction test? >> > > Yes but in order to fall on the bug you need to alloc a very small buffer > (like 12 byte) > and this is only in -T P . So is it only in the performance test or would it occur in the transaction test if I specified a small buffer? Why do we need to translate the virtual address to a physical address in all cases? Will this interact properly with the transaction test's -M (memory type) option? From jlentini at netapp.com Tue Jun 7 14:16:01 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 7 Jun 2005 17:16:01 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: remove use of HANDLEs In-Reply-To: <20050604115201.GA18264@lst.de> References: <1117837957.16881.21.camel@duffman> <20050604115201.GA18264@lst.de> Message-ID: On Sat, 4 Jun 2005, Christoph Hellwig wrote: hch> On Fri, Jun 03, 2005 at 03:32:37PM -0700, Tom Duffy wrote: hch> > This patch is HUGE. Sorry. Not much I could do to split it. hch> > hch> > It makes a superclass for all the dat objects, and subclasses hch> > dat-provider (dapl) off of it. No more opaque handles. No more magic hch> > checking. I actually found a few bugs from the compiler because of it hch> > (more than reported to openib). hch> > hch> > The dapl_header struct is renamed dapl_common. Most of the _handle hch> > variables are renamed removing _handle. It removes the hch> > get_handle_type() function since this is no longer used. Also, it hch> > unimplements [get|set]_consumer_context (for now) since nobody was using hch> > it. DAT_BAD_HANDLE() goes as it becomes a simple NULL check. hch> hch> And event that should go away in most cases. In the kernel we should't hch> check for NULL pointers all over the place, if client programmers don't hch> stick to the APIs they deserve to get panics. Any objections to removing the NULL pointer checks of API parameters? From jlentini at netapp.com Tue Jun 7 14:19:54 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 7 Jun 2005 17:19:54 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [TRIVIAL] kdapl: Eliminate some dapl_evd_connection_callb.c compile warnings In-Reply-To: <1117927726.4493.5025.camel@hal.voltaire.com> References: <1117927726.4493.5025.camel@hal.voltaire.com> Message-ID: This was corrected when the EVD related functions were merged into one function. On Sat, 4 Jun 2005, Hal Rosenstock wrote: halr> kdapl: Eliminate some dapl_evd_connection_callb.c compile warnings halr> halr> Index: dapl_evd_connection_callb.c halr> =================================================================== halr> --- dapl_evd_connection_callb.c (revision 2546) halr> +++ dapl_evd_connection_callb.c (working copy) halr> @@ -31,6 +31,7 @@ halr> halr> #include "dapl.h" halr> #include "dapl_evd_util.h" halr> +#include "dapl_openib_util.h" halr> halr> /* halr> * Connection callback function for ACTIVE connection requests; callbacks halr> halr> From tduffy at sun.com Tue Jun 7 14:27:19 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 07 Jun 2005 14:27:19 -0700 Subject: [openib-general] [PATCH][SDP] state transition simplification. In-Reply-To: <20050607133616.A27322@topspin.com> References: <20050607133616.A27322@topspin.com> Message-ID: <1118179639.15780.9.camel@duffman> On Tue, 2005-06-07 at 13:36 -0700, Libor Michalek wrote: > This patch simplifies the state transitions for connection management > within SDP. I tested this patch on top of 2.6.12-rc6 (with my sk changes as well) and it worked fine with a standard ttcp.aio test. [root at flopteron2 ~]# uname -a Linux flopteron2.sfbay.sun.com 2.6.12-rc6openib #24 SMP Tue Jun 7 11:05:30 PDT 2005 x86_64 x86_64 x86_64 GNU/Linux [root at flopteron2 ~]# ./ttcp -t -l 65536 -n 100000 -a 20 192.168.0.233 ttcp-t: buflen = 65536 nbuf = 100000 align = 16384/0 port = 5001 192.168.0.233 ttcp-t: socket ttcp-t: connect ttcp-t: 6553600000 bytes in 6.97 real seconds = 7173.59 Mbit/sec +++ ttcp-t: 199934 I/O calls, usec/call = 34.86, calls/sec = 28684.89 ttcp-t: user: 57992 sys: 1407786 total: 1465778 real: 6970012 (microseconds) -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Tue Jun 7 14:29:21 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 07 Jun 2005 14:29:21 -0700 Subject: [openib-general] [PATCH] kDAPL: remove use of HANDLEs In-Reply-To: References: <1117837957.16881.21.camel@duffman> <20050604115201.GA18264@lst.de> Message-ID: <1118179761.15780.11.camel@duffman> On Tue, 2005-06-07 at 17:16 -0400, James Lentini wrote: > Any objections to removing the NULL pointer checks of API parameters? Fine by me. Although BUG_ON() may be more appropriate. How about after applying my patch? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mshefty at ichips.intel.com Tue Jun 7 14:47:36 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 07 Jun 2005 14:47:36 -0700 Subject: [openib-general] CM MRA handling In-Reply-To: <000301c56b9b$74a1f080$9c5aa8c0@infiniconsys.com> References: <000301c56b9b$74a1f080$9c5aa8c0@infiniconsys.com> Message-ID: <42A615F8.1040708@ichips.intel.com> Fab Tillier wrote: > Sean, > > When setting the delay for a received MRA, the CM doesn't add a packet lifetime > as documented in the spec (12.7.32 - Service Timeout). > > If you do end up adding the packet lifetime, you'll need to revisit the > cm_convert_to_ms to prevent overflow of the return value. Overflow happens at > IB timeout values > 39. Okay - I'll add this to my todo list. - Sean From sean.hefty at intel.com Tue Jun 7 14:58:38 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 7 Jun 2005 14:58:38 -0700 Subject: [openib-general] [PATCH] [RMPP] fix handling of duplicate RMPP ACK Message-ID: This patch fixes the handling of a duplicate RMPP ACK. Hal, can you test this and see if it corrects the problems that you were seeing on the SA side? Signed-off-by: Sean Hefty Index: mad_rmpp.c =================================================================== --- mad_rmpp.c (revision 2563) +++ mad_rmpp.c (working copy) @@ -657,7 +657,8 @@ static void process_rmpp_ack(struct ib_m ib_reset_mad_timeout(mad_send_wr, mad_send_wr-> send_wr.wr.ud.timeout_ms); } else if (mad_send_wr->refcount == 1 && - mad_send_wr->seg_num < mad_send_wr->newwin) { + mad_send_wr->seg_num < mad_send_wr->newwin && + mad_send_wr->seg_num <= mad_send_wr->total_seg) { /* Send failure will just result in a timeout/retry */ ret = send_next_seg(mad_send_wr); if (ret) From halr at voltaire.com Tue Jun 7 15:09:34 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 18:09:34 -0400 Subject: [openib-general] Re: [PATCH] [RMPP] fix handling of duplicate RMPP ACK In-Reply-To: References: Message-ID: <1118181866.4477.20.camel@localhost.localdomain> On Tue, 2005-06-07 at 17:58, Sean Hefty wrote: > This patch fixes the handling of a duplicate RMPP ACK. > > Hal, can you test this and see if it corrects the problems that you were > seeing on the SA side? Yes, but it will need to be another time. The IB analyzer was on the fritz today and I'm no longer in a place where I can do this. -- Hal From halr at voltaire.com Tue Jun 7 15:12:49 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 18:12:49 -0400 Subject: [openib-general] [PATCH] OpenSM: Set retries to 0 if RMPP Message-ID: <1118182327.4477.30.camel@localhost.localdomain> OpenSM: Set retries to 0 if RMPP (in vendor layer) With this change, able to interoperate with Solaris 10 SA client (which uses RMPP) with OpenSM Signed-off-by: Hal Rosenstock Index: osm_vendor_ibumad.c =================================================================== --- osm_vendor_ibumad.c (revision 2567) +++ osm_vendor_ibumad.c (working copy) @@ -828,6 +828,7 @@ ib_mad_t* const p_mad = osm_madw_get_mad_ptr( p_madw ); ib_sa_mad_t* const p_sa = (ib_sa_mad_t *)p_mad; int ret = -1; + int is_rmpp = 0; #ifndef VENDOR_RMPP_SUPPORT uint32_t paylen = 0; #endif @@ -860,7 +861,8 @@ p_sa->rmpp_flags = 0; p_sa->rmpp_status = 0; #ifdef VENDOR_RMPP_SUPPORT - } + } else + is_rmpp = 1; osm_log( p_vend->p_log, OSM_LOG_ERROR, "osm_vendor_send: RMPP %d length %d\n", ib_rmpp_is_flag_set((ib_rmpp_mad_t *)p_sa, IB_RMPP_FLAG_ACTIVE), p_madw->mad_size); #else } else { @@ -882,7 +884,7 @@ if ((ret = umad_send(p_bind->port_id, p_bind->agent_id, p_vw->umad, p_madw->mad_size, resp_expected ? p_vend->timeout : 0, - p_vend->max_retries)) < 0) { + is_rmpp ? 0 : p_vend->max_retries)) < 0) { osm_log( p_vend->p_log, OSM_LOG_ERROR, "osm_vendor_send: Send failed %d (%m).\n", ret); (*p_bind->send_err_callback)(p_bind->client_context, p_madw); /* cb frees madw */ if (resp_expected) From mshefty at ichips.intel.com Tue Jun 7 15:19:46 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 07 Jun 2005 15:19:46 -0700 Subject: [openib-general] Re: [PATCH] [RMPP] fix handling of duplicate RMPP ACK In-Reply-To: <1118181866.4477.20.camel@localhost.localdomain> References: <1118181866.4477.20.camel@localhost.localdomain> Message-ID: <42A61D82.5020404@ichips.intel.com> Hal Rosenstock wrote: > On Tue, 2005-06-07 at 17:58, Sean Hefty wrote: > >>This patch fixes the handling of a duplicate RMPP ACK. >> >>Hal, can you test this and see if it corrects the problems that you were >>seeing on the SA side? > > > Yes, but it will need to be another time. The IB analyzer was on the > fritz today and I'm no longer in a place where I can do this. No problem. I'm pretty sure that this will fix the issue that you were seeing. I just can't easily reproduce the problem, since it requires resending a duplicate ACK (after all segments have been sent and with a window size greater than the total number of segments to be sent...). - Sean From tduffy at sun.com Tue Jun 7 15:21:09 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 07 Jun 2005 15:21:09 -0700 Subject: [openib-general] [PATCH] OpenSM: Set retries to 0 if RMPP In-Reply-To: <1118182327.4477.30.camel@localhost.localdomain> References: <1118182327.4477.30.camel@localhost.localdomain> Message-ID: <1118182869.22316.8.camel@duffman> On Tue, 2005-06-07 at 18:12 -0400, Hal Rosenstock wrote: > With this change, able to interoperate with Solaris 10 SA client (which > uses RMPP) with OpenSM Cool. I will give it a try. What do you mean by interop? Were you able to bring up IPoIB and ping? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Tue Jun 7 15:19:55 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 18:19:55 -0400 Subject: [openib-general] [PATCH] OpenSM: Set retries to 0 if RMPP In-Reply-To: <1118182869.22316.8.camel@duffman> References: <1118182327.4477.30.camel@localhost.localdomain> <1118182869.22316.8.camel@duffman> Message-ID: <1118182795.4477.40.camel@localhost.localdomain> On Tue, 2005-06-07 at 18:21, Tom Duffy wrote: > On Tue, 2005-06-07 at 18:12 -0400, Hal Rosenstock wrote: > > With this change, able to interoperate with Solaris 10 SA client (which > > uses RMPP) with OpenSM > > Cool. I will give it a try. What do you mean by interop? Were you > able to bring up IPoIB and ping? and snoop as well :-) -- Hal From tduffy at sun.com Tue Jun 7 15:37:26 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 07 Jun 2005 15:37:26 -0700 Subject: [openib-general] [PATCH] OpenSM: Set retries to 0 if RMPP In-Reply-To: <1118182869.22316.8.camel@duffman> References: <1118182327.4477.30.camel@localhost.localdomain> <1118182869.22316.8.camel@duffman> Message-ID: <1118183846.22316.19.camel@duffman> On Tue, 2005-06-07 at 15:21 -0700, Tom Duffy wrote: > On Tue, 2005-06-07 at 18:12 -0400, Hal Rosenstock wrote: > > With this change, able to interoperate with Solaris 10 SA client (which > > uses RMPP) with OpenSM > > Cool. I will give it a try. What do you mean by interop? Were you > able to bring up IPoIB and ping? Nevermind. Just tried it with Solaris Nevada build 16 and it works too! Albeit with some minor caveats. For instance, a broadcast ping from Linux does not see the Solaris node: This is a linux system: [root at sins-stinger-10 ~]# ping 192.168.0.0 -b WARNING: pinging broadcast address PING 192.168.0.0 (192.168.0.0) 56(84) bytes of data. 64 bytes from 192.168.0.233: icmp_seq=0 ttl=64 time=0.054 ms 64 bytes from 192.168.0.26: icmp_seq=0 ttl=64 time=0.159 ms (DUP!) 64 bytes from 192.168.0.233: icmp_seq=1 ttl=64 time=0.016 ms 64 bytes from 192.168.0.26: icmp_seq=1 ttl=64 time=0.114 ms (DUP!) From the Solaris system: [root at nisus ~]# ping -s 192.168.0.0 PING 192.168.0.0: 56 data bytes 64 bytes from 192.168.0.78: icmp_seq=0. time=0.501 ms 64 bytes from 192.168.0.233: icmp_seq=0. time=5.95 ms 64 bytes from 192.168.0.26: icmp_seq=0. time=10.4 ms Solaris 10 was unhappy: [root at blissom ~]# ifconfig ibd0 plumb ifconfig: SIOCSLIFNAME for ip: ibd0: no such interface Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_ATTACH_REQ failed: DL_BADPPA Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_BIND_REQ failed: DL_OUTSTATE Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_PHYS_ADDR_REQ failed: DL_OUTSTATE Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_UNBIND_REQ failed: DL_OUTSTATE Which made my Nevada box unhappy: < from dmesg > interface ibd0 to 192.168.0.78 broken: in=3 ierr=0 out=5 oerr=2 -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From libor at topspin.com Tue Jun 7 15:56:46 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 7 Jun 2005 15:56:46 -0700 Subject: [openib-general] Re: UCM accesses internal cm_id state In-Reply-To: ; from sean.hefty@intel.com on Thu, Jun 02, 2005 at 12:55:34PM -0700 References: Message-ID: <20050607155646.B27322@topspin.com> On Thu, Jun 02, 2005 at 12:55:34PM -0700, Sean Hefty wrote: > The ucm code in ib_ucm_event_handler() reads the cm_id->state information in > response to an event occurrence. The cm_id state can change dynamically in > response to an event, and in general should not be accessed. (I.e. the > current cm_id state may not match up with the event being reported.) > > Access to the state requires coordination with the CM code itself, so trying > to rely on it to drive external states will eventually lead to problems. > Clients of the CM should either maintain their own state information or > perform operations based on the occurrence of the reported event. > > I had originally exposed the state only for debugging purposes. A comment > was added that it should only be used for internal/debug use, but I will > make a note to remove the state from the externally visible cm_id structure. Makes sense to me. This patch removes the state variable from both kernel and userspace UCM code. I'll also work on a patch for SDP when I get a chance, but it should be fairly straigh forward. -Libor Index: linux-kernel/infiniband/include/ib_user_cm.h =================================================================== --- linux-kernel/infiniband/include/ib_user_cm.h (revision 2564) +++ linux-kernel/infiniband/include/ib_user_cm.h (working copy) @@ -90,8 +90,6 @@ struct ib_ucm_attr_id_resp { __u64 service_id; __u64 service_mask; - __u32 state; - __u32 lap_state; __u32 local_id; __u32 remote_id; }; @@ -310,7 +308,6 @@ struct ib_ucm_event_resp { __u32 id; - __u32 state; __u32 event; __u32 present; union { Index: linux-kernel/infiniband/core/ucm.c =================================================================== --- linux-kernel/infiniband/core/ucm.c (revision 2564) +++ linux-kernel/infiniband/core/ucm.c (working copy) @@ -58,7 +58,7 @@ static struct semaphore ctx_id_mutex; static struct idr ctx_id_table; -static int ctx_id_rover = 0; +static int ctx_id_rover = 0; static struct ib_ucm_context *ib_ucm_ctx_get(int id) { @@ -428,7 +428,6 @@ uevent->resp.id = id; uevent->resp.event = event->event; - uevent->resp.state = cm_id->state; result = ib_ucm_event_process(event, uevent); if (result) @@ -652,8 +651,6 @@ resp.service_id = ctx->cm_id->service_id; resp.service_mask = ctx->cm_id->service_mask; - resp.state = ctx->cm_id->state; - resp.lap_state = ctx->cm_id->lap_state; resp.local_id = ctx->cm_id->local_id; resp.remote_id = ctx->cm_id->remote_id; Index: userspace/libibcm/include/infiniband/cm.h =================================================================== --- userspace/libibcm/include/infiniband/cm.h (revision 2521) +++ userspace/libibcm/include/infiniband/cm.h (working copy) @@ -39,33 +39,6 @@ #include #include -enum ib_cm_state { - IB_CM_IDLE, - IB_CM_LISTEN, - IB_CM_REQ_SENT, - IB_CM_REQ_RCVD, - IB_CM_MRA_REQ_SENT, - IB_CM_MRA_REQ_RCVD, - IB_CM_REP_SENT, - IB_CM_REP_RCVD, - IB_CM_MRA_REP_SENT, - IB_CM_MRA_REP_RCVD, - IB_CM_ESTABLISHED, - IB_CM_DREQ_SENT, - IB_CM_DREQ_RCVD, - IB_CM_TIMEWAIT, - IB_CM_SIDR_REQ_SENT, - IB_CM_SIDR_REQ_RCVD -}; - -enum ib_cm_lap_state { - IB_CM_LAP_IDLE, - IB_CM_LAP_SENT, - IB_CM_LAP_RCVD, - IB_CM_MRA_LAP_SENT, - IB_CM_MRA_LAP_RCVD, -}; - enum ib_cm_event_type { IB_CM_REQ_ERROR, IB_CM_REQ_RECEIVED, @@ -240,7 +213,6 @@ struct ib_cm_event { uint32_t cm_id; enum ib_cm_event_type event; - enum ib_cm_state state; union { struct ib_cm_req_event_param req_rcvd; struct ib_cm_rep_event_param rep_rcvd; @@ -313,8 +285,6 @@ struct ib_cm_attr_param { uint64_t service_id; uint64_t service_mask; - enum ib_cm_state state; - enum ib_cm_lap_state lap_state; uint32_t local_id; uint32_t remote_id; }; Index: userspace/libibcm/include/infiniband/cm_abi.h =================================================================== --- userspace/libibcm/include/infiniband/cm_abi.h (revision 2521) +++ userspace/libibcm/include/infiniband/cm_abi.h (working copy) @@ -94,8 +94,6 @@ struct cm_abi_attr_id_resp { __u64 service_id; __u64 service_mask; - __u32 state; - __u32 lap_state; __u32 local_id; __u32 remote_id; }; @@ -314,7 +312,6 @@ struct cm_abi_event_resp { __u32 id; - __u32 state; __u32 event; __u32 present; union { Index: userspace/libibcm/src/cm.c =================================================================== --- userspace/libibcm/src/cm.c (revision 2521) +++ userspace/libibcm/src/cm.c (working copy) @@ -188,8 +188,6 @@ param->service_id = resp->service_id; param->service_mask = resp->service_mask; - param->state = resp->state; - param->lap_state = resp->lap_state; param->local_id = resp->local_id; param->remote_id = resp->remote_id; @@ -773,7 +771,6 @@ evt->cm_id = resp->id; evt->event = resp->event; - evt->state = resp->state; if (resp->present & CM_ABI_PRES_PRIMARY) { Index: userspace/libibcm/examples/simple.c =================================================================== --- userspace/libibcm/examples/simple.c (revision 2521) +++ userspace/libibcm/examples/simple.c (working copy) @@ -168,11 +168,10 @@ goto done; } - printf("CM ID <%d> Event <%d> State <%d>\n", - event->cm_id, event->event, event->state); + printf("CM ID <%d> Event <%d>\n", event->cm_id, event->event); - switch (event->state) { - case IB_CM_REQ_RCVD: + switch (event->event) { + case IB_CM_REQ_RECEIVED: result = ib_cm_destroy_id(cm_id); if (result < 0) { @@ -205,7 +204,7 @@ } break; - case IB_CM_REP_RCVD: + case IB_CM_REP_RECEIVED: result = ib_cm_send_rtu(cm_id, NULL, 0); if (result < 0) { @@ -215,7 +214,7 @@ } break; - case IB_CM_ESTABLISHED: + case IB_CM_RTU_RECEIVED: result = ib_cm_send_dreq(cm_id, NULL, 0); if (result < 0) { @@ -225,7 +224,7 @@ } break; - case IB_CM_DREQ_RCVD: + case IB_CM_DREQ_RECEIVED: result = ib_cm_send_drep(cm_id, NULL, 0); if (result < 0) { @@ -235,15 +234,14 @@ } break; - case IB_CM_TIMEWAIT: + case IB_CM_DREP_RECEIVED: break; - case IB_CM_IDLE: + case IB_CM_TIMEWAIT_EXIT: status = 1; break; default: status = EINVAL; - printf("Unhandled state <%d:%d>\n", - event->state, event->event); + printf("Unhandled event <%d>\n", event->event); break; } From halr at voltaire.com Tue Jun 7 16:27:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 19:27:51 -0400 Subject: [openib-general] [PATCH] OpenSM: Set retries to 0 if RMPP In-Reply-To: <1118183846.22316.19.camel@duffman> References: <1118182327.4477.30.camel@localhost.localdomain> <1118182869.22316.8.camel@duffman> <1118183846.22316.19.camel@duffman> Message-ID: <1118186678.4477.239.camel@localhost.localdomain> On Tue, 2005-06-07 at 18:37, Tom Duffy wrote: > On Tue, 2005-06-07 at 15:21 -0700, Tom Duffy wrote: > > On Tue, 2005-06-07 at 18:12 -0400, Hal Rosenstock wrote: > > > With this change, able to interoperate with Solaris 10 SA client (which > > > uses RMPP) with OpenSM > > > > Cool. I will give it a try. What do you mean by interop? Were you > > able to bring up IPoIB and ping? > > Nevermind. Just tried it with Solaris Nevada build 16 and it works > too! > > Albeit with some minor caveats. For instance, a broadcast ping from > Linux does not see the Solaris node: > > This is a linux system: > > [root at sins-stinger-10 ~]# ping 192.168.0.0 -b > WARNING: pinging broadcast address > PING 192.168.0.0 (192.168.0.0) 56(84) bytes of data. > 64 bytes from 192.168.0.233: icmp_seq=0 ttl=64 time=0.054 ms > 64 bytes from 192.168.0.26: icmp_seq=0 ttl=64 time=0.159 ms (DUP!) > 64 bytes from 192.168.0.233: icmp_seq=1 ttl=64 time=0.016 ms > 64 bytes from 192.168.0.26: icmp_seq=1 ttl=64 time=0.114 ms (DUP!) I didn't try broadcast but the limited broadcast (192.168.0.255_ and it worked. Does it for you ? I will try this next time. Not sure this is an OpenSM issue. > From the Solaris system: > > [root at nisus ~]# ping -s 192.168.0.0 > PING 192.168.0.0: 56 data bytes > 64 bytes from 192.168.0.78: icmp_seq=0. time=0.501 ms > 64 bytes from 192.168.0.233: icmp_seq=0. time=5.95 ms > 64 bytes from 192.168.0.26: icmp_seq=0. time=10.4 ms > > Solaris 10 was unhappy: > > [root at blissom ~]# ifconfig ibd0 plumb > ifconfig: SIOCSLIFNAME for ip: ibd0: no such interface > Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_ATTACH_REQ failed: DL_BADPPA > Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_BIND_REQ failed: DL_OUTSTATE > Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_PHYS_ADDR_REQ failed: DL_OUTSTATE > Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_UNBIND_REQ failed: DL_OUTSTATE > > Which made my Nevada box unhappy: > > < from dmesg > > interface ibd0 to 192.168.0.78 broken: in=3 ierr=0 out=5 oerr=2 It needs a little more work. Unfortunately I could not capture IB traces today for analysis. Was this with Sean's latest RMPP change or not ? Thanks. -- Hal From kjreilly at us.ibm.com Tue Jun 7 16:35:44 2005 From: kjreilly at us.ibm.com (Kevin Reilly) Date: Tue, 7 Jun 2005 19:35:44 -0400 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: <52vf4rl1tv.fsf@topspin.com> Message-ID: Thanks for the ibv_get_devices()!!! I will go take a look at it. Below you mentioned "spec", is there a spec for the gen2 ibverbs? Or are you refering to the VAPI spec or the IBTA spec chapter? I've been assuming the code was the spec and that writing a specification was still something that still needed to be "contributed". Kevin J. Reilly STSM, HPC Architecture -Federation/HPS Chief Engineer -HPC interconnect architect (office) 845-433-7976 (tieline) 8-293-7976 Roland Dreier To Kevin Reilly/Poughkeepsie/IBM at IBMUS 06/06/2005 03:31 cc PM openib-general at openib.org Subject Re: [openib-general] What is the correct method of determining the number of ports Kevin> Roland, Do you know other things that are missing that you Kevin> can rattle off the top of your head? Else we keep looking Kevin> for what we need then ask questions if we don't find them. There are a couple of fairly large features that have not been implemented at all, even in the kernel. For example, shared receive queues (SRQ) and memory windows. Other than that I think libibverbs support is pretty complete right now. Of course the best thing is for application developers to ask for the features they need. That way I don't waste time implementing something that happens to be in the spec but no one cares about. - R. From halr at voltaire.com Tue Jun 7 16:34:09 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 19:34:09 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> Message-ID: <1118187249.4477.278.camel@localhost.localdomain> On Tue, 2005-05-31 at 14:17, James Lentini wrote: > Here's the specification's exact description: > > timeout: Duration of time, in microseconds, that a consumer waits for > connection establishment. The value of DAT_TIMEOUT_INFINITE > represents no timeout, indefinite wait. Values must be > positive. What is the purpose of an infinite timeout (other than the obvious) ? The quit test uses this feature. Not sure if other tests do as well. What happens if the REQ is lost ? Why would someone want an infinite timeout ? -- Hal From halr at voltaire.com Tue Jun 7 17:08:24 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 20:08:24 -0400 Subject: [openib-general] RMPP and timeouts/retries Message-ID: <1118189304.4477.387.camel@localhost.localdomain> Hi Sean, I'm having some trouble using timeouts and retries with RMPP. The issue I see is that what constitutes a response is different than normal MADs for RMPP. With the current "definition" of response, is the timeout/retry only usable on the SA client side where the normal definition is more closely followed ? If so, is there still an issue with SA GetTable as the initial request is not RMPP ? On the SA side, there does not appear to be a way to use this feature. Is that correct ? Thanks for your insights. -- Hal From iod00d at hp.com Tue Jun 7 17:24:26 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 7 Jun 2005 17:24:26 -0700 Subject: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050604213422.GA23153@mellanox.co.il> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> <20050601232123.GN25321@esmail.cup.hp.com> <20050604213422.GA23153@mellanox.co.il> Message-ID: <20050608002426.GI15489@esmail.cup.hp.com> On Sun, Jun 05, 2005 at 12:34:22AM +0300, Michael S. Tsirkin wrote: > Weird. What does this error mean? Could you try debugging it? I tried it again today and it worked. I updated to and am running SVN 2525. grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw -C 10.0.0.51 local address: LID 0x06, QPN 0xa0406, PSN 0x8fdf4e RKey 0x3c0434 VAddr 0x6000000000014001 remote address: LID 0x04, QPN 0xc0406, PSN 0x4e0787, RKey 0x480434 VAddr 0x6000000000014001 Bandwidth peak (#1 to #999): 3.11048e-10 MByte/cycles Bandwidth average: 3.10762e-10 MByte/cycles The numbers (MByte/cycles) don't mean anything to me. Can we report this as cycles/MB (or KB) instead? (aka Service Demand) The other metric I care about is MB/s. thanks, grant From hch at lst.de Tue Jun 7 17:25:53 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 8 Jun 2005 02:25:53 +0200 Subject: [openib-general] [PATCH] kDAPL: remove use of HANDLEs In-Reply-To: <1118179761.15780.11.camel@duffman> References: <1117837957.16881.21.camel@duffman> <20050604115201.GA18264@lst.de> <1118179761.15780.11.camel@duffman> Message-ID: <20050608002553.GA5297@lst.de> On Tue, Jun 07, 2005 at 02:29:21PM -0700, Tom Duffy wrote: > On Tue, 2005-06-07 at 17:16 -0400, James Lentini wrote: > > Any objections to removing the NULL pointer checks of API parameters? > > Fine by me. Although BUG_ON() may be more appropriate. a BUG_ON doesn're provide any more info than a NULL-pointer dereference. > How about after applying my patch? absolutely. let's get your patch in ASAP, it's a huge improvement. From halr at voltaire.com Tue Jun 7 17:53:45 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 Jun 2005 20:53:45 -0400 Subject: [openib-general] RMPP and timeouts/retries In-Reply-To: <1118189304.4477.387.camel@localhost.localdomain> References: <1118189304.4477.387.camel@localhost.localdomain> Message-ID: <1118192024.4477.525.camel@localhost.localdomain> On Tue, 2005-06-07 at 20:08, Hal Rosenstock wrote: > Hi Sean, > > I'm having some trouble using timeouts and retries with RMPP. The issue > I see is that what constitutes a response is different than normal MADs > for RMPP. With the current "definition" of response, is the > timeout/retry only usable on the SA client side where the normal > definition is more closely followed ? If so, is there still an issue > with SA GetTable as the initial request is not RMPP ? Is the response not at the RMPP level ? So a response would be for dual ended RMPP ? > On the SA side, there does not appear to be a way to use this feature. > Is that correct ? > > Thanks for your insights. > > -- Hal > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From roland at topspin.com Tue Jun 7 19:46:05 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 07 Jun 2005 19:46:05 -0700 Subject: [openib-general] What is the correct method of determining the number of ports In-Reply-To: (Kevin Reilly's message of "Tue, 7 Jun 2005 19:35:44 -0400") References: Message-ID: <5264wpftw2.fsf@topspin.com> Kevin> Thanks for the ibv_get_devices()!!! I will go take a look Kevin> at it. Below you mentioned "spec", is there a spec for the Kevin> gen2 ibverbs? Or are you refering to the VAPI spec or the Kevin> IBTA spec chapter? By "spec" I meant the IBA spec -- however at this point there's not much left of chapter 11 that someone hasn't asked for at some point. - R. From sean.hefty at intel.com Tue Jun 7 22:04:41 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 7 Jun 2005 22:04:41 -0700 Subject: [openib-general] RMPP and timeouts/retries In-Reply-To: <1118192024.4477.525.camel@localhost.localdomain> Message-ID: >> I'm having some trouble using timeouts and retries with RMPP. The issue >> I see is that what constitutes a response is different than normal MADs >> for RMPP. With the current "definition" of response, is the >> timeout/retry only usable on the SA client side where the normal >> definition is more closely followed ? If so, is there still an issue >> with SA GetTable as the initial request is not RMPP ? The SA client should set the timeout > 0 to indicate that a response MAD is expected. It can set retries, but isn't required to do so. >Is the response not at the RMPP level ? So a response would be for dual >ended RMPP ? Request/response is separate from RMPP. Request/response is defined at the message level from the viewpoint of the user of the MAD layer. (I.e. is the response bit set in the MAD header, regardless of the size of the MAD.) Setting timeout > 0 when sending indicates that a MAD should be received with the response bit set that is in reply to the request. If timeout = 0 but RMPP is marked active, the MAD will still invoke RMPP and not complete until all segments have been ACKed. (I.e. the completion of an RMPP send indicates that the MAD was received.) Retries is a little more complex. It indicates the number of times to send a request in hopes of receiving a response (if one is expected) or an ACK (if using RMPP). >> On the SA side, there does not appear to be a way to use this feature. >> Is that correct ? For the SA side responding to a request, you would want to set retries > 1, but timeout = 0. Timeout of 0 indicates that no response is expected, since the SA is sending the response. Having a retry count would allow a retransmission if an ACK were lost. You should be able to mix RMPP and non-RMPP MADs without restriction: non-RMPP request - RMPP response, RMPP request - non-RMPP response, RMPP request and response, non-RMPP request and response. If one of these conditions doesn't work, then there's a bug in the code. I'm pretty sure that I tested all of these combinations, but that doesn't mean that it won't hit a bug talking with another RMPP implementation. - Sean From eitan at mellanox.co.il Tue Jun 7 22:15:27 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 8 Jun 2005 08:15:27 +0300 Subject: [openib-general] RMPP and timeouts/retries Message-ID: <506C3D7B14CDD411A52C00025558DED607C302CF@mtlex01.yok.mtl.com> Hi Sean, This seems very nice. Thank you. The only thing I dislike in this API is the "implicit" behavior. I would prefer a clear flag saying - is_response_expected. Rather than the timeout to flag this. But this is a matter of taste not functionality. Eitan Zahavi > -----Original Message----- > From: Sean Hefty [mailto:sean.hefty at intel.com] > Sent: Wednesday, June 08, 2005 8:05 AM > To: 'Hal Rosenstock'; Sean Hefty > Cc: openib-general at openib.org > Subject: RE: [openib-general] RMPP and timeouts/retries > > >> I'm having some trouble using timeouts and retries with RMPP. The issue > >> I see is that what constitutes a response is different than normal MADs > >> for RMPP. With the current "definition" of response, is the > >> timeout/retry only usable on the SA client side where the normal > >> definition is more closely followed ? If so, is there still an issue > >> with SA GetTable as the initial request is not RMPP ? > > The SA client should set the timeout > 0 to indicate that a response MAD is > expected. It can set retries, but isn't required to do so. > > >Is the response not at the RMPP level ? So a response would be for dual > >ended RMPP ? > > Request/response is separate from RMPP. Request/response is defined at the > message level from the viewpoint of the user of the MAD layer. (I.e. is the > response bit set in the MAD header, regardless of the size of the MAD.) > > Setting timeout > 0 when sending indicates that a MAD should be received with > the response bit set that is in reply to the request. If timeout = 0 but RMPP > is marked active, the MAD will still invoke RMPP and not complete until all > segments have been ACKed. (I.e. the completion of an RMPP send indicates that > the MAD was received.) > > Retries is a little more complex. It indicates the number of times to send a > request in hopes of receiving a response (if one is expected) or an ACK (if > using RMPP). > > >> On the SA side, there does not appear to be a way to use this feature. > >> Is that correct ? > > For the SA side responding to a request, you would want to set retries > 1, but > timeout = 0. Timeout of 0 indicates that no response is expected, since the SA > is sending the response. Having a retry count would allow a retransmission if > an ACK were lost. > > You should be able to mix RMPP and non-RMPP MADs without restriction: non- > RMPP > request - RMPP response, RMPP request - non-RMPP response, RMPP request and > response, non-RMPP request and response. If one of these conditions doesn't > work, then there's a bug in the code. I'm pretty sure that I tested all of > these combinations, but that doesn't mean that it won't hit a bug talking with > another RMPP implementation. > > - Sean > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hefty at intel.com Tue Jun 7 22:25:22 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 7 Jun 2005 22:25:22 -0700 Subject: [openib-general] RMPP and timeouts/retries In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C302CF@mtlex01.yok.mtl.com> Message-ID: This seems very nice. Thank you. The only thing I dislike in this API is the "implicit" behavior. I would prefer a clear flag saying - is_response_expected. Rather than the timeout to flag this. But this is a matter of taste not functionality. I agree that a clear flag would be useful. Some of the behavior with retries/RMPP is the result of slowly adding in the features and not wanting to change the API/existing implementations. Timeout could be used for RMPP MADs that do not generate a response, so adding the flag at some point makes sense. - Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Tue Jun 7 22:50:30 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 8 Jun 2005 08:50:30 +0300 Subject: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050608002426.GI15489@esmail.cup.hp.com> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> <20050601232123.GN25321@esmail.cup.hp.com> <20050604213422.GA23153@mellanox.co.il> <20050608002426.GI15489@esmail.cup.hp.com> Message-ID: <20050608055030.GA7809@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest > > On Sun, Jun 05, 2005 at 12:34:22AM +0300, Michael S. Tsirkin wrote: > > Weird. What does this error mean? Could you try debugging it? > > I tried it again today and it worked. > I updated to and am running SVN 2525. > > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw -C 10.0.0.51 > local address: LID 0x06, QPN 0xa0406, PSN 0x8fdf4e RKey 0x3c0434 VAddr 0x6000000000014001 > remote address: LID 0x04, QPN 0xc0406, PSN 0x4e0787, RKey 0x480434 VAddr 0x6000000000014001 > Bandwidth peak (#1 to #999): 3.11048e-10 MByte/cycles > Bandwidth average: 3.10762e-10 MByte/cycles > > The numbers (MByte/cycles) don't mean anything to me. > Can we report this as cycles/MB (or KB) instead? (aka Service Demand) Sure. Patch? > The other metric I care about is MB/s. > > thanks, > grant > Dont pass -C then. -- MST From itamar at mellanox.co.il Wed Jun 8 03:00:16 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 8 Jun 2005 13:00:16 +0300 Subject: [openib-general] RE: [patch][kdapl] enable kdapltest -T P Message-ID: <91DB792C7985D411BEC300B40080D29CC35C4D@mtvex01.mtv.mtl.com> > > Current openib gen2 code is not reporting the max cq size > and i dont think > > that we should put > > a fix number . > > if we want to get the number we need Roland to fill this > number in mthca but > > as Roland said before > > "what real App will meed this number?" > > I see the initialization code you are refering to in > mthca_query_device of mthca_provider.c. > > I think a "real app" would use this number in exactly the same way > that dapltest's performance subtest uses it: > > pipeline_length = min(max_cqe, max_qp_wr) > > Why would a limit like the one above be unnecessary? you can try to printk the value and you will see it is not initialized. it is not enough to assign value to the output attr struct if this value is not initialized (;-) i dont mind if Roland will implement it but for now it is not implemented. Please ci the patch for now in order to enable kdapltest -T P and if Roland will implement it then unremark the lines. > >> > >> Was this also a problem in the transaction test? > >> > > > > Yes but in order to fall on the bug you need to alloc a > very small buffer > > (like 12 byte) > > and this is only in -T P . > > So is it only in the performance test or would it occur in the > transaction test if I specified a small buffer? It will fail for every small buffer (include in -T T ) > > Why do we need to translate the virtual address to a physical address > in all cases? Will this interact properly with the transaction test's > -M (memory type) option? Current openib gen2 register only physical memory. the code will work also for any -M option . -Itamar From itamar at mellanox.co.il Wed Jun 8 03:02:00 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 8 Jun 2005 13:02:00 +0300 Subject: [openib-general] [PATCH] [CM] add OS administered service IDs Message-ID: <91DB792C7985D411BEC300B40080D29CC35C4E@mtvex01.mtv.mtl.com> Thanks Sean, I will update the kdapl code soon. -Itamar > -----Original Message----- > From: Sean Hefty [mailto:sean.hefty at intel.com] > Sent: Tuesday, June 07, 2005 8:50 PM > To: openib-general at openib.org > Subject: [openib-general] [PATCH] [CM] add OS administered service IDs > > > The following patch adds support for OS administered service IDs. > > Signed-off-by: Sean Hefty > > > Index: include/ib_cm.h > =================================================================== > --- include/ib_cm.h (revision 2563) > +++ include/ib_cm.h (working copy) > @@ -311,16 +311,20 @@ struct ib_cm_id *ib_create_cm_id(ib_cm_h > */ > void ib_destroy_cm_id(struct ib_cm_id *cm_id); > > +#define IB_CM_ASSIGN_SERVICE_ID > __constant_cpu_to_be64(0x0200000000000000ULL) > + > /** > * ib_cm_listen - Initiates listening on the specified service ID for > * connection and service ID resolution requests. > * @cm_id: Connection identifier associated with the listen request. > * @service_id: Service identifier matched against incoming > connection > * and service ID resolution requests. The service ID > should be specified > - * network-byte order. > + * network-byte order. If set to IB_CM_ASSIGN_SERVICE_ID, > the CM will > + * assign a service ID to the caller. > * @service_mask: Mask applied to service ID used to listen across a > * range of service IDs. If set to 0, the service ID is matched > - * exactly. > + * exactly. This parameter is ignored if %service_id is set to > + * IB_CM_ASSIGN_SERVICE_ID. > */ > int ib_cm_listen(struct ib_cm_id *cm_id, > u64 service_id, > Index: core/cm.c > =================================================================== > --- core/cm.c (revision 2563) > +++ core/cm.c (working copy) > @@ -64,6 +64,7 @@ static struct ib_cm { > struct list_head device_list; > rwlock_t device_lock; > struct rb_root listen_service_table; > + u64 listen_service_id; > /* struct rb_root peer_service_table; todo: fix peer to peer */ > struct rb_root remote_qp_table; > struct rb_root remote_id_table; > @@ -718,14 +719,23 @@ int ib_cm_listen(struct ib_cm_id *cm_id, > unsigned long flags; > int ret = 0; > > + if ((service_id & IB_CM_ASSIGN_SERVICE_ID) == > IB_CM_ASSIGN_SERVICE_ID && > + (service_id != IB_CM_ASSIGN_SERVICE_ID)) > + return -EINVAL; > + > cm_id_priv = container_of(cm_id, struct cm_id_private, id); > BUG_ON(cm_id->state != IB_CM_IDLE); > > cm_id->state = IB_CM_LISTEN; > - cm_id->service_id = service_id; > - cm_id->service_mask = service_mask ? service_mask : ~0ULL; > > spin_lock_irqsave(&cm.lock, flags); > + if (service_id == IB_CM_ASSIGN_SERVICE_ID) { > + cm_id->service_id = > __cpu_to_be64(cm.listen_service_id++); > + cm_id->service_mask = ~0ULL; > + } else { > + cm_id->service_id = service_id; > + cm_id->service_mask = service_mask ? > service_mask : ~0ULL; > + } > cur_cm_id_priv = cm_insert_listen(cm_id_priv); > spin_unlock_irqrestore(&cm.lock, flags); > > @@ -3251,6 +3261,7 @@ static int __init ib_cm_init(void) > rwlock_init(&cm.device_lock); > spin_lock_init(&cm.lock); > cm.listen_service_table = RB_ROOT; > + cm.listen_service_id = > __constant_be64_to_cpu(IB_CM_ASSIGN_SERVICE_ID); > cm.remote_id_table = RB_ROOT; > cm.remote_qp_table = RB_ROOT; > cm.remote_sidr_table = RB_ROOT; > > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From halr at voltaire.com Wed Jun 8 03:58:12 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 08 Jun 2005 06:58:12 -0400 Subject: [openib-general] RMPP and timeouts/retries In-Reply-To: References: Message-ID: <1118228291.4477.594.camel@localhost.localdomain> On Wed, 2005-06-08 at 01:04, Sean Hefty wrote: > >> I'm having some trouble using timeouts and retries with RMPP. The issue > >> I see is that what constitutes a response is different than normal MADs > >> for RMPP. With the current "definition" of response, is the > >> timeout/retry only usable on the SA client side where the normal > >> definition is more closely followed ? If so, is there still an issue > >> with SA GetTable as the initial request is not RMPP ? > > The SA client should set the timeout > 0 to indicate that a response MAD is > expected. It can set retries, but isn't required to do so. OK. > >Is the response not at the RMPP level ? So a response would be for dual > >ended RMPP ? > > Request/response is separate from RMPP. Request/response is defined at the > message level from the viewpoint of the user of the MAD layer. (I.e. is the > response bit set in the MAD header, regardless of the size of the MAD.) Understood. It's method based (response bit and trap repress). > Setting timeout > 0 when sending indicates that a MAD should be received with > the response bit set that is in reply to the request. If timeout = 0 but RMPP > is marked active, the MAD will still invoke RMPP and not complete until all > segments have been ACKed. (I.e. the completion of an RMPP send indicates that > the MAD was received.) If timeout = 0, does RMPP still indicate a timeout if an ACK is lost and the other end does not reACK ? > Retries is a little more complex. It indicates the number of times to send a > request in hopes of receiving a response (if one is expected) or an ACK (if > using RMPP). Any effect on ABORT and/or STOP ? > >> On the SA side, there does not appear to be a way to use this feature. > >> Is that correct ? > > For the SA side responding to a request, you would want to set retries > 1, but > timeout = 0. Timeout of 0 indicates that no response is expected, since the SA > is sending the response. Having a retry count would allow a retransmission if > an ACK were lost. I will try that. Using retry and timeout on the SA side is definitely problematic. Right now it is set to no retries and no timeout on the SA side. I will change and retest. > You should be able to mix RMPP and non-RMPP MADs without restriction: non-RMPP > request - RMPP response, RMPP request - non-RMPP response, RMPP request and > response, non-RMPP request and response. If one of these conditions doesn't > work, then there's a bug in the code. I'm pretty sure that I tested all of > these combinations, but that doesn't mean that it won't hit a bug talking with > another RMPP implementation. The main new one right now is non RMPP request - RMPP response. SA does not currently support any of the dual sided RMPP methods (so there is no RMPP request - RMPP response). (Non RMPP request - non RMPP response is what is mainly used). I'm not sure the other combination has a use. Thanks. -- Hal From halr at voltaire.com Wed Jun 8 04:12:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 08 Jun 2005 07:12:02 -0400 Subject: [openib-general] RMPP and timeouts/retries In-Reply-To: <1118228291.4477.594.camel@localhost.localdomain> References: <1118228291.4477.594.camel@localhost.localdomain> Message-ID: <1118229122.4477.610.camel@localhost.localdomain> On Wed, 2005-06-08 at 06:58, Hal Rosenstock wrote: > Right now it is set to no retries and no timeout on the SA > side. I will change and retest. Let me clarify a little: The no timeout is set based on resp_expected flag which is set at a higher level (presumably method based). When I did have retries set, things did not work properly. Unfortunately I was unable to get IB traces yesterday of what was going on but will try again once the analyzer is straightened out. I did have retries on with the original traces I obtained and discussed on the list where there were bigger problems than the ones observed by Tom yesterday. -- Hal From sinate at yahoo.com Wed Jun 8 04:29:16 2005 From: sinate at yahoo.com (Steven Wooding) Date: Wed, 8 Jun 2005 12:29:16 +0100 (BST) Subject: [openib-general] uverbs performance; ibv_pingpong poll vs sleep Message-ID: <20050608112916.38824.qmail@web32509.mail.mud.yahoo.com> Hi, I wonder if anyone could help me with uverbs performance. I have compaired the data rates using ibv_pingpong with and without the -e option. Therefore, using polling and waiting the CQ events (sleeping). For a data message size of 16K I get the following results: Poll: 718 MB/s with CPU at 100% Sleep: 479 MB/s with CPU at 16% I suppose the decrease in throughput is due to the time it takes to get the CQ event. Is there any way to trade off the data rate with the CPU usage (I was thinking of some timeout from polling). Any suggests would be very welcome. Regards, Steve. --------------------------------- How much free photo storage do you get? Store your holiday snaps for FREE with Yahoo! Photos. Get Yahoo! Photos -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Wed Jun 8 05:01:10 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 8 Jun 2005 15:01:10 +0300 Subject: [openib-general] Re: uverbs performance; ibv_pingpong poll vs sleep In-Reply-To: <20050608112916.38824.qmail@web32509.mail.mud.yahoo.com> References: <20050608112916.38824.qmail@web32509.mail.mud.yahoo.com> Message-ID: <20050608120110.GQ6662@mellanox.co.il> Quoting r. Steven Wooding : > Subject: uverbs performance; ibv_pingpong poll vs sleep > > Hi, > > I wonder if anyone could help me with uverbs performance. > > I have compaired the data rates using ibv_pingpong with and without the -e > option. Therefore, using polling and waiting the CQ events (sleeping). > > For a data message size of 16K I get the following results: > > Poll: 718 MB/s with CPU at 100% > Sleep: 479 MB/s with CPU at 16% What do you do for CPU utilisation measurement? > I suppose the decrease in throughput is due to the time it takes to get the CQ > event. Yes, pingpong is not really a bandwidth benchmark. It never has more than one outstanding transaction on a qp, so its bandwidth is very sensitive to roundtrip times. > Is there any way to trade off the data rate with the CPU usage (I was thinking > of some timeout from polling). > > Any suggests would be very welcome. > > Regards, > > Steve. Steve, I think an application should create some kind of pipelining, performing multiple sends before waiting for events. I have a working example of how this can be done, but unfortunately its not yet ready for publication. The idea basically is to have n outstanding buffers and tx of size n. Count the number of send requests sent and completed. Now basically replace each post send with while (send - completed < n) post send As n grows, you shall see different cpu utilization/bandwidth points. -- MST From halr at voltaire.com Wed Jun 8 07:07:37 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 08 Jun 2005 10:07:37 -0400 Subject: [openib-general] RE: [ib_at] oops in ib_at when running kdapltest In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C43@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C43@mtvex01.mtv.mtl.com> Message-ID: <1118239656.4477.59.camel@localhost.localdomain> On Tue, 2005-06-07 at 11:56, Itamar Rabenstein wrote: > attached is the output of /var/log/messages that contain ib_at oops with > ib_at debug trun on. > svn rev 2564 (ib + kdapl + at from users/jlentini) > test is : ./kdapltest -T T -s 11.1.4.43 -D mthca0a -d -t 2 -w 8 -i 20 client > SR server SR > i run the test 4-5 times before the oops occur > (In order to get the oops your kernel must be compiled with > CONFIG_DEBUG_PAGEALLOC) I've done this over 30 times and still not recreated this :-( My kernel is compiled with CONFIG_DEBUG_PAGEALLOC ever since the last time... The only difference now is my CM is instrumented which may change the timing slightly if this is due to a timing window. Out of curiousity, what SM are you using ? The sequence appears the same as all the ones which work prior to this. Also, in looking at resolve_path, I do not see a way for this to occur as long at the pointer to the path_rec struct is valid other than some scribbling problem (stack trashed somehow ?). So, can you redo this with one more change as follows: In resolve_path right before the call to ib_sa_path_rec_get insert the line: DEBUG("ib_sa_path_rec_get"); I want to rule in or out a possible theory. Thanks. -- Hal From jlentini at netapp.com Wed Jun 8 08:44:42 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 11:44:42 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1118187249.4477.278.camel@localhost.localdomain> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> Message-ID: On Tue, 7 Jun 2005, Hal Rosenstock wrote: > On Tue, 2005-05-31 at 14:17, James Lentini wrote: >> Here's the specification's exact description: >> >> timeout: Duration of time, in microseconds, that a consumer waits for >> connection establishment. The value of DAT_TIMEOUT_INFINITE >> represents no timeout, indefinite wait. Values must be >> positive. > > What is the purpose of an infinite timeout (other than the obvious) ? > The quit test uses this feature. Not sure if other tests do as well. > What happens if the REQ is lost ? Why would someone want an infinite > timeout ? We interpreted the above to mean "give the connection protocol as much time as it needs to establish a connection, but don't mask errors (no path to the remove node, etc.)". For that reason we changed the variable name to DAT_TIMEOUT_MAX. From tduffy at sun.com Wed Jun 8 08:55:33 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 08:55:33 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> Message-ID: <1118246133.19775.9.camel@duffman> On Wed, 2005-06-08 at 11:44 -0400, James Lentini wrote: > We interpreted the above to mean "give the connection protocol as > much time as it needs to establish a connection, but don't mask > errors (no path to the remove node, etc.)". For that reason we changed > the variable name to DAT_TIMEOUT_MAX. Well, let's say the end node is not there yet. Should the CM keep trying indefinitely waiting for somebody to show up and respond? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Wed Jun 8 08:50:14 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 08 Jun 2005 11:50:14 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> Message-ID: <1118245813.4477.92.camel@localhost.localdomain> On Wed, 2005-06-08 at 11:44, James Lentini wrote: > We interpreted the above to mean "give the connection protocol as > much time as it needs to establish a connection, but don't mask > errors (no path to the remove node, etc.)". For that reason we changed > the variable name to DAT_TIMEOUT_MAX. But if the REQ is lost, the timeout is really really long (longer than most will wait for an error). Transaction test also appears to be using this as well as the quit test. -- Hal From mshefty at ichips.intel.com Wed Jun 8 09:13:51 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 08 Jun 2005 09:13:51 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1118245813.4477.92.camel@localhost.localdomain> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> Message-ID: <42A7193F.4000003@ichips.intel.com> Hal Rosenstock wrote: > On Wed, 2005-06-08 at 11:44, James Lentini wrote: > >>We interpreted the above to mean "give the connection protocol as >>much time as it needs to establish a connection, but don't mask >>errors (no path to the remove node, etc.)". For that reason we changed >>the variable name to DAT_TIMEOUT_MAX. > > > But if the REQ is lost, the timeout is really really long (longer than > most will wait for an error). Transaction test also appears to be using > this as well as the quit test. My interpretation was that this is a DAPL level timeout and did not necessarily relate to a timeout for a single CM REQ. That is, there could still be a different timeout specified to the CM, but the number of retries could be infinite. Note that I'm not saying that an infinite timeout makes sense, but the use of TIMEOUT_MAX seems reasonable. To me that indicates that DAPL decides how long is needed to establish a timeout, and it manages all retries. - Sean From mshefty at ichips.intel.com Wed Jun 8 09:13:54 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 08 Jun 2005 09:13:54 -0700 Subject: [openib-general] RMPP and timeouts/retries In-Reply-To: <1118228291.4477.594.camel@localhost.localdomain> References: <1118228291.4477.594.camel@localhost.localdomain> Message-ID: <42A71942.5010505@ichips.intel.com> Hal Rosenstock wrote: >>Setting timeout > 0 when sending indicates that a MAD should be received with >>the response bit set that is in reply to the request. If timeout = 0 but RMPP >>is marked active, the MAD will still invoke RMPP and not complete until all >>segments have been ACKed. (I.e. the completion of an RMPP send indicates that >>the MAD was received.) > > If timeout = 0, does RMPP still indicate a timeout if an ACK is lost and > the other end does not reACK ? It should, yes. The send will complete with a timeout error. >>Retries is a little more complex. It indicates the number of times to send a >>request in hopes of receiving a response (if one is expected) or an ACK (if >>using RMPP). > > Any effect on ABORT and/or STOP ? I wouldn't think so, receiving an ABORT or STOP should fail the send immediately. > I will try that. Using retry and timeout on the SA side is definitely > problematic. Right now it is set to no retries and no timeout on the SA > side. I will change and retest. This should work, but will not be able to recover from a lost or delayed MAD. - Sean From roland at topspin.com Wed Jun 8 09:24:39 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 08 Jun 2005 09:24:39 -0700 Subject: [openib-general] uverbs performance; ibv_pingpong poll vs sleep In-Reply-To: <20050608112916.38824.qmail@web32509.mail.mud.yahoo.com> (Steven Wooding's message of "Wed, 8 Jun 2005 12:29:16 +0100 (BST)") References: <20050608112916.38824.qmail@web32509.mail.mud.yahoo.com> Message-ID: <527jh4erzs.fsf@topspin.com> Steven> I have compaired the data rates using ibv_pingpong with Steven> and without the -e option. Therefore, using polling and Steven> waiting the CQ events (sleeping). Steven> Is there any way to trade off the data rate with the CPU Steven> usage (I was thinking of some timeout from polling). I suppose you could have some sort of adaptive polling scheme that spins polling for a while and then sleeps waiting for an event. However, as Michael said, it's probably better to use pipelining and post multiple send work requests. This hides the latency of getting a completion event by keeping the HCA busy. - R. From jlentini at netapp.com Wed Jun 8 09:35:40 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 12:35:40 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kdapltest: fix pointer to pointer bug In-Reply-To: <1117765223.4119.15.camel@duffman> References: <1117765223.4119.15.camel@duffman> Message-ID: Committed in revision 2572. On Thu, 2 Jun 2005, Tom Duffy wrote: tduffy> In my work going through trying to get rid of the opaque dat handles, I tduffy> came across what looks like a bug in kdapltest. I don't think tduffy> DT_Performance_Test_Create() and DT_Performance_Test_Client() should tduffy> take a DAT_IA_HANDLE * as an argument as this would be a pointer to a tduffy> pointer. tduffy> tduffy> Of course, the compiler didn't catch it until I changed it to a real tduffy> struct pointers thus showing one of the problems with using opaques. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-clean/test/dapltest/test/dapl_performance_util.c tduffy> =================================================================== tduffy> --- linux-kernel-clean/test/dapltest/test/dapl_performance_util.c (revision 2532) tduffy> +++ linux-kernel-clean/test/dapltest/test/dapl_performance_util.c (working copy) tduffy> @@ -33,7 +33,7 @@ tduffy> boolean_t tduffy> DT_Performance_Test_Create ( tduffy> Per_Test_Data_t *pt_ptr, tduffy> - DAT_IA_HANDLE *ia_handle, tduffy> + DAT_IA_HANDLE ia_handle, tduffy> struct sockaddr * remote_ia_addr, tduffy> boolean_t is_server, tduffy> boolean_t is_remote_little_endian, tduffy> Index: linux-kernel-clean/test/dapltest/test/dapl_performance_client.c tduffy> =================================================================== tduffy> --- linux-kernel-clean/test/dapltest/test/dapl_performance_client.c (revision 2532) tduffy> +++ linux-kernel-clean/test/dapltest/test/dapl_performance_client.c (working copy) tduffy> @@ -34,7 +34,7 @@ int tduffy> DT_Performance_Test_Client ( tduffy> Params_t *params_ptr, tduffy> Per_Test_Data_t *pt_ptr, tduffy> - DAT_IA_HANDLE *ia_handle, tduffy> + DAT_IA_HANDLE ia_handle, tduffy> struct sockaddr * remote_ia_addr) tduffy> { tduffy> Performance_Test_t *test_ptr = NULL; tduffy> Index: linux-kernel-clean/test/dapltest/include/dapl_proto.h tduffy> =================================================================== tduffy> --- linux-kernel-clean/test/dapltest/include/dapl_proto.h (revision 2532) tduffy> +++ linux-kernel-clean/test/dapltest/include/dapl_proto.h (working copy) tduffy> @@ -236,7 +236,7 @@ void DT_Performance_Cmd_Endia tduffy> /* dapl_performance_client.c */ tduffy> int DT_Performance_Test_Client ( Params_t *params_ptr, tduffy> Per_Test_Data_t * pt_ptr, tduffy> - DAT_IA_HANDLE * ia_handle, tduffy> + DAT_IA_HANDLE ia_handle, tduffy> struct sockaddr *remote); tduffy> tduffy> boolean_t DT_Performance_Test_Client_Connect ( tduffy> @@ -261,7 +261,7 @@ boolean_t DT_Performance_Test tduffy> tduffy> /* dapl_performance_util.c */ tduffy> boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, tduffy> - DAT_IA_HANDLE * ia_handle, tduffy> + DAT_IA_HANDLE ia_handle, tduffy> struct sockaddr *remote_ia_addr, tduffy> boolean_t is_server, tduffy> boolean_t is_remote_little_endian, tduffy> From mshefty at ichips.intel.com Wed Jun 8 09:40:41 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 08 Jun 2005 09:40:41 -0700 Subject: [openib-general] RMPP and timeouts/retries In-Reply-To: <1118229122.4477.610.camel@localhost.localdomain> References: <1118228291.4477.594.camel@localhost.localdomain> <1118229122.4477.610.camel@localhost.localdomain> Message-ID: <42A71F89.6070804@ichips.intel.com> Hal Rosenstock wrote: > Let me clarify a little: > The no timeout is set based on resp_expected flag which is set at a > higher level (presumably method based). > > When I did have retries set, things did not work properly. Unfortunately > I was unable to get IB traces yesterday of what was going on but will > try again once the analyzer is straightened out. I did have retries on > with the original traces I obtained and discussed on the list where > there were bigger problems than the ones observed by Tom yesterday. If you could just describe what you did/saw, I will take a look at the code to see if I can find anything. Traces are helpful, but not necessary. - Sean From bjordan.ics at gmail.com Wed Jun 8 10:29:42 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Wed, 8 Jun 2005 13:29:42 -0400 Subject: [openib-general] shutdown issue with SDP Message-ID: <78d18e20506081029183adcfc@mail.gmail.com> I'm trying to run netperf over SDP using libsdp. I'm having intermittent success, but usually, I get a failure. The client does a shutdown (SHUT_WR), but the server gets an ECONNRESET, "Connection reset by peer", on the recv instead of a zero length recv. I'm running 2568. Anyone seen similar/different results? -- Bill Jordan SilverStorm Technologies From tduffy at sun.com Wed Jun 8 11:05:07 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 11:05:07 -0700 Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: References: <1117765223.4119.15.camel@duffman> Message-ID: <1118253907.31223.2.camel@duffman> On Wed, 2005-06-08 at 12:35 -0400, James Lentini wrote: > Committed in revision 2572. Updated patch against 2572. Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -33,7 +33,7 @@ boolean_t DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -61,10 +61,10 @@ DT_Performance_Test_Create ( test_ptr->remote_ia_addr = remote_ia_addr; test_ptr->is_remote_little_endian = is_remote_little_endian; test_ptr->base_port = (DAT_CONN_QUAL) pt_ptr->Server_Info.first_port_number; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; - ret = dat_ia_query (test_ptr->ia_handle, + ret = dat_ia_query (test_ptr->ia, NULL, &test_ptr->ia_attr, NULL); @@ -95,17 +95,17 @@ DT_Performance_Test_Create ( test_ptr->creq_evd_length = DT_PERF_DFLT_EVD_LENGTH; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; return FALSE; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->recv_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG, @@ -114,11 +114,11 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; return FALSE; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->reqt_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -127,7 +127,7 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: 0x%x dat_evd_create (request) error: %s\n", test_ptr->reqt_evd_length,test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; return FALSE; } @@ -135,30 +135,30 @@ DT_Performance_Test_Create ( if ( is_server ) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->creq_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CR_FLAG, &test_ptr->creq_evd_hdl); /* cr */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; return FALSE; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->conn_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CONNECTION_FLAG, &test_ptr->conn_evd_hdl); /* conn */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; return FALSE; } @@ -176,18 +176,18 @@ DT_Performance_Test_Create ( test_ptr->ep_context.ep_attr.max_request_dtos = pipeline_len; /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &test_ptr->ep_context.ep_attr, /* EP attrs */ - &test_ptr->ep_context.ep_handle); + &test_ptr->ep_context.ep); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->ep_context.ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context.ep = NULL; return FALSE; } @@ -197,10 +197,10 @@ DT_Performance_Test_Create ( */ test_ptr->ep_context.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, + NULL, /* rmr */ DT_PERF_SYNC_BUFF_SIZE, 2, /* 2 RMIs */ 255, /* FIXME should query for this */ @@ -226,7 +226,7 @@ DT_Performance_Test_Create ( * Post recv and sync buffers */ if ( !DT_post_recv_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_RECV_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE) ) @@ -269,13 +269,13 @@ DT_Performance_Test_Destroy ( boolean_t is_server) { u32 ret; - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; DT_Tdep_Print_Head *phead; int status = 0; phead = pt_ptr->Params.phead; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ if (test_ptr->ep_context.op.bp) @@ -310,9 +310,9 @@ DT_Performance_Test_Destroy ( * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context.ep_handle) + if (test_ptr->ep_context.ep) { - ret = dat_ep_disconnect (test_ptr->ep_context.ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context.ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -323,7 +323,7 @@ DT_Performance_Test_Destroy ( status = 1; } else if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -331,10 +331,10 @@ DT_Performance_Test_Destroy ( } } - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free error: %s\n", @@ -394,9 +394,9 @@ DT_Performance_Test_Destroy ( } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -417,7 +417,7 @@ DT_Performance_Test_Destroy ( boolean_t DT_performance_post_rdma_op ( Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats) { unsigned int j; @@ -456,7 +456,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -473,7 +473,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -502,17 +502,17 @@ DT_performance_post_rdma_op ( unsigned int DT_performance_reap ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Mode_Type mode, Performance_Stats_t *stats) { if ( BLOCKING_MODE == mode ) { - return DT_performance_wait (phead, evd_handle, stats); + return DT_performance_wait (phead, evd, stats); } else { - return DT_performance_poll (phead, evd_handle, stats); + return DT_performance_poll (phead, evd, stats); } } @@ -520,7 +520,7 @@ DT_performance_reap ( unsigned int DT_performance_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { int i; @@ -537,7 +537,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_wait ( evd_handle, + ret = DT_Tdep_evd_wait ( evd, DAT_TIMEOUT_MAX, &event); @@ -571,7 +571,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); @@ -611,7 +611,7 @@ DT_performance_wait ( unsigned int DT_performance_poll ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { u32 ret; @@ -626,7 +626,7 @@ DT_performance_poll ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: linux-kernel/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_client.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_performance_client.c (working copy) @@ -34,7 +34,7 @@ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr) { Performance_Test_t *test_ptr = NULL; @@ -47,7 +47,7 @@ DT_Performance_Test_Client ( DT_Tdep_PT_Debug (1,(phead,"Client: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - ia_handle, + ia, remote_ia_addr, FALSE, pt_ptr->Server_Info.is_little_endian, @@ -102,7 +102,7 @@ DT_Performance_Test_Client_Connect ( test_ptr->base_port, test_ptr->ep_context.port)); retry: - ret = dat_ep_connect (test_ptr->ep_context.ep_handle, + ret = dat_ep_connect (test_ptr->ep_context.ep, test_ptr->remote_ia_addr, test_ptr->ep_context.port, DAT_TIMEOUT_MAX, @@ -119,7 +119,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -296,7 +296,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -307,7 +307,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -374,9 +374,9 @@ DT_Performance_Test_Client_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -405,7 +405,7 @@ DT_Performance_Test_Client_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) @@ -460,7 +460,7 @@ DT_Performance_Test_Client_Exchange ( DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Sending Sync Msg\n", test_ptr->base_port)); if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -477,7 +477,7 @@ DT_Performance_Test_Client_Exchange ( if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Client_Sync_Send")) Index: linux-kernel/test/dapltest/test/dapl_cnxn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_cnxn.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_cnxn.c (working copy) @@ -29,7 +29,7 @@ /****************************************************************************/ int -get_ep_connection_state (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle) +get_ep_connection_state (DT_Tdep_Print_Head *phead, struct dat_ep *ep) { enum dat_ep_state ep_state; boolean_t in_dto_idle; @@ -39,7 +39,7 @@ get_ep_connection_state (DT_Tdep_Print_H char *req_status = "Idle"; - ret = dat_ep_get_status (ep_handle, &ep_state, &in_dto_idle, + ret = dat_ep_get_status (ep, &ep_state, &in_dto_idle, &out_dto_idle); if (ret != 0) { Index: linux-kernel/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_server.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_server.c (working copy) @@ -103,7 +103,7 @@ DT_cs_Server (Params_t * params_ptr) ret = dat_ia_open (Server_Cmd->dapl_name, DFLT_QLEN, &ps_ptr->async_evd_hdl, - &ps_ptr->ia_handle); + &ps_ptr->ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -111,28 +111,28 @@ DT_cs_Server (Params_t * params_ptr) module, Server_Cmd->dapl_name, DT_RetToString (ret)); - ps_ptr->ia_handle = DAT_HANDLE_NULL; + ps_ptr->ia = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: IA %s opened\n", module, Server_Cmd->dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ps_ptr->ia_handle, &ps_ptr->pz_handle); + ret = dat_pz_create (ps_ptr->ia, &ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->pz_handle = DAT_HANDLE_NULL; + ps_ptr->pz = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: PZ created\n", module)); /* Create 4 events - recv, request, connection-request, connect */ - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -142,11 +142,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->recv_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -156,11 +156,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->reqt_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CR_FLAG, @@ -170,11 +170,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (cr) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->creq_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -184,44 +184,44 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->conn_evd_hdl = NULL; status = 1; goto server_exit; } /* Create the EP */ - ret = dat_ep_create (ps_ptr->ia_handle, /* IA */ - ps_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (ps_ptr->ia, /* IA */ + ps_ptr->pz, /* PZ */ ps_ptr->recv_evd_hdl, /* recv */ ps_ptr->reqt_evd_hdl, /* request */ ps_ptr->conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ps_ptr->ep_handle); + &ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->ep_handle = DAT_HANDLE_NULL; + ps_ptr->ep = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: EP created\n", module)); /* Create PSP */ - ret = dat_psp_create (ps_ptr->ia_handle, + ret = dat_psp_create (ps_ptr->ia, SERVER_PORT_NUMBER, ps_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &ps_ptr->psp_handle); + &ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->psp_handle = DAT_HANDLE_NULL; + ps_ptr->psp = NULL; status = 1; goto server_exit; } @@ -233,10 +233,10 @@ DT_cs_Server (Params_t * params_ptr) */ ps_ptr->bpool = DT_BpoolAlloc (NULL, phead, - ps_ptr->ia_handle, - ps_ptr->pz_handle, - ps_ptr->ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ps_ptr->ia, + ps_ptr->pz, + ps_ptr->ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ 256, /* FIXME query for this value */ @@ -275,7 +275,7 @@ DT_cs_Server (Params_t * params_ptr) */ for (;/* EVER */;) { - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; @@ -299,7 +299,7 @@ DT_cs_Server (Params_t * params_ptr) /* Server_Info, Client_Info, Params set up below */ /* Gather whatever info we want about defaults */ - if (!DT_query (pt_ptr, ps_ptr->ia_handle, ps_ptr->ep_handle)) + if (!DT_query (pt_ptr, ps_ptr->ia, ps_ptr->ep)) { status = 1; goto server_exit; @@ -308,7 +308,7 @@ DT_cs_Server (Params_t * params_ptr) /* Post recv buffers for ClientInfo and Transaction_Cmd_t */ DT_Tdep_PT_Debug (1,(phead,"%s: Posting 2 recvs\n", module)); if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 0, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0))) @@ -320,7 +320,7 @@ DT_cs_Server (Params_t * params_ptr) goto server_exit; } if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 1, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1))) @@ -343,9 +343,9 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_cr_event_wait (phead, ps_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check (phead, &cr_stat, - ps_ptr->psp_handle, + ps_ptr->psp, SERVER_PORT_NUMBER, - &cr_handle, + &cr, module)) { @@ -357,7 +357,7 @@ DT_cs_Server (Params_t * params_ptr) } DT_Tdep_PT_Debug (1,(phead,"%s: Accepting Connection Request\n", module)); - ret = dat_cr_accept (cr_handle, ps_ptr->ep_handle, 0, (void *)0); + ret = dat_cr_accept (cr, ps_ptr->ep, 0, (void *)0); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -370,7 +370,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Debug (1,(phead,"%s: Awaiting connection ...\n", module)); if (!DT_conn_event_wait (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->conn_evd_hdl, &event_num)) { @@ -384,7 +384,7 @@ DT_cs_Server (Params_t * params_ptr) if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ps_ptr->ep_handle); + get_ep_connection_state (phead, ps_ptr->ep); } /* Wait for Client_Info */ @@ -395,7 +395,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0), dto_cookie, "Client_Info_Recv")) @@ -420,7 +420,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1), dto_cookie, "Client_Cmd_Recv")) @@ -585,7 +585,7 @@ DT_cs_Server (Params_t * params_ptr) /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -601,7 +601,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) @@ -619,7 +619,7 @@ DT_cs_Server (Params_t * params_ptr) /* we passed the pt_ptr to the thread and must now 'forget' it */ pt_ptr = NULL; - ret = dat_ep_disconnect (ps_ptr->ep_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect fails: %s\n", @@ -635,7 +635,7 @@ DT_cs_Server (Params_t * params_ptr) } /* reset the EP to get back into the game */ - dat_ep_reset (ps_ptr->ep_handle); + dat_ep_reset (ps_ptr->ep); DT_Tdep_PT_Debug (1,(phead,"%s: Waiting for another client...\n", module)); } /* end loop accepting connections */ @@ -669,9 +669,9 @@ server_exit: * we also get here on error, hence abrupt closure to * flush any lingering buffers posted. */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_disconnect (ps_ptr->ep_handle, + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -704,9 +704,9 @@ server_exit: } /* Free the PSP */ - if (ps_ptr->psp_handle) + if (ps_ptr->psp) { - ret = dat_psp_free (ps_ptr->psp_handle); + ret = dat_psp_free (ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_free error: %s\n", @@ -718,9 +718,9 @@ server_exit: } /* Free the EP */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_free (ps_ptr->ep_handle); + ret = dat_ep_free (ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free error: %s\n", @@ -781,9 +781,9 @@ server_exit: } /* Free the PZ */ - if (ps_ptr->pz_handle) + if (ps_ptr->pz) { - ret = dat_pz_free (ps_ptr->pz_handle); + ret = dat_pz_free (ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free error: %s\n", @@ -794,17 +794,17 @@ server_exit: } /* Close the IA */ - if (ps_ptr->ia_handle) + if (ps_ptr->ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) error: %s\n", module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -871,7 +871,7 @@ send_control_data ( sizeof (Server_Info_t)); DT_Server_Info_Endian ((Server_Info_t *) buffp); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -886,7 +886,7 @@ send_control_data ( if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) Index: linux-kernel/test/dapltest/test/dapl_fft_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) @@ -55,22 +55,22 @@ void DT_assert_fail (DT_Tdep_Print_Head } /* helper function to open an IA */ -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle) +int DT_ia_open (char *dev_name, struct dat_ia **ia) { - DAT_EVD_HANDLE evd_handle; - evd_handle = DAT_HANDLE_NULL; - return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, ia_handle); + struct dat_evd *evd; + evd = NULL; + return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, ia); } /* helper function to create an endpoint and its associated EVDs */ int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle) + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep) { u32 status; DT_Tdep_Print_Head *phead; @@ -80,7 +80,7 @@ int DT_ep_create (Params_t *params_ptr, *cr_evd = NULL; phead = params_ptr->phead; - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CR_FLAG, cr_evd); if (status != DAT_SUCCESS) { @@ -88,7 +88,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CONNECTION_FLAG, conn_evd); if (status != DAT_SUCCESS) { @@ -96,7 +96,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, send_evd); if (status != DAT_SUCCESS) @@ -105,7 +105,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, recv_evd); if (status != DAT_SUCCESS) { @@ -113,8 +113,8 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = dat_ep_create (ia_handle, pz_handle, *recv_evd, - *send_evd, *conn_evd, NULL, ep_handle); + status = dat_ep_create (ia, pz, *recv_evd, + *send_evd, *conn_evd, NULL, ep); if (status != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "dat_ep_create failed %s\n", DT_RetToString (status)); @@ -125,15 +125,15 @@ int DT_ep_create (Params_t *params_ptr, /* function that initializes the connection struct */ void DT_fft_init_conn_struct (FFT_Connection_t *conn) { - conn->ia_handle = NULL; - conn->pz_handle = NULL; - conn->psp_handle = NULL; - conn->ep_handle = NULL; + conn->ia = NULL; + conn->pz = NULL; + conn->psp = NULL; + conn->ep = NULL; conn->cr_evd = NULL; conn->send_evd = NULL; conn->conn_evd = NULL; conn->recv_evd = NULL; - conn->cr_handle = NULL; + conn->cr = NULL; conn->remote_netaddr = NULL; conn->bpool = NULL; conn->pt_ptr = NULL; @@ -154,27 +154,27 @@ void DT_fft_init_client (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); if (rc != DAT_SUCCESS) { /* make sure the handle has an invalid value */ - conn->ia_handle = NULL; + conn->ia = NULL; } DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* if a server name is given, allocate memory for a net address and set it @@ -198,11 +198,11 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_Print_Head *phead; phead = params_ptr->phead; - if (conn->ep_handle) + if (conn->ep) { if (conn->connected) { - rc = dat_ep_disconnect (conn->ep_handle, DAT_CLOSE_DEFAULT); + rc = dat_ep_disconnect (conn->ep, DAT_CLOSE_DEFAULT); DT_assert_clean (phead, rc == DAT_SUCCESS); if (!DT_disco_event_wait ( phead, conn->cr_evd, NULL )) @@ -211,16 +211,16 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_PT_Printf (phead, "DT_fft_destroy_conn_struct: bad disconnect event\n"); } } - rc = dat_ep_free (conn->ep_handle); + rc = dat_ep_free (conn->ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->bpool) { DT_Bpool_Destroy (NULL, phead, conn->bpool); } - if (conn->psp_handle) + if (conn->psp) { - rc = dat_psp_free (conn->psp_handle); + rc = dat_psp_free (conn->psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->cr_evd) @@ -247,14 +247,14 @@ int DT_fft_destroy_conn_struct (Params_t { DT_Free_Per_Test_Data (conn->pt_ptr); } - if (conn->pz_handle) + if (conn->pz) { - rc = dat_pz_free (conn->pz_handle); + rc = dat_pz_free (conn->pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn->ia_handle) + if (conn->ia) { - rc = dat_ia_close (conn->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (conn->ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return rc; @@ -272,31 +272,31 @@ void DT_fft_init_server (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PSP */ - rc = dat_psp_create (conn->ia_handle, SERVER_PORT_NUMBER, conn->cr_evd, - DAT_PSP_CONSUMER_FLAG, &conn->psp_handle); + rc = dat_psp_create (conn->ia, SERVER_PORT_NUMBER, conn->cr_evd, + DAT_PSP_CONSUMER_FLAG, &conn->psp); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate memory for buffers */ - conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia_handle, conn->pz_handle, + conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia, conn->pz, NULL, NULL, 8192, 2, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, conn->bpool); @@ -316,19 +316,19 @@ void DT_fft_listen (Params_t *params_ptr DT_assert_dat (phead, DT_cr_event_wait (phead, conn->cr_evd, &conn->cr_stat) && DT_cr_check (phead, &conn->cr_stat, - conn->psp_handle, + conn->psp, SERVER_PORT_NUMBER, - &conn->cr_handle, + &conn->cr, "DT_fft_listen")); /* accept the connection */ - rc =dat_cr_accept (conn->cr_handle, conn->ep_handle, 0, (void *)0); + rc =dat_cr_accept (conn->cr, conn->ep, 0, (void *)0); DT_assert_dat (phead, rc == DAT_SUCCESS); /* wait on a conn event via the conn EVD */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); conn->connected = TRUE; @@ -351,7 +351,7 @@ int DT_fft_connect (Params_t *params_ptr DT_Tdep_PT_Printf (phead, "Connection to server, attempt #%d\n", wait_count+1); /* attempt to connect, timeout = 10 secs */ - rc = dat_ep_connect (conn->ep_handle, conn->remote_netaddr, + rc = dat_ep_connect (conn->ep, conn->remote_netaddr, SERVER_PORT_NUMBER, 10*1000000, 0, (void *)0, DAT_QOS_BEST_EFFORT, DAT_CONNECT_DEFAULT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); @@ -359,7 +359,7 @@ int DT_fft_connect (Params_t *params_ptr /* wait on conn event */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); Index: linux-kernel/test/dapltest/test/dapl_fft_mem.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_mem.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_mem.c (working copy) @@ -37,28 +37,28 @@ int DT_mem_generic (Params_t *params_ptr FFT_Connection_t conn; DAT_REGION_DESCRIPTION region; u64 reg_size; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; DAT_LMR_CONTEXT lmr_context; u64 reg_addr; unsigned char *alloc_ptr; int res; u64 buffer_size; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; rc = 0; expect = 0; res = 1; - lmr_handle = NULL; + lmr = NULL; lmr_context = 0; reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; DT_fft_init_client (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); if (flag == 2) { @@ -76,24 +76,24 @@ int DT_mem_generic (Params_t *params_ptr memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - ia_handle = conn.ia_handle; + ia = conn.ia; if (flag != 3) { - pz_handle = conn.pz_handle; + pz = conn.pz; } if (flag != 4) { DT_Tdep_PT_Printf (phead, "Registering memory\n"); - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -110,21 +110,21 @@ int DT_mem_generic (Params_t *params_ptr } if (flag == 1) { - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_dat (phead, rc == DAT_SUCCESS); } - lmr_handle = NULL; + lmr = NULL; - rc = dat_lmr_kcreate (conn.ia_handle, + rc = dat_lmr_kcreate (conn.ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -133,9 +133,9 @@ int DT_mem_generic (Params_t *params_ptr } cleanup: - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (alloc_ptr) @@ -197,7 +197,7 @@ int DT_mem_case4 ( Params_t *params_ptr, DT_Tdep_Print_Head *phead; phead = params_ptr->phead; DT_Tdep_PT_Printf (phead, "\ - Description: Try to deregister memory with null lmr_handle\n"); + Description: Try to deregister memory with null lmr\n"); return DT_mem_generic (params_ptr, cmd, 4); } Index: linux-kernel/test/dapltest/test/dapl_limit.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) @@ -33,13 +33,13 @@ static boolean_t more_handles (DT_Tdep_Print_Head *phead, - DAT_HANDLE **old_ptrptr, /* pointer to current pointer */ + void ***old_ptrptr, /* pointer to current pointer */ unsigned int *old_count, /* number pointed to */ unsigned int size) /* size of one datum */ { unsigned int count = *old_count; - DAT_HANDLE *old_handles = *old_ptrptr; - DAT_HANDLE *handle_tmp = DT_Mdep_Malloc (count * 2 * size); + void * *old_handles = *old_ptrptr; + void * *handle_tmp = DT_Mdep_Malloc (count * 2 * size); if (!handle_tmp) { @@ -74,19 +74,19 @@ limit_test ( DT_Tdep_Print_Head *phead, Limit_Cmd_t *cmd, Limit_Index depth) { - DAT_EVD_HANDLE conn_handle; + struct dat_evd * conn_handle; typedef struct obj_set { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; - DAT_LMR_HANDLE lmr_handle; + struct dat_evd * evd; + struct dat_ep * ep; + struct dat_lmr * lmr; char * lmr_buffer; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; } Obj_Set; @@ -135,19 +135,19 @@ limit_test ( DT_Tdep_Print_Head *phead, for (w = 0; w < cmd->width; w++) { /* Specify that we want to get back an async EVD. */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdl_sets[w].ia_async_handle, - &hdl_sets[w].ia_handle); + &hdl_sets[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", module, cmd->device_name, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; - hdl_sets[w].ia_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; + hdl_sets[w].ia = NULL; goto clean_up_now; } } @@ -159,8 +159,8 @@ limit_test ( DT_Tdep_Print_Head *phead, */ typedef struct _ia { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; } OneOpen; unsigned int count = START_COUNT; @@ -178,7 +178,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -187,11 +187,11 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } /* Specify that we want to get back an async EVD. */ - hdlptr[w].ia_async_handle = DAT_HANDLE_NULL; + hdlptr[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdlptr[w].ia_async_handle, - &hdlptr[w].ia_handle); + &hdlptr[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", @@ -209,14 +209,14 @@ limit_test ( DT_Tdep_Print_Head *phead, for (tmp = 0; tmp < w; tmp++) { DT_Mdep_Schedule(); - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) fails: %s\n", module, DT_RetToString (ret)); retval = FALSE; - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -245,14 +245,14 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_pz_create (hdl_sets[w].ia_handle, - &hdl_sets[w].pz_handle); + ret = dat_pz_create (hdl_sets[w].ia, + &hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].pz_handle = DAT_HANDLE_NULL; + hdl_sets[w].pz = NULL; goto clean_up_now; } } @@ -263,7 +263,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PZs we can create */ unsigned int count = START_COUNT; - DAT_PZ_HANDLE *hdlptr = (DAT_PZ_HANDLE *) + struct dat_pz **hdlptr = (struct dat_pz **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PZ Exhaustion test loop */ @@ -279,7 +279,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -287,7 +287,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_pz_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_pz_create (hdl_sets[w % cmd->width].ia, &hdlptr[w]); if (ret != DAT_SUCCESS) { @@ -335,13 +335,13 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_cno_create (hdl_sets[w].ia_handle, + ret = dat_cno_create (hdl_sets[w].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdl_sets[w].cno_handle); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { DT_Tdep_PT_Printf (phead, "%s: dat_cno_create unimplemented\n", module); - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; /* ignore this error */ break; } @@ -350,7 +350,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_cno_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; goto clean_up_now; } } @@ -377,7 +377,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -385,7 +385,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_cno_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_cno_create (hdl_sets[w % cmd->width].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) @@ -446,7 +446,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -456,22 +456,22 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; goto clean_up_now; } for (w = 0; w < cmd->width; w++) { - ret = DT_Tdep_evd_create (hdl_sets[w].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w].ia, DFLT_QLEN, hdl_sets[w].cno_handle, flags, - &hdl_sets[w].evd_handle); + &hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].evd_handle = DAT_HANDLE_NULL; + hdl_sets[w].evd = NULL; goto clean_up_now; } } @@ -482,7 +482,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EVDs we can create */ unsigned int count = START_COUNT; - DAT_EVD_HANDLE *hdlptr = (DAT_EVD_HANDLE *) + struct dat_evd **hdlptr = (struct dat_evd **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); enum dat_evd_flags flags = ( DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG @@ -500,7 +500,7 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -510,13 +510,13 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->maximum; w++) { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -524,7 +524,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia, DFLT_QLEN, hdl_sets[w % cmd->width].cno_handle, flags, @@ -542,10 +542,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; /* EVD Cleanup loop */ - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (tmp = 0; tmp < w; tmp++) { @@ -578,19 +578,19 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_ep_create (hdl_sets[w].ia_handle, - hdl_sets[w].pz_handle, - hdl_sets[w].evd_handle, /* recv */ - hdl_sets[w].evd_handle, /* request */ + ret = dat_ep_create (hdl_sets[w].ia, + hdl_sets[w].pz, + hdl_sets[w].evd, /* recv */ + hdl_sets[w].evd, /* request */ conn_handle, /* connect */ (struct dat_ep_attr *) NULL, - &hdl_sets[w].ep_handle); + &hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ep_handle = DAT_HANDLE_NULL; + hdl_sets[w].ep = NULL; goto clean_up_now; } } @@ -601,7 +601,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EPs we can create */ unsigned int count = START_COUNT; - DAT_EP_HANDLE *hdlptr = (DAT_EP_HANDLE *) + struct dat_ep **hdlptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* EP Exhaustion test loop */ @@ -615,7 +615,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -623,10 +623,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, /* connect */ (struct dat_ep_attr *) NULL, &hdlptr[w]); @@ -672,9 +672,9 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many RSPs we can create */ unsigned int count = START_COUNT; - DAT_RSP_HANDLE *hdlptr = (DAT_RSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); - DAT_EP_HANDLE *epptr = (DAT_EP_HANDLE *) + struct dat_ep **epptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*epptr)); /* RSP Exhaustion test loop */ @@ -692,7 +692,7 @@ limit_test ( DT_Tdep_Print_Head *phead, unsigned int count1 = count; unsigned int count2 = count; - if (!more_handles (phead, (DAT_HANDLE **) &hdlptr, + if (!more_handles (phead, (void ***) &hdlptr, &count1, sizeof (*hdlptr))) { @@ -700,7 +700,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - if (!more_handles (phead, (DAT_HANDLE **) &epptr, + if (!more_handles (phead, (void ***) &epptr, &count2, sizeof (*epptr))) { @@ -723,10 +723,10 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * Each RSP needs a unique EP, so create one first */ - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, (struct dat_ep_attr *) NULL, &epptr[w]); @@ -738,10 +738,10 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } - ret = dat_rsp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_rsp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, epptr[w], - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { @@ -808,7 +808,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PSPs we can create */ unsigned int count = START_COUNT; - DAT_PSP_HANDLE *hdlptr = (DAT_PSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PSP Exhaustion test loop */ @@ -822,7 +822,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -830,9 +830,9 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_psp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_psp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, DAT_PSP_CONSUMER_FLAG, &hdlptr[w]); if (ret != DAT_SUCCESS) @@ -840,7 +840,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_psp_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); retval = TRUE; - hdlptr[w] = DAT_HANDLE_NULL; + hdlptr[w] = NULL; break; } } @@ -897,14 +897,14 @@ limit_test ( DT_Tdep_Print_Head *phead, memset (®ion, 0, sizeof (region)); region.for_va = hdl_sets[w].lmr_buffer; - ret = dat_lmr_kcreate (hdl_sets[w].ia_handle, + ret = dat_lmr_kcreate (hdl_sets[w].ia, DAT_MEM_TYPE_VIRTUAL, region, DFLT_BUFFSZ, - hdl_sets[w].pz_handle, + hdl_sets[w].pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &hdl_sets[w].lmr_handle, + &hdl_sets[w].lmr, &hdl_sets[w].lmr_context, NULL, /* FIXME */ ®_size, ®_addr); @@ -913,7 +913,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_lmr_kcreate #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].lmr_handle = DAT_HANDLE_NULL; + hdl_sets[w].lmr = NULL; goto clean_up_now; } if ((uintptr_t)reg_addr > (uintptr_t)hdl_sets[w].lmr_buffer @@ -949,7 +949,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -966,10 +966,10 @@ limit_test ( DT_Tdep_Print_Head *phead, */ hdlptr[w] = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].ep_handle, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].ep, + hdl_sets[w % cmd->width].evd, DFLT_BUFFSZ, 1, 256, /* FIXME should query for this */ @@ -1026,7 +1026,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, cmd->width * sizeof (*hdlptr))) { @@ -1054,7 +1054,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_ep_post_recv #%d\n", module, w * cmd->width + i + 1); - ret = dat_ep_post_recv (hdl_sets[i].ep_handle, + ret = dat_ep_post_recv (hdl_sets[i].ep, 1, iovp, cookie, @@ -1091,7 +1091,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * outstanding recv DTOs in error, and otherwise * be a no-op. */ - ret = dat_ep_reset (hdl_sets[i].ep_handle); + ret = dat_ep_reset (hdl_sets[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect (abrupt) fails: %s\n", @@ -1107,7 +1107,7 @@ limit_test ( DT_Tdep_Print_Head *phead, */ do { - ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd_handle, + ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd, &event); } while (ret == DAT_SUCCESS); } @@ -1130,10 +1130,10 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Mdep_Schedule(); test_bpool = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[0].ia_handle, - hdl_sets[0].pz_handle, - hdl_sets[0].ep_handle, - hdl_sets[0].evd_handle, + hdl_sets[0].ia, + hdl_sets[0].pz, + hdl_sets[0].ep, + hdl_sets[0].evd, test_size, 1, 256, /* FIXME should query for this */ @@ -1185,9 +1185,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].lmr_handle) + if (hdl_sets[w].lmr) { - ret = dat_lmr_free (hdl_sets[w].lmr_handle); + ret = dat_lmr_free (hdl_sets[w].lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_lmr_free fails: %s\n", @@ -1220,9 +1220,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ep_handle) + if (hdl_sets[w].ep) { - ret = dat_ep_free (hdl_sets[w].ep_handle); + ret = dat_ep_free (hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free fails: %s\n", @@ -1237,16 +1237,16 @@ clean_up_now: { unsigned int w; - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].evd_handle) + if (hdl_sets[w].evd) { - ret = DT_Tdep_evd_free (hdl_sets[w].evd_handle); + ret = DT_Tdep_evd_free (hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_free fails: %s\n", @@ -1284,9 +1284,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].pz_handle) + if (hdl_sets[w].pz) { - ret = dat_pz_free (hdl_sets[w].pz_handle); + ret = dat_pz_free (hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free fails: %s\n", @@ -1303,10 +1303,10 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ia_handle) + if (hdl_sets[w].ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { @@ -1318,7 +1318,7 @@ clean_up_now: * we may as well try the largest hammer we have. */ retval = FALSE; - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { Index: linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -37,31 +37,31 @@ int DT_queryinfo_basic (Params_t *params u32 result_wanted) { char *dev_name; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; struct dat_ia_attr ia_attributes; struct dat_provider_attr provider_attributes; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE cr_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *cr_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; + struct dat_ep *ep; struct dat_ep_param ep_param; DAT_CNO_HANDLE cno_handle; #ifndef __KDAPLTEST__ DAT_CNO_PARAM cno_param; #endif struct dat_evd_param evd_param; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * psp; struct dat_psp_param psp_param; - DAT_RSP_HANDLE rsp_handle; + struct dat_sp * rsp; struct dat_rsp_param rsp_param; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; struct dat_pz_param pz_param; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; struct dat_lmr_param lmr_param; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; struct dat_rmr_param rmr_param; DAT_REGION_DESCRIPTION region; u64 reg_size; @@ -77,30 +77,30 @@ int DT_queryinfo_basic (Params_t *params reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - lmr_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + lmr = NULL; rmr_handle = NULL; - pz_handle = NULL; - psp_handle = NULL; - rsp_handle = NULL; + pz = NULL; + psp = NULL; + rsp = NULL; cno_handle = NULL; - evd_handle = DAT_HANDLE_NULL; - conn_evd_handle = DAT_HANDLE_NULL; - cr_evd_handle = DAT_HANDLE_NULL; - recv_evd_handle = DAT_HANDLE_NULL; - send_evd_handle = DAT_HANDLE_NULL; + evd = NULL; + conn_evd = NULL; + cr_evd = NULL; + recv_evd = NULL; + send_evd = NULL; dev_name = cmd->device_name; - /* All functions require an ia_handle to be created */ + /* All functions require an ia to be created */ rc = dat_ia_open ((const char *)dev_name, DEFAULT_QUEUE_LEN, - &evd_handle, - &ia_handle); + &evd, + &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - /* These functions require a pz_handle to be created */ + /* These functions require a pz to be created */ if ( (object_to_query == QUERY_EVD) || (object_to_query == QUERY_RMR) || (object_to_query == QUERY_LMR) || @@ -108,43 +108,43 @@ int DT_queryinfo_basic (Params_t *params (object_to_query == QUERY_RSP) || (object_to_query == QUERY_PZ) ) { - rc = dat_pz_create (ia_handle, - &pz_handle); + rc = dat_pz_create (ia, + &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - /* These functions require a ep_handle to be created */ + /* These functions require a ep to be created */ if ( (object_to_query == QUERY_EP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CONNECTION_FLAG, - &conn_evd_handle); + &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, - pz_handle, - recv_evd_handle, - send_evd_handle, - conn_evd_handle, + rc = dat_ep_create (ia, + pz, + recv_evd, + send_evd, + conn_evd, NULL, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -152,11 +152,11 @@ int DT_queryinfo_basic (Params_t *params if ( (object_to_query == QUERY_PSP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CR_FLAG, - &cr_evd_handle); + &cr_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -165,8 +165,8 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -177,15 +177,15 @@ int DT_queryinfo_basic (Params_t *params * NULL out ia_attr and for the DAT_IA_ATTR_MASK to * have values */ - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, NULL, &provider_attributes); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_ia_query (evd_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -197,11 +197,11 @@ int DT_queryinfo_basic (Params_t *params #ifndef __KDAPLTEST__ #if defined(WIN32) - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, NULLPROXY, &cno_handle); #else - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &cno_handle); #endif @@ -220,7 +220,7 @@ int DT_queryinfo_basic (Params_t *params } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_cno_query (ia_handle, + rc = dat_cno_query (ia, &cno_param); } #endif @@ -230,17 +230,17 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, &evd_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_evd_query (ia_handle, + rc = dat_evd_query (evd, &evd_param); } } @@ -248,25 +248,25 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_psp_query function */ else if (object_to_query == QUERY_PSP) { - rc = dat_psp_create (ia_handle, + rc = dat_psp_create (ia, SERVER_PORT_NUMBER, - cr_evd_handle, + cr_evd, DAT_PSP_PROVIDER_FLAG, - &psp_handle); + &psp); DT_assert_dat (phead, rc == DAT_SUCCESS); if (result_wanted == DAT_SUCCESS) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, &psp_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_psp_query (evd_handle, + rc = dat_psp_query (psp, &psp_param); } } @@ -274,13 +274,13 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_rsp_query function */ else if (object_to_query == QUERY_RSP) { - rc = dat_rsp_create (ia_handle, + rc = dat_rsp_create (ia, SERVER_PORT_NUMBER, - ep_handle, - cr_evd_handle, - &rsp_handle); + ep, + cr_evd, + &rsp); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_rsp_query (rsp_handle, + rc = dat_rsp_query (rsp, &rsp_param); } @@ -294,14 +294,14 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_ep_query function */ else if (object_to_query == QUERY_EP) { - rc = dat_ep_query (ep_handle, + rc = dat_ep_query (ep, &ep_param); } /* Test dat_pz_query function */ else if (object_to_query == QUERY_PZ) { - rc = dat_pz_query (pz_handle, + rc = dat_pz_query (pz, &pz_param); } @@ -312,27 +312,27 @@ int DT_queryinfo_basic (Params_t *params DT_assert (phead, alloc_ptr); memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, ®_addr); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_lmr_query (lmr_handle, + rc = dat_lmr_query (lmr, &lmr_param); } /* Test dat_rmr_query function */ else if (object_to_query == QUERY_RMR) { - rc = dat_rmr_create (pz_handle, + rc = dat_rmr_create (pz, &rmr_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_rmr_query (rmr_handle, @@ -342,39 +342,39 @@ int DT_queryinfo_basic (Params_t *params DT_assert_dat (phead, DAT_GET_TYPE (rc) == result_wanted); cleanup: - if (rsp_handle) + if (rsp) { - rc = dat_rsp_free (rsp_handle); + rc = dat_rsp_free (rsp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -391,27 +391,27 @@ cleanup: DT_assert_clean (phead, rc == DAT_SUCCESS); } #endif - if (psp_handle) + if (psp) { - rc = dat_psp_free (psp_handle); + rc = dat_psp_free (psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (cr_evd_handle) + if (cr_evd) { - rc = DT_Tdep_evd_free (cr_evd_handle); + rc = DT_Tdep_evd_free (cr_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -47,7 +47,7 @@ /****************************************************************************/ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr) { Transaction_Cmd_t *cmd = &pt_ptr->Params.u.Transaction_Cmd; @@ -68,7 +68,7 @@ DT_Transaction_Test_Client (Per_Test_Dat DT_Tdep_PT_Debug (1,(phead,"Client: Starting Client side of test\n")); if (!DT_Transaction_Create_Test (pt_ptr, - ia_handle, + ia, FALSE, port_num, pt_ptr->Server_Info.is_little_endian, @@ -110,7 +110,7 @@ DT_Transaction_Test_Server (void *params + i * cmd->eps_per_thread; if (!DT_Transaction_Create_Test (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, TRUE, port_num, pt_ptr->Client_Info.is_little_endian, @@ -156,7 +156,7 @@ DT_Transaction_Test_Server (void *params boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -181,7 +181,7 @@ DT_Transaction_Create_Test (Per_Test_Dat test_ptr->remote_is_little_endian = remote_is_little_endian; test_ptr->is_server = is_server; test_ptr->pt_ptr = pt_ptr; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->base_port = (DAT_CONN_QUAL) port_num; test_ptr->cmd = &pt_ptr->Params.u.Transaction_Cmd; test_ptr->time_out = DFLT_TMO * 1000; /* DFLT_TMO seconds */ @@ -244,18 +244,18 @@ DT_Transaction_Main (void *param) private_data_str = "DAPL and RDMA rule! Test 4321."; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; status = 1; goto test_failure; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG, @@ -264,12 +264,12 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; status = 1; goto test_failure; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -278,7 +278,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (request) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; status = 1; goto test_failure; } @@ -286,7 +286,7 @@ DT_Transaction_Main (void *param) if (pt_ptr->local_is_server) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CR_FLAG, @@ -295,13 +295,13 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; status = 1; goto test_failure; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CONNECTION_FLAG, @@ -310,7 +310,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; status = 1; goto test_failure; } @@ -363,18 +363,18 @@ DT_Transaction_Main (void *param) } /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &ep_attr, /* EP attrs */ - &test_ptr->ep_context[i].ep_handle); + &test_ptr->ep_context[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - test_ptr->ep_context[i].ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context[i].ep = NULL; status = 1; goto test_failure; } @@ -385,10 +385,10 @@ DT_Transaction_Main (void *param) */ test_ptr->ep_context[i].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ buff_size, 4, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -423,7 +423,7 @@ DT_Transaction_Main (void *param) * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -433,7 +433,7 @@ DT_Transaction_Main (void *param) goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -457,11 +457,11 @@ DT_Transaction_Main (void *param) * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->creq_evd_hdl, - &test_ptr->ep_context[i].rsp_handle); + &test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create #%d error: %s\n", @@ -472,11 +472,11 @@ DT_Transaction_Main (void *param) } else { - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &test_ptr->ep_context[i].psp_handle); + &test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create #%d error: %s\n", @@ -519,7 +519,7 @@ DT_Transaction_Main (void *param) RemoteMemoryInfo *RemoteMemInfo; struct dat_dto_completion_event_data dto_stat; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; /* * Establish the connection @@ -538,16 +538,16 @@ DT_Transaction_Main (void *param) &cr_stat) || !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].rsp_handle, + test_ptr->ep_context[i].rsp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error:(%x) %s\n", @@ -571,21 +571,21 @@ DT_Transaction_Main (void *param) } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, + ret = dat_cr_accept (cr, NULL, /* NULL for RSP */ 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -594,7 +594,7 @@ DT_Transaction_Main (void *param) goto test_failure; } /* throw away single-use PSP */ - ret = dat_rsp_free (test_ptr->ep_context[i].rsp_handle); + ret = dat_rsp_free (test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free #%d error: %s\n", @@ -622,16 +622,16 @@ DT_Transaction_Main (void *param) if ( !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].psp_handle, + test_ptr->ep_context[i].psp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error: %s\n", @@ -656,33 +656,33 @@ DT_Transaction_Main (void *param) /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context[i].ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context[i].ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ - (void) dat_psp_free (test_ptr->ep_context[i].psp_handle); + /* cr consumed on failure */ + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { /* error message printed by DT_cr_event_wait */ - (void) dat_psp_free (&test_ptr->ep_context[i].psp_handle); + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* throw away single-use PSP */ - ret = dat_psp_free (test_ptr->ep_context[i].psp_handle); + ret = dat_psp_free (test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free #%d error: %s\n", @@ -709,7 +709,7 @@ DT_Transaction_Main (void *param) test_ptr->ep_context[i].ia_port)); retry: - ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_connect (test_ptr->ep_context[i].ep, test_ptr->remote_ia_addr, test_ptr->ep_context[i].ia_port, DAT_TIMEOUT_MAX, @@ -726,7 +726,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -742,7 +742,7 @@ retry: struct dat_event event; int drained = 0; - dat_ep_reset (test_ptr->ep_context[i].ep_handle); + dat_ep_reset (test_ptr->ep_context[i].ep); do { ret = DT_Tdep_evd_dequeue ( test_ptr->recv_evd_hdl, @@ -756,7 +756,7 @@ retry: * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -766,7 +766,7 @@ retry: goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -835,9 +835,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -871,9 +871,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -906,10 +906,10 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -976,7 +976,7 @@ retry: /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_SEND_BUFFER_ID, buff_size)) @@ -993,7 +993,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Send" @@ -1017,7 +1017,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Recv" @@ -1082,9 +1082,9 @@ test_failure: /* Foreach EP */ for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ for (j = 0; j < test_ptr->cmd->num_ops; j++) @@ -1122,9 +1122,9 @@ test_failure: * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context[i].ep_handle) + if (test_ptr->ep_context[i].ep) { - ret = dat_ep_disconnect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context[i].ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -1148,7 +1148,7 @@ test_failure: { if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -1164,16 +1164,16 @@ test_failure: */ for (j = 0; j < test_ptr->cmd->eps_per_thread; j++) { - if ( test_ptr->ep_context[j].ep_handle == ep_handle ) + if ( test_ptr->ep_context[j].ep == ep ) { - test_ptr->ep_context[j].ep_handle = NULL; + test_ptr->ep_context[j].ep = NULL; } } } } else /* !success - QP may be in error state */ { - ep_handle = test_ptr->ep_context[i].ep_handle; + ep = test_ptr->ep_context[i].ep; } /* @@ -1182,7 +1182,7 @@ test_failure: * disconnected as we are racing with the remote side * disconnects. */ - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { struct dat_event event; /* @@ -1195,7 +1195,7 @@ test_failure: &event); } while (ret == DAT_SUCCESS); /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free #%d error: %s\n", @@ -1259,9 +1259,9 @@ test_failure: } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -1357,7 +1357,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1432,7 +1432,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1923,8 +1923,8 @@ DT_Print_Transaction_Test (DT_Tdep_Print test_ptr->remote_is_little_endian); DT_Tdep_PT_Printf (phead, "TransTest.base_port : " F64x "\n", test_ptr->base_port); - DT_Tdep_PT_Printf (phead, "TransTest.pz_handle : %p\n", - test_ptr->pz_handle); + DT_Tdep_PT_Printf (phead, "TransTest.pz : %p\n", + test_ptr->pz); /* statistics */ DT_Tdep_PT_Printf (phead, "TransTest.bytes_send : %d\n", test_ptr->stats.stat_bytes_send); Index: linux-kernel/test/dapltest/test/dapl_performance_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_server.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_performance_server.c (working copy) @@ -41,7 +41,7 @@ DT_Performance_Test_Server ( DT_Tdep_PT_Debug (1,(phead,"Server: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, (struct sockaddr *) 0, TRUE, pt_ptr->Client_Info.is_little_endian, @@ -97,15 +97,15 @@ DT_Performance_Test_Server_Connect ( { u32 ret; boolean_t status; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; enum dat_event_number event_num; - rsp_handle = DAT_HANDLE_NULL; - psp_handle = DAT_HANDLE_NULL; + rsp = NULL; + psp = NULL; #if 0 /* FIXME */ if (test_ptr->cmd->use_rsp) { @@ -113,11 +113,11 @@ DT_Performance_Test_Server_Connect ( * Server - create a single-use RSP and * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context.port, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->creq_evd_hdl, - &rsp_handle); + &rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create error: %s\n", @@ -132,9 +132,9 @@ DT_Performance_Test_Server_Connect ( /* wait for the connection request */ if (!DT_cr_event_wait (test_ptr->conn_evd_hdl, &cr_stat) || !DT_cr_check ( &cr_stat, - DAT_HANDLE_NULL, + NULL, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -142,20 +142,20 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ - if (!DT_conn_event_wait ( test_ptr->ep_context.ep_handle, + if (!DT_conn_event_wait ( test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -173,17 +173,17 @@ DT_Performance_Test_Server_Connect ( */ status = TRUE; - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context.port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &psp_handle); + &psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); status = FALSE; - psp_handle = DAT_HANDLE_NULL; + psp = NULL; return (status); } @@ -204,9 +204,9 @@ DT_Performance_Test_Server_Connect ( if (!DT_cr_event_wait (phead, test_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check ( phead, &cr_stat, - psp_handle, + psp, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -214,22 +214,22 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num ) ) { @@ -241,10 +241,10 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_PT_Debug (1, (phead,"Server[" F64x "]: Accept on port 0x" F64x "\n", test_ptr->base_port, test_ptr->ep_context.port)); psp_free: - if ( DAT_HANDLE_NULL != psp_handle ) + if ( NULL != psp ) { /* throw away single-use PSP */ - ret = dat_psp_free (psp_handle); + ret = dat_psp_free (psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free error: %s\n", @@ -252,10 +252,10 @@ psp_free: status = FALSE; } } - if ( DAT_HANDLE_NULL != rsp_handle ) + if ( NULL != rsp ) { /* throw away single-use PSP */ - ret = dat_rsp_free (rsp_handle); + ret = dat_rsp_free (rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free error: %s\n", @@ -282,9 +282,9 @@ DT_Performance_Test_Server_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -340,7 +340,7 @@ DT_Performance_Test_Server_Exchange ( /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -358,7 +358,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Send Sync_Msg") ) @@ -380,7 +380,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (working copy) @@ -40,7 +40,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ u32 rc=0; DT_fft_init_client (phead, cmd, &conn); - DT_assert_dat (phead, conn.ia_handle != NULL) + DT_assert_dat (phead, conn.ia != NULL) DT_assert (phead, DT_fft_connect (phead, &conn)); @@ -58,8 +58,8 @@ int DT_dataxfer_client_generic (DT_Tdep_ { conn.bpool = DT_BpoolAlloc (0, phead, - conn.ia_handle, - conn.pz_handle, + conn.ia, + conn.pz, NULL, NULL, 4096, @@ -68,7 +68,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ FALSE, FALSE); DT_assert (phead, conn.bpool != 0); - rc = DT_post_send_buffer (phead, conn.ep_handle, conn.bpool, 0, + rc = DT_post_send_buffer (phead, conn.ep, conn.bpool, 0, DT_Bpool_GetBuffSize (conn.bpool, 0)); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_evd_wait (conn.send_evd, 10*1000000, 1, &conn.event, @@ -80,11 +80,11 @@ int DT_dataxfer_client_generic (DT_Tdep_ /* cleanup */ cleanup: - if (conn.ep_handle) + if (conn.ep) { /* disconnect */ DT_Tdep_PT_Printf (phead, "Disconnect\n"); - rc = dat_ep_disconnect (conn.ep_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ep_disconnect (conn.ep, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } rc = DT_fft_destroy_conn_struct (phead, &conn); Index: linux-kernel/test/dapltest/test/dapl_fft_connmgt.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (working copy) @@ -39,7 +39,7 @@ int DT_connmgt_case0 (Params_t *params_p Description: Ensure time in dat_evd_wait works correctly\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 10000, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); @@ -62,7 +62,7 @@ int DT_connmgt_case1 (Params_t *params_p Description: Attempt to use timeout of 0 in dat_evd_wait\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 0, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -86,10 +86,10 @@ Bpool * DT_BpoolAlloc ( Per_Test_Data_t *pt_ptr, DT_Tdep_Print_Head *phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia * ia, + struct dat_pz * pz, + struct dat_ep * ep, + struct dat_evd * rmr_evd, int seg_size, int num_segs, int alignment, @@ -131,16 +131,16 @@ DT_BpoolAlloc ( bpool_ptr->alloc_ptr = alloc_ptr; bpool_ptr->alloc_size = alloc_size; - bpool_ptr->pz_handle = pz_handle; + bpool_ptr->pz = pz; bpool_ptr->num_segs = num_segs; - bpool_ptr->ep_handle = ep_handle; + bpool_ptr->ep = ep; bpool_ptr->buffer_size = seg_size * num_segs; bpool_ptr->buffer_start = DT_AlignPtr (alloc_ptr, alignment) + GG_ALLOC_OFFSET; bpool_ptr->tripl_start = (struct dat_lmr_triplet *) (bpool_ptr + 1); bpool_ptr->seg_size = seg_size; bpool_ptr->enable_rdma_write = enable_rdma_write; bpool_ptr->enable_rdma_read = enable_rdma_read; - bpool_ptr->rmr_evd_handle = rmr_evd_handle; + bpool_ptr->rmr_evd = rmr_evd; DT_Tdep_PT_Debug (3, (phead, "lmr_create [%p, " F64x "]\n", @@ -160,14 +160,14 @@ DT_BpoolAlloc ( region.for_pa = virt_to_phys(region.for_va); } - ret = dat_lmr_kcreate (ia_handle, + ret = dat_lmr_kcreate (ia, DT_mem_type, region, bp_len, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &bpool_ptr->lmr_handle, + &bpool_ptr->lmr, &bpool_ptr->lmr_context, &bpool_ptr->rmr_context, &bpool_ptr->reg_size, @@ -213,7 +213,7 @@ DT_BpoolAlloc ( struct dat_rmr_bind_completion_event_data rmr_stat; /* create the RMR */ - ret = dat_rmr_create (pz_handle, &bpool_ptr->rmr_handle); + ret = dat_rmr_create (pz, &bpool_ptr->rmr_handle); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_rmr_create failed %s\n", @@ -238,7 +238,7 @@ DT_BpoolAlloc ( ret = dat_rmr_bind ( bpool_ptr->rmr_handle, &iov, mflags, - bpool_ptr->ep_handle, + bpool_ptr->ep, cookie, DAT_COMPLETION_DEFAULT_FLAG, &bpool_ptr->rmr_context); @@ -253,7 +253,7 @@ DT_BpoolAlloc ( /* await the bind result */ if (!DT_rmr_event_wait (phead, - bpool_ptr->rmr_evd_handle, + bpool_ptr->rmr_evd, &rmr_stat) || !DT_rmr_check (phead, &rmr_stat, @@ -290,9 +290,9 @@ err: DT_RetToString (ret)); } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - ret = dat_lmr_free (bpool_ptr->lmr_handle); + ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -357,9 +357,9 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - u32 ret = dat_lmr_free (bpool_ptr->lmr_handle); + u32 ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -433,8 +433,8 @@ DT_Bpool_print (DT_Tdep_Print_Head *phea "BPOOL alloc_size %x\n", (int) bpool_ptr->alloc_size); DT_Tdep_PT_Printf (phead, - "BPOOL pz_handle %p\n", - bpool_ptr->pz_handle); + "BPOOL pz %p\n", + bpool_ptr->pz); DT_Tdep_PT_Printf (phead, "BPOOL num_segs %x\n", (int) bpool_ptr->num_segs); Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -32,11 +32,11 @@ */ boolean_t DT_query ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle) + struct dat_ia * ia, + struct dat_ep * ep) { unsigned char *module = "DT_query"; - DAT_EVD_HANDLE async_evd_hdl; /* not used */ + struct dat_evd * async_evd_hdl; /* not used */ struct dat_ep_param ep_params; u32 ret; DT_Tdep_Print_Head *phead; @@ -44,7 +44,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, phead = pt_ptr->Params.phead; /* Query the IA */ - ret = dat_ia_query (ia_handle, + ret = dat_ia_query (ia, &async_evd_hdl, &pt_ptr->ia_attr, &pt_ptr->provider_attr); @@ -57,7 +57,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, } /* Query the EP */ - ret = dat_ep_query (ep_handle, &ep_params); + ret = dat_ep_query (ep, &ep_params); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_query error: %s\n", @@ -160,7 +160,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, */ boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -184,7 +184,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Recv #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_recv (ep_handle, + ret = dat_ep_post_recv (ep, 1, iov, cookie, @@ -205,7 +205,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head */ boolean_t DT_post_send_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -228,7 +228,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Send #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_send (ep_handle, + ret = dat_ep_post_send (ep, 1, iov, cookie, @@ -249,7 +249,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head */ boolean_t DT_cr_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p) { int err_cnt; @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -306,8 +306,8 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p */ boolean_t DT_conn_event_wait (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep * ep, + struct dat_evd *evd, enum dat_event_number *event_number) { @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -341,7 +341,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * { /* * Could return struct dat_connection_event_data and verify: - * event.event_data.connect_event_data.ep_handle + * event.event_data.connect_event_data.ep * event.event_data.connect_event_data.private_data_size * event.event_data.connect_event_data.private_data */ @@ -361,15 +361,15 @@ DT_conn_event_wait (DT_Tdep_Print_Head * */ boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ) + struct dat_evd *evd, + struct dat_ep * *ep ) { for (;;) { u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -392,9 +392,9 @@ DT_disco_event_wait ( DT_Tdep_Print_Head if (event.event_number == DAT_CONNECTION_EVENT_DISCONNECTED) { - if ( ep_handle != NULL ) + if ( ep != NULL ) { - *ep_handle = event.event_data.connect_event_data.ep_handle; + *ep = event.event_data.connect_event_data.ep; } return (TRUE); } @@ -412,17 +412,17 @@ DT_disco_event_wait ( DT_Tdep_Print_Head */ boolean_t DT_dto_event_reap (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dto_statusp) { if (poll) { - return DT_dto_event_poll (phead, evd_handle, dto_statusp); + return DT_dto_event_poll (phead, evd, dto_statusp); } else { - return DT_dto_event_wait (phead, evd_handle, dto_statusp); + return DT_dto_event_wait (phead, evd, dto_statusp); } } @@ -432,7 +432,7 @@ DT_dto_event_reap (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_poll (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -440,7 +440,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); if (DAT_GET_TYPE (ret) == DAT_QUEUE_EMPTY) @@ -460,7 +460,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -484,7 +484,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -505,7 +505,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -528,7 +528,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p */ boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_statusp) { for (;;) @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", @@ -572,12 +572,12 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p boolean_t DT_dto_check ( DT_Tdep_Print_Head *phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message ) { - if ( ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) || dto_p->transfered_length != len_expected || dto_p->user_cookie.as_64 != cookie_expected.as_64 || dto_p->status != DAT_DTO_SUCCESS ) @@ -588,10 +588,10 @@ DT_dto_check ( DT_Tdep_Print_Head *phead : (dto_p->status == DAT_DTO_ERR_FLUSHED ? "FAILURE" : "LengthError"))); DT_Test_Error (); - if ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) { DT_Tdep_PT_Printf (phead, "\tEndPoint mismatch (got %p wanted %p)\n", - dto_p->ep_handle, + dto_p->ep, ep_expected); } if (dto_p->transfered_length != len_expected) @@ -620,11 +620,11 @@ DT_dto_check ( DT_Tdep_Print_Head *phead boolean_t DT_rmr_check ( DT_Tdep_Print_Head *phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message) { - if ( rmr_p->rmr_handle != rmr_expected + if ( rmr_p->rmr != rmr_expected || rmr_p->user_cookie.as_ptr != cookie_expected || rmr_p->status != DAT_RMR_BIND_SUCCESS ) { @@ -634,10 +634,10 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead (rmr_p->status == DAT_RMR_BIND_SUCCESS ? "OK" : "FAILURE")); DT_Test_Error (); - if (rmr_p->rmr_handle != rmr_expected) + if (rmr_p->rmr != rmr_expected) { DT_Tdep_PT_Printf (phead, "\tRMR handle mismatch (got 0x%p wanted 0x%p)\n", - rmr_p->rmr_handle, + rmr_p->rmr, rmr_expected); } if (rmr_p->user_cookie.as_ptr != cookie_expected) @@ -659,21 +659,21 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead boolean_t DT_cr_check ( DT_Tdep_Print_Head *phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp * psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr * *crp, char *message) { u32 ret; - if (cr_handlep) + if (crp) { - *cr_handlep = (DAT_CR_HANDLE) 0; + *crp = (struct dat_cr *) 0; } if (cr_stat_p->conn_qual != port_expected || - (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected)) + (psp_expected && + cr_stat_p->sp != psp_expected)) { DT_Tdep_PT_Printf (phead, "Test Error: %s CR data problem\n", message); @@ -684,20 +684,20 @@ DT_cr_check ( DT_Tdep_Print_Head *phead " (got 0x" F64x " wanted 0x" F64x ")\n", cr_stat_p->conn_qual, port_expected); } - if (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected) + if (psp_expected && + cr_stat_p->sp != psp_expected) { DT_Tdep_PT_Printf (phead, "\tPSP mismatch (got 0x%p wanted 0x%p)\n", - cr_stat_p->sp_handle.psp_handle, - psp_handle_expected); + cr_stat_p->sp, + psp_expected); } - if (!cr_stat_p->cr_handle) + if (!cr_stat_p->cr) { - DT_Tdep_PT_Printf (phead, "\tGot NULL cr_handle\n"); + DT_Tdep_PT_Printf (phead, "\tGot NULL cr\n"); } else { - ret = dat_cr_reject (cr_stat_p->cr_handle); + ret = dat_cr_reject (cr_stat_p->cr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "\tdat_cr_reject error: %s\n", @@ -707,9 +707,9 @@ DT_cr_check ( DT_Tdep_Print_Head *phead return ( FALSE ); } - if (cr_handlep) + if (crp) { - *cr_handlep = cr_stat_p->cr_handle; + *crp = cr_stat_p->cr; } return ( TRUE ); } Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -41,13 +41,13 @@ DT_cs_Client (Params_t * params_ptr, u32 total_threads) { Per_Test_Data_t *pt_ptr = NULL; - DAT_IA_HANDLE ia_handle = DAT_HANDLE_NULL; - DAT_PZ_HANDLE pz_handle = DAT_HANDLE_NULL; - DAT_EVD_HANDLE recv_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE reqt_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE conn_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE async_evd_hdl = DAT_HANDLE_NULL; - DAT_EP_HANDLE ep_handle = DAT_HANDLE_NULL; + struct dat_ia *ia = NULL; + struct dat_pz *pz = NULL; + struct dat_evd *recv_evd_hdl = NULL; + struct dat_evd *reqt_evd_hdl = NULL; + struct dat_evd *conn_evd_hdl = NULL; + struct dat_evd *async_evd_hdl = NULL; + struct dat_ep *ep = NULL; Server_Info_t *sinfo = NULL; Transaction_Cmd_t *Transaction_Cmd = NULL; Quit_Cmd_t *Quit_Cmd = NULL; @@ -98,7 +98,7 @@ DT_cs_Client (Params_t * params_ptr, ret = dat_ia_open (dapl_name, DFLT_QLEN, &async_evd_hdl, - &ia_handle); + &ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -106,28 +106,28 @@ DT_cs_Client (Params_t * params_ptr, module, dapl_name, DT_RetToString (ret)); - ia_handle = DAT_HANDLE_NULL; + ia = NULL; status = 1; goto client_exit; } DT_Tdep_PT_Debug (1,(phead, "%s: IA %s opened\n", module, dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ia_handle, &pz_handle); + ret = dat_pz_create (ia, &pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - pz_handle = DAT_HANDLE_NULL; + pz = NULL; status = 1; goto client_exit; } /* Create 3 events - recv, request, connect */ - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -138,11 +138,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - recv_evd_hdl = DAT_HANDLE_NULL; + recv_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -153,11 +153,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - reqt_evd_hdl = DAT_HANDLE_NULL; + reqt_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -168,26 +168,26 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - conn_evd_hdl = DAT_HANDLE_NULL; + conn_evd_hdl = NULL; status = 1; goto client_exit; } /* Create an EP */ - ret = dat_ep_create (ia_handle, /* IA */ - pz_handle, /* PZ */ + ret = dat_ep_create (ia, /* IA */ + pz, /* PZ */ recv_evd_hdl, /* recv */ reqt_evd_hdl, /* request */ conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ep_handle); + &ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ep_handle = DAT_HANDLE_NULL; + ep = NULL; status = 1; goto client_exit; } @@ -197,7 +197,7 @@ DT_cs_Client (Params_t * params_ptr, * Gather whatever info we want about defaults, * and check that we can handle the requested parameters. */ - if (!DT_query (pt_ptr, ia_handle, ep_handle) || + if (!DT_query (pt_ptr, ia, ep) || !DT_check_params (pt_ptr, module)) { status = 1; @@ -206,10 +206,10 @@ DT_cs_Client (Params_t * params_ptr, bpool = DT_BpoolAlloc (pt_ptr, phead, - ia_handle, - pz_handle, - ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ia, + pz, + ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ pt_ptr->provider_attr.optimal_buffer_alignment, @@ -238,7 +238,7 @@ DT_cs_Client (Params_t * params_ptr, DT_Tdep_PT_Debug (1,(phead, "%s: Posting 1 recv buffer\n", module)); retry_repost: if (!DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, DT_Bpool_GetBuffSize (bpool, 0))) @@ -253,7 +253,7 @@ retry_repost: DT_Tdep_PT_Debug (1,(phead, "%s: Connect Endpoint\n", module)); try_connect =1; retry: - ret = dat_ep_connect (ep_handle, + ret = dat_ep_connect (ep, server_netaddr, SERVER_PORT_NUMBER, DAT_TIMEOUT_MAX, @@ -271,7 +271,7 @@ retry: } DT_Tdep_PT_Debug (1,(phead, "%s: Await connection ...\n", module)); - if (!DT_conn_event_wait (phead, ep_handle, conn_evd_hdl, &event_num)) + if (!DT_conn_event_wait (phead, ep, conn_evd_hdl, &event_num)) { if ( event_num == DAT_CONNECTION_EVENT_PEER_REJECTED ) { @@ -287,7 +287,7 @@ retry: * See if any buffers were flushed as a result of * the REJECT; clean them up and repost if so */ - dat_ep_reset (ep_handle); + dat_ep_reset (ep); do { @@ -315,7 +315,7 @@ retry: if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ep_handle); + get_ep_connection_state (phead, ep); } /* Send Client_Info (using 2nd buffer in the pool) */ @@ -326,7 +326,7 @@ retry: sizeof (Client_Info_t)); DT_Client_Info_Endian ((Client_Info_t *) buffp); if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 1, DT_Bpool_GetBuffSize (bpool, 1))) @@ -343,7 +343,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 1), dto_cookie, "Client_Info_Send")) @@ -396,7 +396,7 @@ retry: /* Send the Command buffer */ if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 2, DT_Bpool_GetBuffSize (bpool, 2))) @@ -414,7 +414,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 2), dto_cookie, "Client_Cmd_Send")) @@ -430,7 +430,7 @@ retry: if (!DT_dto_event_wait (phead, recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 0), dto_cookie, "Server_Info_Recv")) @@ -476,7 +476,7 @@ retry: DT_Transaction_Cmd_PT_Print (phead, Transaction_Cmd); } status = DT_Transaction_Test_Client (pt_ptr, - ia_handle, + ia, server_netaddr); params_ptr->Client_Stats_T.bytes_rdma_read = pt_ptr->Client_Stats.bytes_rdma_read; @@ -508,7 +508,7 @@ retry: status = DT_Performance_Test_Client (params_ptr, pt_ptr, - ia_handle, + (struct dat_ia *)ia, server_netaddr); break; } @@ -521,13 +521,13 @@ client_exit: DT_Tdep_PT_Debug (1,(phead, "%s: Cleaning Up ...\n", module)); /* Disconnect the EP */ - if (ep_handle && try_connect) + if (ep && try_connect) { /* * graceful attempt might fail because we got here due to * some error above, so we may as well try harder. */ - ret = dat_ep_disconnect (ep_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ep_disconnect (ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -547,7 +547,7 @@ client_exit: DT_Bpool_Destroy (pt_ptr, phead, bpool); /* Free the EP */ - if (ep_handle) + if (ep) { struct dat_event event; /* @@ -560,7 +560,7 @@ client_exit: &event); } while (ret == DAT_SUCCESS); - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -614,9 +614,9 @@ client_exit: } /* Free the PZ */ - if (pz_handle) + if (pz) { - ret = dat_pz_free (pz_handle); + ret = dat_pz_free (pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -629,10 +629,10 @@ client_exit: } /* Close the IA */ - if (ia_handle) + if (ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -640,7 +640,7 @@ client_exit: module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, Index: linux-kernel/test/dapltest/test/dapl_fft_endpoint.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) @@ -36,59 +36,59 @@ int DT_endpoint_generic (Params_t *param boolean_t destroy_pz_early) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; u32 rc, wanted; int res; DT_Tdep_Print_Head *phead; res = 1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - evd_handle = NULL; - conn_evd_handle = NULL; - send_evd_handle = NULL; - recv_evd_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + evd = NULL; + conn_evd = NULL; + send_evd = NULL; + recv_evd = NULL; dev_name = cmd->device_name; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; phead = params_ptr->phead; - rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, &ia_handle); + rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); if (destroy_pz_early) { - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } } - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, - DAT_EVD_CONNECTION_FLAG, &conn_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, + DAT_EVD_CONNECTION_FLAG, &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, pz_handle, recv_evd_handle, send_evd_handle, - conn_evd_handle, NULL, &ep_handle); + rc = dat_ep_create (ia, pz, recv_evd, send_evd, + conn_evd, NULL, &ep); if (destroy_pz_early) { wanted = DAT_INVALID_HANDLE; @@ -100,39 +100,39 @@ int DT_endpoint_generic (Params_t *param DT_assert_dat (phead, DAT_GET_TYPE (rc) == wanted); cleanup: - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (!destroy_pz_early && pz_handle) + if (!destroy_pz_early && pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; @@ -163,10 +163,10 @@ int DT_endpoint_case1 (Params_t *params_ int DT_endpoint_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE send_evd, conn_evd, recv_evd, cr_evd; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_ep *ep; + struct dat_evd *send_evd, *conn_evd, *recv_evd, *cr_evd; + struct dat_pz *pz; struct dat_event event; Bpool *bpool; int res; @@ -177,39 +177,39 @@ int DT_endpoint_case2 (Params_t *params_ Description: try to destroy ep with descriptor still in working queue\n"); res = 1; bpool = NULL; - pz_handle = NULL; - ia_handle = NULL; - ep_handle = NULL; + pz = NULL; + ia = NULL; + ep = NULL; send_evd = NULL; conn_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name = cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, 4096, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, 4096, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); DT_assert (phead, DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, 4096) == TRUE); - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -229,14 +229,14 @@ cleanup: rc = DT_Bpool_Destroy (NULL, phead, bpool); DT_assert_clean (phead, rc != FALSE); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_util.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c (working copy) @@ -59,7 +59,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the recv */ - ret = dat_ep_post_recv ( ep_context[i].ep_handle, + ret = dat_ep_post_recv ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -85,7 +85,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H boolean_t DT_handle_send_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll) @@ -122,7 +122,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the send */ - ret = dat_ep_post_send ( ep_context[i].ep_handle, + ret = dat_ep_post_send ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -173,7 +173,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -188,7 +188,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Send")) @@ -201,7 +201,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -215,7 +215,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -235,8 +235,8 @@ DT_handle_send_op (DT_Tdep_Print_Head *p boolean_t DT_handle_recv_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -283,7 +283,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Receive: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -298,7 +298,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Recv")) @@ -315,7 +315,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -353,7 +353,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: "F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -374,11 +374,11 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p /* * If we have multiple EPs we can't guarantee the order of - * completions, so disable ep_handle check + * completions, so disable ep check */ if (!DT_dto_check (phead, &dto_stat, - num_eps == 1?ep_context[i].ep_handle: NULL, + num_eps == 1?ep_context[i].ep: NULL, op->num_segs * op->seg_size, dto_cookie, "Send-reaped-on-recv")) @@ -395,7 +395,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -411,7 +411,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -428,7 +428,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -463,7 +463,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -516,7 +516,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (opcode == RDMA_WRITE) { - ret = dat_ep_post_rdma_write (ep_context[i].ep_handle, + ret = dat_ep_post_rdma_write (ep_context[i].ep, op->num_segs, iov, cookie, @@ -527,7 +527,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p else /* opcode == RDMA_READ */ { - ret = dat_ep_post_rdma_read ( ep_context[i].ep_handle, + ret = dat_ep_post_rdma_read ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -574,7 +574,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: %s: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -588,7 +588,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, (opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD"))) @@ -601,7 +601,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: Secondary completion seen for endpoint 0x%p (%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -619,7 +619,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: No completion seen for endpoint 0x%p (#%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); Index: linux-kernel/test/dapltest/test/dapl_fft_pz.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) @@ -35,9 +35,9 @@ int DT_pz_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_evd *evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -47,25 +47,25 @@ int DT_pz_case0 ( Params_t *params_ptr, Description: Test if we can normally create pz and destroy it.\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + ia = NULL; + pz = NULL; + evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); cleanup: - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); } return res; @@ -75,10 +75,10 @@ cleanup: int DT_pz_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE conn_evd, send_evd, recv_evd, cr_evd; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *conn_evd, *send_evd, *recv_evd, *cr_evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -88,42 +88,42 @@ int DT_pz_case1 (Params_t *params_ptr, F Description: try to destroy pz with vi still associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; conn_evd = NULL; send_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } cleanup: /* corrrect order */ - if (ep_handle) + if (ep) { - rc=dat_ep_free (ep_handle); + rc=dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn_evd) @@ -141,15 +141,15 @@ cleanup: rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -160,8 +160,8 @@ cleanup: int DT_pz_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz *pz; Bpool *bpool; u32 rc; int res; @@ -174,26 +174,26 @@ int DT_pz_case2 (Params_t *params_ptr, F associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; bpool = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate and register bpool */ - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, BUFFSIZE, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } @@ -205,15 +205,15 @@ cleanup: DT_Tdep_PT_Printf (phead, "Warning: Destroy bpool fails, reboot for cleanup\n"); return 0; } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: linux-kernel/test/dapltest/test/dapl_fft_hwconn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (working copy) @@ -31,8 +31,8 @@ int DT_hwconn_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE nic_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * nic_handle; + struct dat_evd *evd; u32 rc; int res = 1; DT_Tdep_Print_Head *phead; @@ -43,9 +43,9 @@ int DT_hwconn_case0 ( Params_t *params_p dev_name= cmd->device_name; nic_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); @@ -58,9 +58,9 @@ cleanup: /*--------------------------------------------------------*/ int DT_hwconn_case1 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; char dev_name[100]; int i; DT_Tdep_Print_Head *phead; @@ -94,8 +94,8 @@ int DT_hwconn_case1 ( Params_t *params_p sprintf (dev_name, "%s", "34df"); /* number_letter */ } - evd_handle = DAT_HANDLE_NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + evd = NULL; + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); if (DAT_GET_TYPE (rc) != DAT_PROVIDER_NOT_FOUND) { const char *major_msg, *minor_msg; @@ -120,7 +120,7 @@ int DT_hwconn_case1 ( Params_t *params_p /*--------------------------------------------------------*/ int DT_hwconn_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; int res=1; DT_Tdep_Print_Head *phead; @@ -156,9 +156,9 @@ int DT_hwconn_case3 (Params_t *params_pt DT_fft_init_client (params_ptr, cmd, &conn); /* try to close nic when vi have not destroyed */ - if (conn.ia_handle) + if (conn.ia) { - rc= dat_ia_close (conn.ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc= dat_ia_close (conn.ia, DAT_CLOSE_ABRUPT_FLAG); if (rc !=DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Warning: dat_ia_close fails %s, reboot for cleanup\n", Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (revision 2572) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (working copy) @@ -34,7 +34,7 @@ int DT_dataxfer_generic ( DT_Tdep_Print_ u32 rc=0; int res=1; DT_fft_init_server (phead, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); DT_fft_listen (phead, &conn); @@ -81,7 +81,7 @@ cleanup: int DT_dataxfer_case0 ( DT_Tdep_Print_Head *phead, FFT_Cmd_t *cmd) { DT_Tdep_PT_Printf (phead, "\ - Description: Call dat_ep_post_send with null ep_handle.\n"); + Description: Call dat_ep_post_send with null ep.\n"); return DT_dataxfer_generic (phead, cmd, 0); } Index: linux-kernel/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_tdep.h (revision 2572) +++ linux-kernel/test/dapltest/include/dapl_tdep.h (working copy) @@ -31,7 +31,7 @@ #include "dapl_proto.h" #ifdef __KDAPL__ -typedef DAT_HANDLE DAT_CNO_HANDLE; +typedef void * DAT_CNO_HANDLE; #endif /* function prototypes */ @@ -45,21 +45,21 @@ int DT_Tdep_Execute_Test ( Params_t *params_ptr ) ; u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr); + struct dat_evd * *evd_ptr); u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle); +DT_Tdep_evd_free (struct dat_evd * evd); u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd * evd, unsigned long timeout, struct dat_event *event); u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd * evd, struct dat_event *event); #endif Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2572) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -54,7 +54,7 @@ typedef struct { typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL port; int pipeline_len; @@ -70,17 +70,17 @@ typedef struct boolean_t is_remote_little_endian; DAT_CONN_QUAL base_port; struct dat_ia_attr ia_attr; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; int reqt_evd_length; - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ int recv_evd_length; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ + struct dat_evd * recv_evd_hdl; /* receive */ int conn_evd_length; - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ + struct dat_evd * conn_evd_hdl; /* connect */ int creq_evd_length; - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_evd * creq_evd_hdl; /* "" request */ Performance_Ep_Context_t ep_context; } Performance_Test_t; Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2572) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -81,10 +81,10 @@ extern int g_status; /* dapl_bpool.c */ Bpool * DT_BpoolAlloc (Per_Test_Data_t * pt_ptr, DT_Tdep_Print_Head* phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_ep *ep, + struct dat_evd *rmr_evd, int seg_size, int num_segs, int alignment, @@ -106,7 +106,7 @@ void DT_Bpool_print (DT_Tdep_ /* dapl_cnxn.c */ int get_ep_connection_state (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle); + struct dat_ep *ep); /* dapl_client.c */ int DT_cs_Client (Params_t * params_ptr, @@ -236,7 +236,7 @@ void DT_Performance_Cmd_Endia /* dapl_performance_client.c */ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote); boolean_t DT_Performance_Test_Client_Connect ( @@ -261,7 +261,7 @@ boolean_t DT_Performance_Test /* dapl_performance_util.c */ boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -272,20 +272,20 @@ int DT_Performance_Test_Dest boolean_t is_server); boolean_t DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats); unsigned int DT_performance_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Mode_Type mode, Performance_Stats_t *stats); unsigned int DT_performance_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); unsigned int DT_performance_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); /* dapl_performance_stats.c */ @@ -346,69 +346,69 @@ void DT_Free_Per_Test_Data (Per_Test_D /* dapl_test_util.c */ boolean_t DT_query (Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle); + struct dat_ia * ia, + struct dat_ep * ep); boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_post_send_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_conn_event_wait (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep *ep, + struct dat_evd *evd, enum dat_event_number *event_number); boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ); + struct dat_evd *evd, + struct dat_ep * *ep ); boolean_t DT_cr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p); boolean_t DT_dto_event_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_ptr); boolean_t DT_dto_check ( DT_Tdep_Print_Head* phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message); boolean_t DT_rmr_check ( DT_Tdep_Print_Head* phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message); boolean_t DT_cr_check (DT_Tdep_Print_Head* phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp *psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr **crp, char *message); /* dapl_thread.c */ @@ -460,13 +460,13 @@ void DT_Transaction_Cmd_Endia boolean_t to_wire); /* dapl_transaction_test.c */ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote); void DT_Transaction_Test_Server (void *params); boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -491,15 +491,15 @@ boolean_t DT_handle_post_recv boolean_t DT_handle_send_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll); boolean_t DT_handle_recv_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -507,7 +507,7 @@ boolean_t DT_handle_recv_op ( boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -600,15 +600,15 @@ void DT_assert_fail (DT_Tdep_Print_H char *file, char *baseFile, int line); -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle); +int DT_ia_open (char *dev_name, struct dat_ia **ia); int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle); + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep); void DT_fft_init_conn_struct (FFT_Connection_t *conn); void DT_fft_init_client (Params_t *params_ptr, FFT_Cmd_t *cmd, Index: linux-kernel/test/dapltest/include/dapl_test_data.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_test_data.h (revision 2572) +++ linux-kernel/test/dapltest/include/dapl_test_data.h (working copy) @@ -46,16 +46,16 @@ typedef struct int NextPortNumber; int num_clients; spinlock_t num_clients_lock; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; - DAT_EVD_HANDLE reqt_evd_hdl; - DAT_EVD_HANDLE conn_evd_hdl; - DAT_EVD_HANDLE creq_evd_hdl; - DAT_EVD_HANDLE async_evd_hdl; - DAT_EVD_HANDLE rmr_evd_hdl; - DAT_EP_HANDLE ep_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_evd *recv_evd_hdl; + struct dat_evd *reqt_evd_hdl; + struct dat_evd *conn_evd_hdl; + struct dat_evd *creq_evd_hdl; + struct dat_evd *async_evd_hdl; + struct dat_evd *rmr_evd_hdl; + struct dat_ep *ep; + struct dat_sp *psp; Bpool *bpool; } Per_Server_Data_t; Index: linux-kernel/test/dapltest/include/dapl_bpool.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2572) +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) @@ -38,21 +38,21 @@ typedef struct Bpool_tag Bpool; struct Bpool_tag { unsigned char *alloc_ptr; uint32_t alloc_size; - DAT_PZ_HANDLE pz_handle; + struct dat_pz * pz; int seg_size; int num_segs; /* num segments */ unsigned char *buffer_start; /* Start of buffer area */ u64 buffer_size; /* Size of data buffer (rounded) */ u64 reg_addr; /* start of registered area */ u64 reg_size; /* size of registered area */ - DAT_EP_HANDLE ep_handle; /* EP area is registered to */ - DAT_LMR_HANDLE lmr_handle; /* local access */ + struct dat_ep * ep; /* EP area is registered to */ + struct dat_lmr * lmr; /* local access */ DAT_LMR_CONTEXT lmr_context; struct dat_lmr_triplet *tripl_start; /* local IOV */ boolean_t enable_rdma_write; /* remote access */ boolean_t enable_rdma_read; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; - DAT_EVD_HANDLE rmr_evd_handle; + struct dat_evd * rmr_evd; }; #endif Index: linux-kernel/test/dapltest/include/dapl_fft_util.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2572) +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) @@ -55,14 +55,14 @@ typedef struct { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_PSP_HANDLE psp_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE cr_evd, conn_evd, send_evd, recv_evd; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_sp *psp; + struct dat_ep *ep; + struct dat_evd *cr_evd, *conn_evd, *send_evd, *recv_evd; struct dat_event event; int count; - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; Bpool *bpool; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2572) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -49,13 +49,13 @@ typedef struct typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL ia_port; Bpool *bp; Transaction_Test_Op_t op[ MAX_OPS ]; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; } Ep_Context_t; @@ -75,7 +75,7 @@ typedef struct boolean_t is_server; boolean_t remote_is_little_endian; Per_Test_Data_t *pt_ptr; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; @@ -84,11 +84,11 @@ typedef struct Thread *thread; /* This group set up by each thread in DT_Transaction_Main() */ - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_pz * pz; + struct dat_evd * recv_evd_hdl; /* receive */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ + struct dat_evd * conn_evd_hdl; /* connect */ + struct dat_evd * creq_evd_hdl; /* "" request */ Ep_Context_t *ep_context; /* Statistics set by DT_Transaction_Run() */ Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2572) +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -39,7 +39,7 @@ typedef struct Tdep_Evd_Tag struct Tdep_Evd_Tag *evd_next; struct Tdep_Event_Tag *event_next; struct Tdep_Event_Tag *event_last; - DAT_EVD_HANDLE evd_handle; + struct dat_evd * evd; DT_WAIT_OBJECT wait_object; } Tdep_Evd; @@ -94,11 +94,11 @@ KDT_Evd_Destroy(void) } u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr) + struct dat_evd **evd_ptr_ptr) { u32 dat_status; @@ -120,12 +120,12 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha upcall.upcall_func = DT_Tdep_Event_Callback; upcall.instance_data = evd_ptr; - dat_status = dat_evd_kcreate (ia_handle, + dat_status = dat_evd_kcreate (ia, evd_min_qlen, DAT_UPCALL_SINGLE_INSTANCE, &upcall, evd_flags, - evd_handle_ptr); + evd_ptr_ptr); if (dat_status != DAT_SUCCESS) { goto error; @@ -134,7 +134,7 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha /* initialize evd_ptr */ evd_ptr->event_next = NULL; evd_ptr->event_last = NULL; - evd_ptr->evd_handle = *evd_handle_ptr; + evd_ptr->evd = *evd_ptr_ptr; DT_Mdep_wait_object_init (&evd_ptr->wait_object); /* add evd_ptr to front of evd list */ @@ -154,7 +154,7 @@ error: return (dat_status); } u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd *evd, struct dat_event *dat_event) { u32 dat_status; @@ -163,11 +163,11 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ unsigned long flags; dat_status = DAT_SUCCESS; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -181,7 +181,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } /* Get event */ spin_lock_irqsave (&DT_Evd_Lock, flags); - event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd_handle */ + event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd */ if (event) { evd_ptr->event_next = event->event_next; /* point to next event */ @@ -212,7 +212,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd *evd, unsigned long timeout, struct dat_event *dat_event) { @@ -223,11 +223,11 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han dat_status = DAT_CLASS_ERROR | DAT_INTERNAL_ERROR; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -290,7 +290,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han } u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle) +DT_Tdep_evd_free (struct dat_evd *evd) { Tdep_Evd *next; Tdep_Evd *last; @@ -298,13 +298,13 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han last = NULL; spin_lock_irq (&DT_Evd_Lock); next = DT_Evd_List; - if (next->evd_handle == evd_handle) + if (next->evd == evd) { DT_Evd_List = next->evd_next; } else { - while (next && next->evd_handle != evd_handle) + while (next && next->evd != evd) { last = next; next = next->evd_next; @@ -317,7 +317,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han spin_unlock_irq (&DT_Evd_Lock); DT_Mdep_Free (next); - return dat_evd_free(evd_handle); + return dat_evd_free(evd); } static void DT_Tdep_Event_Callback ( Index: linux-kernel/dat-provider/dapl_cr_reject.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_reject.c (revision 2572) +++ linux-kernel/dat-provider/dapl_cr_reject.c (working copy) @@ -40,36 +40,36 @@ * Reject a connection request from the active remote side requesting * an Endpoint. */ -u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) +u32 dapl_cr_reject(struct dat_cr *cr) { struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; struct dapl_sp *sp_ptr; u32 dat_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* * Clean up provider created EP if there is one: only if * DAT_PSP_PROVIDER_FLAG was set on the PSP */ - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - entry_ep_handle = cr_ptr->param.local_ep_handle; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; + entry_ep = cr_ptr->param.local_ep; entry_ep_state = 0; if (ep_ptr) { entry_ep_state = ep_ptr->param.ep_state; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - cr_ptr->param.local_ep_handle = NULL; + cr_ptr->param.local_ep = NULL; } dat_status = dapl_ib_reject_connection(cr_ptr->ib_cm_handle); @@ -77,8 +77,8 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand if (ep_ptr != NULL) { /* Revert our state to the beginning */ ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; - cr_ptr->param.local_ep_handle = (DAT_EP_HANDLE) ep_ptr; + cr_ptr->param.local_ep = entry_ep; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; } } else { /* @@ -88,13 +88,13 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand */ sp_ptr = cr_ptr->sp_ptr; if (ep_ptr && sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) - (void)dapl_ep_free(ep_ptr); + (void)dapl_ep_free((struct dat_ep *)ep_ptr); /* Remove the CR from the queue, then free it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); } Index: linux-kernel/dat-provider/dapl_srq.c =================================================================== --- linux-kernel/dat-provider/dapl_srq.c (revision 2572) +++ linux-kernel/dat-provider/dapl_srq.c (working copy) @@ -51,13 +51,9 @@ */ static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) { - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); - - /* reset magic to prevent reuse */ - srq_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); + dapl_ia_unlink_srq(srq_ptr->common.owner_ia, srq_ptr); dapl_cb_free(&srq_ptr->recv_buffer); - /* no need to destroy srq_ptr->header.lock */ + /* no need to destroy srq_ptr->common.lock */ kfree(srq_ptr); } @@ -91,19 +87,14 @@ static struct dapl_srq *dapl_srq_alloc(s /* zero the structure */ memset(srq_ptr, 0, sizeof *srq_ptr); - /* - * initialize the header - */ - srq_ptr->header.provider = ia_ptr->header.provider; - srq_ptr->header.magic = DAPL_MAGIC_SRQ; - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; - srq_ptr->header.owner_ia = ia_ptr; - srq_ptr->header.user_context.as_64 = 0; - srq_ptr->header.user_context.as_ptr = NULL; + srq_ptr->srq.provider = ia_ptr->ia.provider; + srq_ptr->common.owner_ia = ia_ptr; + srq_ptr->common.user_context.as_64 = 0; + srq_ptr->common.user_context.as_ptr = NULL; atomic_set(&srq_ptr->srq_ref_count, 0); - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); - spin_lock_init(&srq_ptr->header.lock); + dapl_llist_init_entry(&srq_ptr->common.ia_list_entry); + spin_lock_init(&srq_ptr->common.lock); /* * Initialize the body. @@ -130,15 +121,15 @@ bail: * dapl_srq_create * * Create an instance of a Shared Receive Queue that is provided to the - * consumer at srq_handle. + * consumer at srq. * * Input: - * ia_handle - * pz_handle + * ia + * pz * srq_attr * * Output: - * srq_handle + * srq * * Returns: * DAT_SUCCESS @@ -148,8 +139,8 @@ bail: * ?DAT_INVALID_ATTRIBUTE?? * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) +u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_srq_attr *srq_attr, struct dat_srq **srq) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; @@ -157,14 +148,14 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_create (%p, %p, %p, %p)\n", - ia_handle, pz_handle, srq_attr, srq_handle); + ia, pz, srq_attr, srq); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -175,12 +166,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - - if (srq_handle == NULL) { + if (srq == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -197,9 +183,9 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han goto bail; } - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; + srq_ptr->param.ia = (struct dat_ia *) ia_ptr; srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; - srq_ptr->param.pz_handle = pz_handle; + srq_ptr->param.pz = pz; /* * XXX Allocate provider resource here!!! @@ -211,7 +197,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han /* Link it onto the IA */ dapl_ia_link_srq(ia_ptr, srq_ptr); - *srq_handle = srq_ptr; + *srq = (struct dat_srq *)srq_ptr; bail: return status; @@ -223,7 +209,7 @@ bail: * Destroy an instance of an SRQ * * Input: - * srq_handle + * srq * * Output: * none @@ -233,22 +219,22 @@ bail: * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) +u32 dapl_srq_free(struct dat_srq *srq) { struct dapl_srq *srq_ptr; struct dapl_ia *ia_ptr; struct dat_srq_param *param; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq); - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; param = &srq_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { + if (!srq_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -263,17 +249,17 @@ u32 dapl_srq_free(DAT_SRQ_HANDLE srq_han goto bail; } - ia_ptr = srq_ptr->header.owner_ia; + ia_ptr = srq_ptr->common.owner_ia; /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_lock_irqsave(&srq_ptr->common.lock, srq_ptr->common.flags); /* Remove the SRQ from the IA */ dapl_ia_unlink_srq(ia_ptr, srq_ptr); - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_unlock_irqrestore(&srq_ptr->common.lock, srq_ptr->common.flags); /* * Finish tearing everything down. @@ -301,7 +287,7 @@ bail: * over a connection of any ep handle into local_iov * * Input: - * srq_handle + * srq * num_segments * local_iov * user_cookie @@ -318,7 +304,7 @@ bail: * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie) { @@ -328,14 +314,14 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_post_recv (%p, %d, %p, %P)\n", - srq_handle, num_segments, local_iov, user_cookie.as_64); + srq, num_segments, local_iov, user_cookie.as_64); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * Synchronization ok since this buffer is only used for receive @@ -369,16 +355,15 @@ bail: return status; } -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) +u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_srq_query (%p, %x, %p)\n", - srq_handle, srq_param); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", srq, + srq_param); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -387,7 +372,7 @@ u32 dapl_srq_query(DAT_SRQ_HANDLE srq_ha goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * XXX Need to calculate available_dto_count and outstanding_dto_count @@ -407,7 +392,7 @@ bail: * Modify the size fo the event queue of a Shared Recieve Queue * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -421,22 +406,22 @@ bail: * DAT_INVALID_STATE */ -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) +u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", - srq_handle, srq_max_recv_dto); + srq, srq_max_recv_dto); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; - ia_ptr = srq_ptr->header.owner_ia; + srq_ptr = (struct dapl_srq *)srq; + ia_ptr = srq_ptr->common.owner_ia; /* * Check for nonsense requests per the spec @@ -463,7 +448,7 @@ bail: * event if it is reached. * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -476,20 +461,20 @@ bail: * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) +u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", - srq_handle, low_watermark); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq, + low_watermark); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* XXX Put implementation here XXX */ Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2572) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -171,7 +171,7 @@ u32 dapl_ib_close_hca(struct dapl_hca *h * Alloc a CQ * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * cqlen minimum QLen * @@ -227,7 +227,7 @@ u32 dapl_ib_cq_resize(struct dapl_ia *ia * Set up CQ completion notifications * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * * Output: @@ -253,7 +253,7 @@ u32 dapl_set_cq_notify(struct dapl_ia *i * Register a virtual memory region * * Input: - * ia_handle IA handle + * ia IA handle * lmr pointer to dapl_lmr struct * virt_addr virtual address of beginning of mem region * length length of memory region @@ -300,7 +300,7 @@ u32 dapl_ib_mr_register_physical(struct iova = buf_list[0].addr; acl = dapl_ib_convert_mem_privileges(privileges); acl |= IB_ACCESS_MW_BIND; - mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz_handle)->pd, + mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz)->pd, buf_list, length, acl, &iova); kfree(buf_list); if (IS_ERR(mr)) { @@ -393,7 +393,7 @@ u32 dapl_ib_mw_alloc(struct dapl_rmr *rm int ib_status; struct ib_mw *mw; - mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz_handle)->pd); + mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz)->pd); if (IS_ERR(mw)) { ib_status = PTR_ERR(mw); dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -540,7 +540,7 @@ u32 dapl_ib_mw_unbind(struct dapl_rmr *r * Set up an asynchronous callbacks of various kinds * * Input: - * ia_handle IA handle + * ia IA handle * handler_type type of handler to set up * callback_handle handle param for completion callbacks * callback callback routine pointer Index: linux-kernel/dat-provider/dapl_ia_memtype_hint.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_memtype_hint.c (revision 2572) +++ linux-kernel/dat-provider/dapl_ia_memtype_hint.c (working copy) @@ -37,7 +37,7 @@ * Provide a hint about optimal memory alignment and size * * Input: - * ia_handle, + * ia, * mem_type, * length, * mem_optimization, @@ -53,7 +53,7 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, @@ -63,11 +63,11 @@ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE i u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", - ia_handle); + ia); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; Index: linux-kernel/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2572) +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) @@ -74,17 +74,17 @@ u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ attr = &ep_ptr->param.ep_attr; - dapl_os_assert(ep_ptr->param.pz_handle != NULL); - ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz_handle)->pd; + dapl_os_assert(ep_ptr->param.pz != NULL); + ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz)->pd; ib_hca_handle = ia_ptr->hca_ptr->ib_hca_handle; cq_recv = NULL; cq_send = NULL; cq_recv = dapl_get_dto_cq(ia_ptr, - (struct dapl_evd *)ep_ptr->param.recv_evd_handle); + (struct dapl_evd *)ep_ptr->param.recv_evd); cq_send = dapl_get_dto_cq(ia_ptr, (struct dapl_evd *)ep_ptr->param. - request_evd_handle); + request_evd); qp_attr.send_cq = cq_send; qp_attr.recv_cq = cq_recv; @@ -160,7 +160,7 @@ struct ib_cq *dapl_get_dto_cq(struct dap int cq_entries; int status; - if (evd_ptr != DAT_HANDLE_NULL) + if (evd_ptr != NULL) cq = evd_ptr->cq; else if (ia_ptr->hca_ptr->null_cq != NULL) cq = ia_ptr->hca_ptr->null_cq; Index: linux-kernel/dat-provider/dapl_openib_util.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.h (revision 2572) +++ linux-kernel/dat-provider/dapl_openib_util.h (working copy) @@ -59,7 +59,7 @@ u32 dapl_ib_qp_free(struct dapl_ep *ep_p u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, struct dat_ep_attr *ep_attr); -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data); @@ -70,12 +70,12 @@ u32 dapl_ib_setup_conn_listener(struct d u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr); -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, + struct dat_ep *ep, int private_data_size, const void *private_data); -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle); +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm); u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, DAPL_ASYNC_HANDLER_TYPE handler_type, @@ -126,14 +126,14 @@ u32 dapl_ib_completion_notify(struct dap enum dat_dto_completion_status dapl_ib_get_dto_status(struct ib_wc *cqe_ptr); -void dapl_ib_reinit_ep(struct dapl_ep * ep_ptr); +void dapl_ib_reinit_ep(struct dapl_ep *ep_ptr); -void dapl_ib_disconnect_clean(struct dapl_ep * ep_ptr, boolean_t passive); +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t passive); u32 dapl_ib_get_async_event(struct ib_event *cause_ptr, enum dat_event_number *async_event); -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address); static inline u32 dapl_ib_status_convert(int32_t ib_status) Index: linux-kernel/dat-provider/dapl_ia_close.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_close.c (revision 2572) +++ linux-kernel/dat-provider/dapl_ia_close.c (working copy) @@ -40,7 +40,7 @@ * Close a provider, clean up resources, etc. * * Input: - * ia_handle + * ia * * Output: * none @@ -50,30 +50,29 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags ia_flags) +u32 dapl_ia_close(struct dat_ia *ia, enum dat_close_flags ia_flags) { struct dapl_ia *ia_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); + "dapl_ia_close (%p, %d)\n", ia, ia_flags); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) { + if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) dat_status = dapl_ia_abrupt_close(ia_ptr); - } else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) { + else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) dat_status = dapl_ia_graceful_close(ia_ptr); - } else { + else dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_cr_callback.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_callback.c (revision 2572) +++ linux-kernel/dat-provider/dapl_cr_callback.c (working copy) @@ -50,9 +50,9 @@ static u32 dapl_connection_request(struc struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; - cr_ptr = dapl_cr_alloc(sp_ptr->header.owner_ia); + cr_ptr = dapl_cr_alloc(sp_ptr->common.owner_ia); if (cr_ptr == NULL) { /* Invoking function will call dapl_ib_cm_reject() */ return DAT_INSUFFICIENT_RESOURCES; @@ -79,7 +79,7 @@ static u32 dapl_connection_request(struc } /* EP will be NULL unless RSP service point */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { /* @@ -89,7 +89,7 @@ static u32 dapl_connection_request(struc * EP we are out of resources and need to tell the * requestor that we cant help them. */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; ep_ptr = dapl_ep_alloc(ia_ptr, NULL); if (ep_ptr == NULL) { dapl_cr_free(cr_ptr); @@ -100,7 +100,7 @@ static u32 dapl_connection_request(struc dapl_ia_link_ep(ia_ptr, ep_ptr); } - cr_ptr->param.local_ep_handle = ep_ptr; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; if (ep_ptr != NULL) { /* Assign valid EP fields: RSP and PSP_PROVIDER_FLAG only */ @@ -109,12 +109,11 @@ static u32 dapl_connection_request(struc DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING; } else { /* RSP */ - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == DAT_SP_TYPE_RSP); ep_ptr->param.ep_state = DAT_EP_STATE_PASSIVE_CONNECTION_PENDING; } - ep_ptr->cm_handle = ib_cm_handle; + ep_ptr->cm = ib_cm_handle; } /* link the CR onto the SP so we can pick it up later */ @@ -122,16 +121,16 @@ static u32 dapl_connection_request(struc /* Post the event. */ /* assign sp_ptr to union to avoid typecast errors from some compilers */ - sp_handle.psp_handle = (DAT_PSP_HANDLE) sp_ptr; - dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp_handle, - (struct sockaddr *) &sp_ptr->header.owner_ia->hca_ptr->hca_address, - sp_ptr->conn_qual, (DAT_CR_HANDLE) cr_ptr); + sp = (struct dat_sp *)sp_ptr; + dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp, + (struct sockaddr *)&sp_ptr->common.owner_ia->hca_ptr->hca_address, + sp_ptr->conn_qual, (struct dat_cr *)cr_ptr); if (dat_status != DAT_SUCCESS) { /* Take the CR off the list, we can't use it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); (void)dapl_ib_reject_connection(ib_cm_handle); @@ -157,13 +156,7 @@ struct dapl_ep *dapl_get_sp_ep(struct da /* * acquire the lock, we may be racing with other threads here */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); - /* Verify under lock that the SP is still valid */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - return NULL; - } + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * There are potentially multiple connections in progress. Need to * go through the list and find the one we are interested @@ -172,16 +165,12 @@ struct dapl_ep *dapl_get_sp_ep(struct da */ cr_ptr = dapl_sp_search_cr(sp_ptr, ib_cm_handle); if (cr_ptr == NULL) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); return NULL; } - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - - /* Quick check to ensure our EP is still valid */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) - ep_ptr = NULL; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; /* The CR record is discarded in all except for the CONNECTED case, * as it will have no further relevance. @@ -211,31 +200,31 @@ struct dapl_ep *dapl_get_sp_ep(struct da "dapl_get_sp_ep disconnect dump sp: %p \n", sp_ptr); /* Decrement the ref count on the EVD */ - if (sp_ptr->evd_handle) { + if (sp_ptr->evd) { atomic_dec(&((struct dapl_evd *)sp_ptr-> - evd_handle)->evd_ref_count); - sp_ptr->evd_handle = NULL; + evd)->evd_ref_count); + sp_ptr->evd = NULL; } sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - (void)dapl_ib_remove_conn_listener(sp_ptr->header. + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + (void)dapl_ib_remove_conn_listener(sp_ptr->common. owner_ia, sp_ptr); dapl_ia_unlink_sp((struct dapl_ia *) - sp_ptr->header.owner_ia, + sp_ptr->common.owner_ia, sp_ptr); dapl_sp_dealloc(sp_ptr); dapl_cr_free(cr_ptr); goto skip_unlock; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); } else - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); skip_unlock: return ep_ptr; @@ -255,16 +244,6 @@ void dapl_cr_callback(struct dapl_cm_id ib_cm_handle, event, sp_ptr); /* - * The SP pointer could have been cleaned up in a racing - * CM callback, check to see if we should just exit here - */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - return; - } - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); - - /* * CONNECT_REQUEST events create an event on the PSP * EVD, which will trigger connection processing. The * sequence is: @@ -277,7 +256,7 @@ void dapl_cr_callback(struct dapl_cm_id */ if (event == DAT_CONNECTION_REQUEST_EVENT) { ep_ptr = NULL; - evd_ptr = sp_ptr->evd_handle; + evd_ptr = (struct dapl_evd *)sp_ptr->evd; } else { /* see if there is an EP connected with this CM handle */ ep_ptr = dapl_get_sp_ep(ib_cm_handle, sp_ptr, event); @@ -286,7 +265,7 @@ void dapl_cr_callback(struct dapl_cm_id if (ep_ptr == NULL) return; - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; /* if something has happened to our EVD, bail. */ if (evd_ptr == NULL) return; @@ -296,30 +275,29 @@ void dapl_cr_callback(struct dapl_cm_id switch (event) { case DAT_CONNECTION_REQUEST_EVENT: - { /* * Requests arriving on a disabled SP are immediately rejected */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); if (sp_ptr->listening == FALSE) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, "---> dapl_cr_callback: conn event on down SP\n"); (void)dapl_ib_reject_connection(ib_cm_handle); return; } - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_RSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_RSP) { /* * RSP connections only allow a single connection. Close * it down NOW so we reject any further connections. */ sp_ptr->listening = FALSE; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* * Only occurs on the passive side of a connection @@ -332,24 +310,20 @@ void dapl_cr_callback(struct dapl_cm_id evd_ptr = NULL; break; - } case DAT_CONNECTION_EVENT_ESTABLISHED: - { /* This is just a notification the connection is now * established, there isn't any private data to deal with. * * Update the EP state and cache a copy of the cm handle, * then let the user know we are ready to go. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->header.magic != DAPL_MAGIC_EP || - ep_ptr->param.ep_state != - DAT_EP_STATE_COMPLETION_PENDING) { + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING) { /* If someone pulled the plug on the EP or connection, * just exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; /* Set evd_ptr = NULL so we don't generate an event below */ evd_ptr = NULL; @@ -358,72 +332,63 @@ void dapl_cr_callback(struct dapl_cm_id } ep_ptr->param.ep_state = DAT_EP_STATE_CONNECTED; - ep_ptr->cm_handle = ib_cm_handle; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + ep_ptr->cm = ib_cm_handle; + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_DISCONNECTED: - { /* * EP is now fully disconnected; initiate any post processing * to reset the underlying QP and get the EP ready for * another connection */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { /* The disconnect has already occurred, we are now * cleaned up and ready to exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); return; } ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: - { /* * After posting an accept the requesting node has * stopped talking. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_BROKEN: - { - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } default: - { evd_ptr = NULL; dapl_os_assert(0); /* shouldn't happen */ break; } - } if (evd_ptr != NULL) dat_status = dapl_evd_post_connection_event(evd_ptr, event, - (DAT_HANDLE) ep_ptr, + (struct dat_ep *)ep_ptr, 0, NULL); if (dat_status != DAT_SUCCESS) { Index: linux-kernel/dat-provider/dapl_cr_accept.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_accept.c (revision 2572) +++ linux-kernel/dat-provider/dapl_cr_accept.c (working copy) @@ -39,42 +39,39 @@ * Establish a connection between active side (remote endpoint) * and passive side (local endpoint). */ -u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - int private_data_size, const void *private_data) +u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, + const void *private_data) { struct dapl_ep *ep_ptr; u32 dat_status; struct dapl_cr *cr_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_cr_accept (%p, %p, %d, %p)\n", - cr_handle, ep_handle, private_data_size, private_data); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_accept (%p, %p, %d, %p)\n", + cr, ep, private_data_size, private_data); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* - * Return an error if we have an ep_handle and the CR already has an + * Return an error if we have an ep and the CR already has an * EP, indicating this is an RSP connection or PSP_PROVIDER_FLAG was * specified. */ - if (ep_handle != NULL && - (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP) || - cr_ptr->param.local_ep_handle != NULL)) { + if (ep != NULL && cr_ptr->param.local_ep != NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* Make sure we have an EP handle in one place or another */ - if (ep_handle == NULL && cr_ptr->param.local_ep_handle == NULL) { + if (ep == NULL && cr_ptr->param.local_ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -86,84 +83,71 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand } /* - * ep_handle is NULL if the user specified DAT_PSP_PROVIDER_FLAG + * ep is NULL if the user specified DAT_PSP_PROVIDER_FLAG * OR this is an RSP connection; retrieve it from the cr. */ - if (ep_handle == NULL) { - ep_handle = cr_ptr->param.local_ep_handle; - if ((((struct dapl_ep *)ep_handle)->param.ep_state != + if (ep == NULL) { + ep = cr_ptr->param.local_ep; + if ((((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) - && (((struct dapl_ep *)ep_handle)->param.ep_state != + && (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING)) { return DAT_INVALID_STATE; } } else { /* ensure this EP isn't connected or in use */ - if (((struct dapl_ep *)ep_handle)->param.ep_state != + if (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_UNCONNECTED) { return DAT_INVALID_STATE; } } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required * We do all verification and state change under lock, at which * point the EP state should protect us from most races. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* The qp must be attached by this point! */ - if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) - { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } entry_ep_state = ep_ptr->param.ep_state; - entry_ep_handle = cr_ptr->param.local_ep_handle; + entry_ep = cr_ptr->param.local_ep; ep_ptr->param.ep_state = DAT_EP_STATE_COMPLETION_PENDING; - ep_ptr->cm_handle = cr_ptr->ib_cm_handle; + ep_ptr->cm = cr_ptr->ib_cm_handle; ep_ptr->cr_ptr = cr_ptr; ep_ptr->param.remote_ia_address_ptr = cr_ptr->param.remote_ia_address_ptr; - cr_ptr->param.local_ep_handle = ep_handle; + cr_ptr->param.local_ep = ep; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); - dat_status = dapl_ib_accept_connection(cr_handle, - ep_handle, + dat_status = dapl_ib_accept_connection(cr, + ep, private_data_size, private_data); /* @@ -176,16 +160,14 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand * system error, app termination, perhaps other reasons. */ dat_status = - dapl_evd_post_connection_event(ep_ptr->param. - request_evd_handle, + dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. + request_evd, DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR, - (DAT_HANDLE) ep_ptr, + (void *) ep_ptr, 0, NULL); - - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } else { ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; + cr_ptr->param.local_ep = entry_ep; ep_ptr->cr_ptr = NULL; ep_ptr->param.remote_ia_address_ptr = NULL; } @@ -199,14 +181,8 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); } - } else { - /* Make this CR invalid. We need to hang on to it until - * the connection terminates, but it's destroyed from - * the app point of view. - */ - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } - bail: +bail: return dat_status; } Index: linux-kernel/dat-provider/dapl_cr_handoff.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_handoff.c (revision 2572) +++ linux-kernel/dat-provider/dapl_cr_handoff.c (working copy) @@ -37,7 +37,7 @@ * Hand the connection request to another Sevice pont specified by the * Connectin Qualifier. */ -u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) +u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL cr_handoff) { /* handoff */ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); } Index: linux-kernel/dat-provider/dapl_pz.c =================================================================== --- linux-kernel/dat-provider/dapl_pz.c (revision 2572) +++ linux-kernel/dat-provider/dapl_pz.c (working copy) @@ -44,18 +44,13 @@ static struct dapl_pz *dapl_pz_alloc(str /* zero the structure */ memset(pz, 0, sizeof *pz); - /* - * initialize the header - */ - pz->header.provider = ia->header.provider; - pz->header.magic = DAPL_MAGIC_PZ; - pz->header.handle_type = DAT_HANDLE_TYPE_PZ; - pz->header.owner_ia = ia; - pz->header.user_context.as_64 = 0; - pz->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&pz->header.ia_list_entry); + pz->pz.provider = ia->ia.provider; + pz->common.owner_ia = ia; + pz->common.user_context.as_64 = 0; + pz->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&pz->common.ia_list_entry); dapl_ia_link_pz(ia, pz); - spin_lock_init(&pz->header.lock); + spin_lock_init(&pz->common.lock); /* * initialize the body @@ -67,83 +62,81 @@ static struct dapl_pz *dapl_pz_alloc(str static void dapl_pz_dealloc(struct dapl_pz *pz) { - /* reset magic to prevent reuse */ - pz->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_pz(pz->header.owner_ia, pz); + dapl_ia_unlink_pz(pz->common.owner_ia, pz); kfree(pz); } -u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) +u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; int ib_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", - ia_handle, pz_handle); + ia, pz); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error1; } - if (NULL == pz_handle) { + if (NULL == pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto error1; } - ia = (struct dapl_ia *)ia_handle; + dapl_ia = (struct dapl_ia *)ia; - pz = dapl_pz_alloc(ia); - if (NULL == pz) { + dapl_pz = dapl_pz_alloc(dapl_ia); + if (!dapl_pz) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - pz->pd = ib_alloc_pd(ia->hca_ptr->ib_hca_handle); - if (IS_ERR(pz->pd)) { - ib_status = PTR_ERR(pz->pd); + dapl_pz->pd = ib_alloc_pd(dapl_ia->hca_ptr->ib_hca_handle); + if (IS_ERR(dapl_pz->pd)) { + ib_status = PTR_ERR(dapl_pz->pd); dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_alloc_pd failed: %X\n", ib_status); status = dapl_ib_status_convert(ib_status); goto error2; } - *pz_handle = pz; + *pz = (struct dat_pz *)dapl_pz; return DAT_SUCCESS; error2: - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); error1: - *pz_handle = NULL; + *pz = NULL; return status; } -u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) +u32 dapl_pz_free(struct dat_pz *pz) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; int ib_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - if (0 != atomic_read(&pz->pz_ref_count)) { + if (0 != atomic_read(&dapl_pz->pz_ref_count)) { status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_PZ_IN_USE); goto error; } - ib_status = ib_dealloc_pd(pz->pd); + ib_status = ib_dealloc_pd(dapl_pz->pd); if (ib_status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_dealloc_pd failed: %X\n", ib_status); @@ -151,22 +144,22 @@ u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle goto error; } - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); return DAT_SUCCESS; error: return status; } -u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) +u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_param) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query(%p, %x, %p)\n", - pz_handle, pz_param); + pz, pz_param); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } @@ -175,8 +168,8 @@ u32 dapl_pz_query(DAT_PZ_HANDLE pz_handl goto error; } - pz = (struct dapl_pz *)pz_handle; - pz_param->ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; + dapl_pz = (struct dapl_pz *)pz; + pz_param->ia = (struct dat_ia *)dapl_pz->common.owner_ia; return DAT_SUCCESS; error: Index: linux-kernel/dat-provider/dapl_lmr.c =================================================================== --- linux-kernel/dat-provider/dapl_lmr.c (revision 2572) +++ linux-kernel/dat-provider/dapl_lmr.c (working copy) @@ -36,7 +36,7 @@ static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_desc, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags mem_priv) { struct dapl_lmr *lmr; @@ -49,27 +49,22 @@ static struct dapl_lmr *dapl_lmr_alloc(s /* zero the structure */ memset(lmr, 0, sizeof *lmr); - /* - * initialize the header - */ - lmr->header.provider = ia->header.provider; - lmr->header.magic = DAPL_MAGIC_LMR; - lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; - lmr->header.owner_ia = ia; - lmr->header.user_context.as_64 = 0; - lmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&lmr->header.ia_list_entry); + lmr->lmr.provider = ia->ia.provider; + lmr->common.owner_ia = ia; + lmr->common.user_context.as_64 = 0; + lmr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&lmr->common.ia_list_entry); dapl_ia_link_lmr(ia, lmr); - spin_lock_init(&lmr->header.lock); + spin_lock_init(&lmr->common.lock); /* * initialize the body */ - lmr->param.ia_handle = (DAT_IA_HANDLE) ia; + lmr->param.ia = (struct dat_ia *)ia; lmr->param.mem_type = mem_type; lmr->param.region_desc = region_desc; lmr->param.length = length; - lmr->param.pz_handle = pz_handle; + lmr->param.pz = pz; lmr->param.mem_priv = mem_priv; atomic_set(&lmr->lmr_ref_count, 0); @@ -78,9 +73,7 @@ static struct dapl_lmr *dapl_lmr_alloc(s static void dapl_lmr_dealloc(struct dapl_lmr *lmr) { - /* reset magic to prevent reuse */ - lmr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); + dapl_ia_unlink_lmr(lmr->common.owner_ia, lmr); kfree(lmr); } @@ -88,49 +81,50 @@ static void dapl_lmr_dealloc(struct dapl static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, u64 length, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; reg_desc.for_va = virt_addr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, + (struct dat_pz *)pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register(ia, lmr, virt_addr, length, privileges); + status = dapl_ib_mr_register(ia, new_lmr, virt_addr, length, + privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64)(unsigned long) virt_addr; if (registered_length) @@ -139,9 +133,9 @@ static inline u32 dapl_lmr_create_virtua return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; @@ -151,26 +145,26 @@ static inline u32 dapl_lmr_create_physic DAT_REGION_DESCRIPTION phys_addr, u64 page_count, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; u64 *array = phys_addr.for_array; u32 status; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, page_count, - (DAT_PZ_HANDLE) pz, privileges); + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, + page_count, (struct dat_pz *) pz, privileges); - if (NULL == lmr) { + if (NULL == new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, + status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr.for_array, page_count, privileges); if (DAT_SUCCESS != status) @@ -178,23 +172,23 @@ static inline u32 dapl_lmr_create_physic /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = array[0]; if (registered_length) @@ -203,9 +197,9 @@ static inline u32 dapl_lmr_create_physic return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } @@ -214,13 +208,13 @@ static inline u32 dapl_lmr_create_lmr(st struct dapl_lmr *original_lmr, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; @@ -230,39 +224,39 @@ static inline u32 dapl_lmr_create_lmr(st if (status != DAT_SUCCESS) goto error1; - reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; + reg_desc.for_lmr = (struct dat_lmr *) original_lmr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, + (struct dat_pz *) pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_shared(ia, lmr, privileges); + status = dapl_ib_mr_register_shared(ia, new_lmr, privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64) (unsigned long) original_lmr->param.region_desc.for_va; @@ -272,73 +266,72 @@ static inline u32 dapl_lmr_create_lmr(st return DAT_SUCCESS; error3: - dapl_ib_mr_deregister(lmr); + dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } -u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags privileges, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - u64 *registered_length, u64 *registered_address) + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, + u64 *registered_address) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", - ia_handle, mem_type); + ia, mem_type); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - ia = (struct dapl_ia *)ia_handle; - pz = (struct dapl_pz *)pz_handle; + dapl_ia = (struct dapl_ia *)ia; + dapl_pz = (struct dapl_pz *)pz; switch (mem_type) { case DAT_MEM_TYPE_VIRTUAL: - status = dapl_lmr_create_virtual(ia, region_description.for_va, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_virtual(dapl_ia, + region_description.for_va, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_PHYSICAL: - status = dapl_lmr_create_physical(ia, region_description, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_physical(dapl_ia, region_description, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_LMR: { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; - if (DAPL_BAD_HANDLE(region_description.for_lmr_handle, - DAPL_MAGIC_LMR)) { + if (!region_description.for_lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto bail; } - lmr = (struct dapl_lmr *)region_description.for_lmr_handle; + dapl_lmr = (struct dapl_lmr *)region_description.for_lmr; - status = dapl_lmr_create_lmr(ia, lmr, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_lmr(dapl_ia, dapl_lmr, dapl_pz, + privileges, lmr, lmr_context, rmr_context, registered_length, registered_address); break; @@ -357,45 +350,45 @@ bail: return status; } -u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) +u32 dapl_lmr_free(struct dat_lmr *lmr) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } - lmr = (struct dapl_lmr *)lmr_handle; + dapl_lmr = (struct dapl_lmr *)lmr; - switch (lmr->param.mem_type) { + switch (dapl_lmr->param.mem_type) { case DAT_MEM_TYPE_PHYSICAL: case DAT_MEM_TYPE_VIRTUAL: case DAT_MEM_TYPE_LMR: { struct dapl_pz *pz; - if (0 != atomic_read(&lmr->lmr_ref_count)) + if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) return DAT_INVALID_STATE; status = dapl_hash_remove( - lmr->header.owner_ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + dapl_lmr->common.owner_ia->hca_ptr->lmr_hash_table, + dapl_lmr->param.lmr_context, NULL); if (status != DAT_SUCCESS) goto error; - status = dapl_ib_mr_deregister(lmr); + status = dapl_ib_mr_deregister(dapl_lmr); if (status == DAT_SUCCESS) { - pz = (struct dapl_pz *)lmr->param.pz_handle; + pz = (struct dapl_pz *)dapl_lmr->param.pz; atomic_dec(&pz->pz_ref_count); - dapl_lmr_dealloc(lmr); - } else /* failure; put lmr back in hash table */ - dapl_hash_insert(lmr->header.owner_ia-> + dapl_lmr_dealloc(dapl_lmr); + } else /* failure; put dapl_lmr back in hash table */ + dapl_hash_insert(dapl_lmr->common.owner_ia-> hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + dapl_lmr->param.lmr_context, dapl_lmr); break; } @@ -412,15 +405,15 @@ error: return status; } -u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) +u32 dapl_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *lmr_param) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", - lmr_handle, lmr_param); + lmr, lmr_param); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } @@ -429,8 +422,8 @@ u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_ha goto error; } - lmr = (struct dapl_lmr *)lmr_handle; - memcpy(lmr_param, &lmr->param, sizeof *lmr_param); + dapl_lmr = (struct dapl_lmr *)lmr; + memcpy(lmr_param, &dapl_lmr->param, sizeof *lmr_param); status = DAT_SUCCESS; error: return status; @@ -441,7 +434,7 @@ error: * will sync the cache with memory. */ -u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -449,11 +442,11 @@ u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDL u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_read(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } @@ -462,7 +455,7 @@ error: return status; } -u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -470,11 +463,11 @@ u32 dapl_lmr_sync_rdma_write(DAT_IA_HAND u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_write(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } Index: linux-kernel/dat-provider/dapl_get_handle_type.c =================================================================== --- linux-kernel/dat-provider/dapl_get_handle_type.c (revision 2572) +++ linux-kernel/dat-provider/dapl_get_handle_type.c (working copy) @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_get_handle_type - * - * Gets the handle type for the given dat_handle - * - * Input: - * dat_handle - * - * Output: - * handle_type - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ - -u32 dapl_get_handle_type(DAT_HANDLE dat_handle, enum dat_handle_type *type) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - *type = header->handle_type; - -bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_get_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_get_consumer_context.c (revision 2572) +++ linux-kernel/dat-provider/dapl_get_consumer_context.c (working copy) @@ -1,80 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_get_consumer_context - * - * DAPL Requirements Version xxx, 6.2.2.2 - * - * Gets the consumer context from the specified dat_object - * - * Input: - * dat_handle - * - * Output: - * context - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ -u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, union dat_context *context) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - if (context == NULL || ((unsigned long)(header) & 3)) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - *context = header->user_context; - - bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_rmr.c =================================================================== --- linux-kernel/dat-provider/dapl_rmr.c (revision 2572) +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) @@ -61,24 +61,19 @@ static struct dapl_rmr *dapl_rmr_alloc(s /* zero the structure */ memset(rmr, 0, sizeof *rmr); - /* - * initialize the header - */ - rmr->header.provider = pz->header.provider; - rmr->header.magic = DAPL_MAGIC_RMR; - rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; - rmr->header.owner_ia = pz->header.owner_ia; - rmr->header.user_context.as_64 = 0; - rmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&rmr->header.ia_list_entry); - dapl_ia_link_rmr(rmr->header.owner_ia, rmr); - spin_lock_init(&rmr->header.lock); + rmr->rmr.provider = pz->pz.provider; + rmr->common.owner_ia = pz->common.owner_ia; + rmr->common.user_context.as_64 = 0; + rmr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&rmr->common.ia_list_entry); + dapl_ia_link_rmr(rmr->common.owner_ia, rmr); + spin_lock_init(&rmr->common.lock); /* * initialize the body */ - rmr->param.ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; - rmr->param.pz_handle = (DAT_PZ_HANDLE)pz; + rmr->param.ia = (struct dat_ia *)pz->common.owner_ia; + rmr->param.pz = (struct dat_pz *)pz; rmr->param.lmr_triplet.lmr_context = 0; rmr->param.lmr_triplet.pad = 0; rmr->param.lmr_triplet.virtual_address = 0; @@ -93,11 +88,8 @@ static struct dapl_rmr *dapl_rmr_alloc(s static void dapl_rmr_dealloc(struct dapl_rmr *rmr) { - /* reset magic to prevent reuse */ - rmr->header.magic = DAPL_MAGIC_INVALID; - - dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); - /* no need to destroy rmr->header.lock */ + dapl_ia_unlink_rmr(rmr->common.owner_ia, rmr); + /* no need to destroy rmr->common.lock */ kfree(rmr); } @@ -115,7 +107,7 @@ static inline u32 dapl_rmr_bind_fuse(str u32 status; boolean_t is_signaled; - status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, + status = dapl_hash_search(rmr->common.owner_ia->hca_ptr->lmr_hash_table, lmr_triplet->lmr_context, (DAPL_HASH_DATA *) &lmr); if (DAT_SUCCESS != status) { @@ -150,8 +142,8 @@ static inline u32 dapl_rmr_bind_fuse(str } /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ - if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || - (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { + if ((ep_ptr->param.pz != lmr->param.pz) || + (ep_ptr->param.pz != rmr->param.pz)) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -240,7 +232,7 @@ static inline u32 dapl_rmr_bind_unfuse(s } /* If the RMR and EP are not in the same PZ, there is an error */ - if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { + if (ep_ptr->param.pz != rmr->param.pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail1; } @@ -306,9 +298,9 @@ bail1: * Input: * Output: */ -u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, const struct dat_lmr_triplet *lmr_triplet, - enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, + enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) @@ -316,14 +308,14 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han struct dapl_rmr *rmr; struct dapl_ep *ep_ptr; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) + if (!rmr_handle) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) + if (!ep) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); rmr = (struct dapl_rmr *)rmr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* if the rmr should be bound */ if (0 != lmr_triplet->segment_length) @@ -341,7 +333,7 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * Create a remote memory region for the specified protection zone * * Input: - * pz_handle + * pz * * Output: * rmr_handle @@ -351,20 +343,20 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) +u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr) { - struct dapl_pz *pz; - struct dapl_rmr *rmr; + struct dapl_pz *dapl_pz; + struct dapl_rmr *dapl_rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - rmr = dapl_rmr_alloc(pz); + dapl_rmr = dapl_rmr_alloc(dapl_pz); if (rmr == NULL) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, @@ -372,18 +364,18 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han goto bail; } - status = dapl_ib_mw_alloc(rmr); + status = dapl_ib_mw_alloc(dapl_rmr); if (status != DAT_SUCCESS) { - dapl_rmr_dealloc(rmr); + dapl_rmr_dealloc(dapl_rmr); status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY_REGION); goto bail; } - atomic_inc(&pz->pz_ref_count); + atomic_inc(&dapl_pz->pz_ref_count); - *rmr_handle = rmr; + *rmr = (struct dat_rmr *)dapl_rmr; bail: return status; @@ -404,12 +396,12 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) +u32 dapl_rmr_free(struct dat_rmr *rmr_handle) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } @@ -438,12 +430,12 @@ bail: return status; } -u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) +u32 dapl_rmr_query(struct dat_rmr *rmr_handle, struct dat_rmr_param *rmr_param) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } Index: linux-kernel/dat-provider/dapl_cr_util.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_util.c (revision 2572) +++ linux-kernel/dat-provider/dapl_cr_util.c (working copy) @@ -49,17 +49,12 @@ struct dapl_cr *dapl_cr_alloc(struct dap /* zero the structure */ memset(cr_ptr, 0, sizeof *cr_ptr); - /* - * initialize the header - */ - cr_ptr->header.provider = ia_ptr->header.provider; - cr_ptr->header.magic = DAPL_MAGIC_CR; - cr_ptr->header.handle_type = DAT_HANDLE_TYPE_CR; - cr_ptr->header.owner_ia = ia_ptr; - cr_ptr->header.user_context.as_64 = 0; - cr_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&cr_ptr->header.ia_list_entry); - spin_lock_init(&cr_ptr->header.lock); + cr_ptr->cr.provider = ia_ptr->ia.provider; + cr_ptr->common.owner_ia = ia_ptr; + cr_ptr->common.user_context.as_64 = 0; + cr_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&cr_ptr->common.ia_list_entry); + spin_lock_init(&cr_ptr->common.lock); return cr_ptr; } @@ -69,9 +64,5 @@ struct dapl_cr *dapl_cr_alloc(struct dap */ void dapl_cr_free(struct dapl_cr *cr_ptr) { - dapl_os_assert(cr_ptr->header.magic == DAPL_MAGIC_CR || - cr_ptr->header.magic == DAPL_MAGIC_CR_DESTROYED); - /* reset magic to prevent reuse */ - cr_ptr->header.magic = DAPL_MAGIC_INVALID; kfree(cr_ptr); } Index: linux-kernel/dat-provider/dapl_psp.c =================================================================== --- linux-kernel/dat-provider/dapl_psp.c (revision 2572) +++ linux-kernel/dat-provider/dapl_psp.c (working copy) @@ -45,13 +45,13 @@ * the user. * * Input: - * ia_handle - * evd_handle + * ia + * evd * psp_flags * * Output: * conn_qual - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -61,9 +61,9 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ struct dapl_ia *ia_ptr; @@ -72,19 +72,19 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia u32 status = DAT_SUCCESS; int i; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } @@ -93,7 +93,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -117,10 +117,10 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; - sp_ptr->evd_handle = evd_handle; + sp_ptr->ia = ia; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -153,7 +153,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia if (status != DAT_SUCCESS) { atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -168,7 +168,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia * Return handle to the user */ *conn_qual = sp_ptr->conn_qual; - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -183,13 +183,13 @@ bail: * in a notification event. * * Input: - * ia_handle + * ia * conn_qual - * evd_handle + * evd * psp_flags * * Output: - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -198,9 +198,9 @@ bail: * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -208,24 +208,24 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -261,11 +261,11 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -298,7 +298,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han * wrong! Decrements the EVD refcount & release it. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -313,7 +313,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -325,7 +325,7 @@ bail: * Destroy a specific instance of a Service Point. * * Input: - * psp_handle + * psp * * Output: * none @@ -334,26 +334,25 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) +u32 dapl_psp_free(struct dat_sp *psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; DAPL_SP_STATE save_state; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if it has been established * and there are no current connections in progress. @@ -361,18 +360,18 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * container until the last connection is disconnected, after * which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); sp_ptr->listening = FALSE; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -389,8 +388,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han sp_ptr->cr_list_count == 0) { save_state = sp_ptr->state; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -407,7 +406,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free: PSP PENDING\n"); } @@ -416,13 +416,12 @@ bail: return status; } -u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) +u32 dapl_psp_query(struct dat_sp *psp, struct dat_psp_param *psp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || - ((struct dapl_sp *)psp_handle)->listening != TRUE) { + if (!psp || ((struct dapl_sp *)psp)->listening != TRUE) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } @@ -432,11 +431,11 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha goto bail; } - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; - psp_param->ia_handle = sp_ptr->ia_handle; + psp_param->ia = sp_ptr->ia; psp_param->conn_qual = sp_ptr->conn_qual; - psp_param->evd_handle = sp_ptr->evd_handle; + psp_param->evd = sp_ptr->evd; psp_param->psp_flags = sp_ptr->psp_flags; status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2572) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -43,11 +43,6 @@ static void dapl_ep_dealloc(struct dapl_ep *ep_ptr) { - dapl_os_assert(ep_ptr->header.magic == DAPL_MAGIC_EP); - - /* reset magic to prevent reuse */ - ep_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_cb_free(&ep_ptr->req_buffer); dapl_cb_free(&ep_ptr->recv_buffer); @@ -73,18 +68,12 @@ struct dapl_ep *dapl_ep_alloc(struct dap /* zero the structure */ memset(ep_ptr, 0, sizeof *ep_ptr + sizeof (struct sockaddr)); - /* - * initialize the header - */ - ep_ptr->header.provider = ia_ptr->header.provider; - ep_ptr->header.magic = DAPL_MAGIC_EP; - ep_ptr->header.handle_type = DAT_HANDLE_TYPE_EP; - ep_ptr->header.owner_ia = ia_ptr; - ep_ptr->header.user_context.as_64 = 0; - ep_ptr->header.user_context.as_ptr = NULL; - - dapl_llist_init_entry(&ep_ptr->header.ia_list_entry); - spin_lock_init(&ep_ptr->header.lock); + ep_ptr->ep.provider = ia_ptr->ia.provider; + ep_ptr->common.owner_ia = ia_ptr; + ep_ptr->common.user_context.as_64 = 0; + ep_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&ep_ptr->common.ia_list_entry); + spin_lock_init(&ep_ptr->common.lock); /* * Initialize the body @@ -105,7 +94,7 @@ struct dapl_ep *dapl_ep_alloc(struct dap ep_ptr->qp = NULL; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - ep_ptr->cm_handle = NULL; + ep_ptr->cm = NULL; atomic_set(&ep_ptr->req_count, 0); atomic_set(&ep_ptr->recv_count, 0); @@ -176,7 +165,7 @@ static u32 dapl_ep_check_recv_completion return DAT_SUCCESS; } -static u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, +static u32 dapl_ep_post_send_req(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -188,13 +177,13 @@ static u32 dapl_ep_post_send_req(DAT_EP_ struct dapl_cookie *cookie; u32 dat_status; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Synchronization ok since this buffer is only used for send @@ -292,11 +281,11 @@ enum dat_return_subtype dapl_ep_state_su return dat_status; } -u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) +u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + const struct dat_ep_attr *ep_attr, struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -305,18 +294,18 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create (%p, %p, %p, %p, %p, %p, %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -328,44 +317,34 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != DAT_HANDLE_NULL && - DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -381,12 +360,12 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -424,25 +403,25 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (recv_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_recv_dtos = 0; - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (request_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_request_dtos = 0; /* @@ -451,9 +430,9 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -461,7 +440,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -471,36 +450,36 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free - * evd_handles: + * pz: dat_pz_free + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = (struct dat_ep *)ep_ptr; bail: return dat_status; } -u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +u32 dapl_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *ep_attr, - DAT_EP_HANDLE *ep_handle) + struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -509,18 +488,18 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create_with_srq (%p, %p, %p, %p, %p, %p, %p %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, srq_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, srq, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -532,35 +511,27 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)-> + evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); @@ -571,13 +542,13 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * Verify the SRQ handle. It is an error to invoke this call with * a NULL handle */ - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -593,12 +564,12 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -647,26 +618,26 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (recv_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_recv_dtos = 0; } - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (request_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_request_dtos = 0; } @@ -676,9 +647,9 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -686,7 +657,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -696,24 +667,24 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free, - * evd_handles: + * pz: dat_pz_free, + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = ep_ptr; #endif /* notdef */ @@ -721,7 +692,7 @@ bail: return dat_status; } -u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_free(struct dat_ep *ep) { struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; @@ -729,15 +700,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle enum ib_qp_state save_qp_state; u32 dat_status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; param = &ep_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -755,7 +726,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle goto bail; } - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* If we are connected, issue a disconnect. If we are in the * disconnect_pending state, disconnect with the ABRUPT flag @@ -765,14 +736,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle /* * Invoke ep_disconnect to clean up outstanding connections */ - (void)dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + (void)dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); dapl_os_assert(ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_UNCONNECTED); /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); @@ -782,24 +754,24 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle * to set handles to NULL. Set handles to NULL so this routine * is idempotent. */ - if (param->pz_handle != NULL) { - atomic_dec(&((struct dapl_pz *)param->pz_handle)->pz_ref_count); - param->pz_handle = NULL; + if (param->pz != NULL) { + atomic_dec(&((struct dapl_pz *)param->pz)->pz_ref_count); + param->pz = NULL; } - if (param->recv_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->recv_evd_handle)-> + if (param->recv_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->recv_evd)-> evd_ref_count); - param->recv_evd_handle = NULL; + param->recv_evd = NULL; } - if (param->request_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->request_evd_handle)-> + if (param->request_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->request_evd)-> evd_ref_count); - param->request_evd_handle = NULL; + param->request_evd = NULL; } - if (param->connect_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->connect_evd_handle)-> + if (param->connect_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->connect_evd)-> evd_ref_count); - param->connect_evd_handle = NULL; + param->connect_evd = NULL; } /* @@ -815,7 +787,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ save_qp_state = ep_ptr->qp_state; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Free the QP. If the EP has never been used, the QP is invalid */ if (save_qp_state != DAPL_QP_STATE_UNATTACHED) { @@ -839,7 +811,7 @@ bail: } -u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, @@ -850,7 +822,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_connect(%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", - ep_handle, + ep, remote_ia_address->sa_data[2], remote_ia_address->sa_data[3], remote_ia_address->sa_data[4], @@ -858,19 +830,19 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han remote_conn_qual, timeout, private_data_size, private_data, qos, connect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state. The connection handle must be good * at this point. */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD)) { + if (!ep_ptr->param.connect_evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; @@ -900,9 +872,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * * Assign the QP and release the lock. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ)) - { + if (ep_ptr->param.pz == NULL) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); @@ -910,7 +880,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } alloc_ep = *ep_ptr; - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, &alloc_ep, ep_ptr); if (dat_status != DAT_SUCCESS) { dat_status = @@ -919,15 +889,15 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han goto bail; } - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * PZ shouldn't have changed since we're only racing with * dapl_cr_accept() */ if (ep_ptr->qp_state != DAPL_QP_STATE_UNATTACHED) { /* Bail, cleaning up. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = dapl_ib_qp_free(&alloc_ep); if (dat_status != DAT_SUCCESS) { dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -943,42 +913,30 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->qp = alloc_ep.qp; ep_ptr->qp_state = alloc_ep.qp_state; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); } /* * We do state checks and transitions under lock. * The only code we're racing against is dapl_cr_accept. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required - */ - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL - && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required + */ + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); goto bail; @@ -988,12 +946,12 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * we need to attach one now. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_TEP); @@ -1002,8 +960,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1014,8 +972,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han /* * At this point we only support one QOS level */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_MODEL_NOT_SUPPORTED, 0); goto bail; } @@ -1029,7 +987,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * At this point we're committed, and done with the endpoint * except for the connect, so we can drop the lock. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Copy the connection qualifiers */ memcpy(ep_ptr->param.remote_ia_address_ptr, remote_ia_address, @@ -1037,7 +995,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->param.remote_port_qual = remote_conn_qual; memcpy(ep_ptr->private_data,private_data,private_data_size); - dat_status = dapl_ib_connect(ep_ptr, remote_ia_address, + dat_status = dapl_ib_connect((struct dat_ep *)ep_ptr, remote_ia_address, remote_conn_qual, timeout, private_data_size, ep_ptr->private_data); @@ -1054,10 +1012,10 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han if (dat_status == DAT_ERROR(DAT_INTERNAL_ERROR, 1)) { dapl_evd_post_connection_event((struct dapl_evd *) ep_ptr->param. - connect_evd_handle, + connect_evd, DAT_CONNECTION_EVENT_UNREACHABLE, - (DAT_HANDLE) ep_ptr, 0, - NULL); + (struct dat_ep *)ep_ptr, + 0, NULL); dat_status = DAT_SUCCESS; } } @@ -1069,7 +1027,7 @@ bail: return dat_status; } -u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, +u32 dapl_ep_dup_connect(struct dat_ep *ep, struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { @@ -1078,13 +1036,13 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep DAT_CONN_QUAL remote_conn_qual; u32 dat_status; - ep_dup_ptr = (struct dapl_ep *)ep_dup_handle; + ep_dup_ptr = (struct dapl_ep *)ep_dup; /* * Verify the dup handle, which must be connected. All other * parameters will be verified by dapl_ep_connect */ - if (DAPL_BAD_HANDLE(ep_dup_handle, DAPL_MAGIC_EP)) { + if (!ep_dup) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1100,10 +1058,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep * there is a problem. Do this under a lock and pull out * the connection parameters for atomicity. */ - spin_lock_irqsave(&ep_dup_ptr->header.lock, ep_dup_ptr->header.flags); + spin_lock_irqsave(&ep_dup_ptr->common.lock, ep_dup_ptr->common.flags); if (ep_dup_ptr->param.ep_state != DAT_EP_STATE_CONNECTED) { - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_dup_ptr)); @@ -1111,10 +1069,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep } remote_ia_address_ptr = ep_dup_ptr->param.remote_ia_address_ptr; remote_conn_qual = ep_dup_ptr->param.remote_port_qual; - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); - dat_status = dapl_ep_connect(ep_handle, + dat_status = dapl_ep_connect(ep, remote_ia_address_ptr, remote_conn_qual, timeout, @@ -1125,7 +1083,7 @@ bail: return dat_status; } -u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags disconnect_flags) { struct dapl_ep *ep_ptr; @@ -1135,14 +1093,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_disconnect (%p, %x)\n", - ep_handle, disconnect_flags); + ep, disconnect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1152,12 +1110,12 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * Do the verification of parameters and the state change * atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Disconnecting a disconnected EP is a no-op. */ if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; goto bail; } @@ -1171,8 +1129,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ ep_ptr->param.ep_state != DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECT_PENDING) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1184,8 +1142,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * If in state DISCONNECT_PENDING then this must be an * ABRUPT disconnect */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } @@ -1198,18 +1156,19 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ */ ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); /* disconnect and make sure we get no callbacks */ (void)dapl_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); /* clean up connection state */ dapl_sp_remove_ep(ep_ptr); - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; dapl_evd_post_connection_event(evd_ptr, DAT_CONNECTION_EVENT_DISCONNECTED, - (DAT_HANDLE) ep_ptr, 0, NULL); + (struct dat_ep *)ep_ptr, 0, + NULL); dat_status = DAT_SUCCESS; goto bail; } @@ -1227,14 +1186,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECT_PENDING; } - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); dat_status = dapl_ib_disconnect(ep_ptr, disconnect_flags); /* * Reacquire the lock and make sure we didn't get a callback * that cleaned up. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (disconnect_flags == DAT_CLOSE_ABRUPT_FLAG && ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { /* @@ -1245,8 +1204,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * on the way out. */ cr_ptr = ep_ptr->cr_ptr; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); if (cr_ptr != NULL) { dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, @@ -1257,13 +1216,13 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ DAT_CONNECTION_EVENT_DISCONNECTED, NULL, cr_ptr->sp_ptr); } else { - dapl_evd_connection_callback(ep_ptr->cm_handle, + dapl_evd_connection_callback(ep_ptr->cm, DAT_CONNECTION_EVENT_DISCONNECTED, NULL, ep_ptr); } } else - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, @@ -1272,7 +1231,7 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ return dat_status; } -u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, enum dat_ep_state *ep_state, +u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle) { struct dapl_ep *ep_ptr; @@ -1280,14 +1239,14 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_get_status (%p, %p, %p, %p)\n", - ep_handle, ep_state, in_dto_idle, out_dto_idle); + ep, ep_state, in_dto_idle, out_dto_idle); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1315,15 +1274,15 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ } static inline u32 -dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, +dapl_ep_modify_validate_parameters(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param, struct dapl_ia **ia_ptr, struct dapl_ep **ep_ptr, struct dat_ep_attr *ep_attr_ptr) { - struct dapl_ia *ia; - struct dapl_ep *ep; + struct dapl_ia *dapl_ia; + struct dapl_ep *dapl_ep; struct dat_ep_attr ep_attr; struct dat_ep_attr ep_attr_limit; struct dat_ep_attr ep_attr_request; @@ -1332,14 +1291,14 @@ dapl_ep_modify_validate_parameters(DAT_E *ia_ptr = NULL; *ep_ptr = NULL; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep = (struct dapl_ep *)ep_handle; - ia = ep->header.owner_ia; + dapl_ep = (struct dapl_ep *)ep; + dapl_ia = dapl_ep->common.owner_ia; /* * Verify parameters valid in current EP state @@ -1359,10 +1318,10 @@ dapl_ep_modify_validate_parameters(DAT_E * TENTATIVE_CONNECTION_PENDING (psp PROVIDER allocated EP) */ if ((mask & DAT_EP_FIELD_PZ_HANDLE) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } @@ -1378,41 +1337,21 @@ dapl_ep_modify_validate_parameters(DAT_E DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS | DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV | DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV)) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_RESERVED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_RESERVED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } /* * Validate handles being modified */ - if (mask & DAT_EP_FIELD_PZ_HANDLE) { - if (ep_param->pz_handle != NULL && - DAPL_BAD_HANDLE(ep_param->pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { - if (ep_param->recv_evd_handle != NULL && - (DAPL_BAD_HANDLE(ep_param->recv_evd_handle, DAPL_MAGIC_EVD) - || !((struct dapl_evd *)ep_param->recv_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - - if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) { - if (ep_param->request_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->request_evd_handle, - DAPL_MAGIC_EVD)) { + if (ep_param->recv_evd != NULL && + !((struct dapl_evd *)ep_param->recv_evd)-> + evd_flags & DAT_EVD_DTO_FLAG) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; @@ -1420,10 +1359,8 @@ dapl_ep_modify_validate_parameters(DAT_E } if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) { - if (ep_param->connect_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->connect_evd_handle, - DAPL_MAGIC_EVD) - && !(((struct dapl_evd *)ep_param->connect_evd_handle)-> + if (ep_param->connect_evd != NULL && + !(((struct dapl_evd *)ep_param->connect_evd)-> evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1434,8 +1371,8 @@ dapl_ep_modify_validate_parameters(DAT_E /* * Validate the attributes against the HCA limits */ - ep_attr = ep->param.ep_attr; - ep_attr_limit = ia->hca_ptr->ep_attr; + ep_attr = dapl_ep->param.ep_attr; + ep_attr_limit = dapl_ia->hca_ptr->ep_attr; ep_attr_request = ep_param->ep_attr; if (mask & DAT_EP_FIELD_EP_ATTR_MAX_MESSAGE_SIZE) { @@ -1482,7 +1419,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS) { if (ep_attr_request.max_recv_dtos > ep_attr_limit.max_recv_dtos - || (ep_param->recv_evd_handle == DAT_HANDLE_NULL + || (ep_param->recv_evd == NULL && (ep_attr_request.max_recv_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1495,7 +1432,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS) { if (ep_attr_request.max_request_dtos > ep_attr_limit.max_request_dtos - || (ep_param->request_evd_handle == DAT_HANDLE_NULL + || (ep_param->request_evd == NULL && (ep_attr_request.max_request_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1552,15 +1489,15 @@ dapl_ep_modify_validate_parameters(DAT_E } } - *ia_ptr = ia; - *ep_ptr = ep; + *ia_ptr = dapl_ia; + *ep_ptr = dapl_ep; *ep_attr_ptr = ep_attr; bail: return dat_status; } -u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, +u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param) { struct dapl_ia *ia; @@ -1586,7 +1523,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand boolean_t rqst_iov_used = FALSE; boolean_t recv_iov_used = FALSE; - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep1, &ep_attr1); @@ -1600,16 +1537,16 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep = *ep1; alloc_ep.param.ep_attr = ep_attr1; if (mask & DAT_EP_FIELD_PZ_HANDLE) - alloc_ep.param.pz_handle = ep_param->pz_handle; + alloc_ep.param.pz = ep_param->pz; if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) - alloc_ep.param.recv_evd_handle = ep_param->recv_evd_handle; + alloc_ep.param.recv_evd = ep_param->recv_evd; if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) - alloc_ep.param.request_evd_handle = ep_param->request_evd_handle; + alloc_ep.param.request_evd = ep_param->request_evd; if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) - alloc_ep.param.connect_evd_handle = ep_param->connect_evd_handle; + alloc_ep.param.connect_evd = ep_param->connect_evd; /* * Allocate everything that might be needed. @@ -1673,18 +1610,18 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * There's a lot of work done here, but there should be no * allocation or blocking. */ - spin_lock_irqsave(&ep1->header.lock, ep1->header.flags); + spin_lock_irqsave(&ep1->common.lock, ep1->common.flags); /* * Revalidate parameters; make sure that races haven't * changed anything important. */ - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep2, &ep_attr2); if (DAT_SUCCESS != dat_status) { - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); goto bail; } @@ -1713,10 +1650,10 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * is stuff changed by dapl_cr_accept, and neither PZ nor EVD is in that * list. */ - new_ep.param.pz_handle = alloc_ep.param.pz_handle; - new_ep.param.recv_evd_handle = alloc_ep.param.recv_evd_handle; - new_ep.param.request_evd_handle = alloc_ep.param.request_evd_handle; - new_ep.param.connect_evd_handle = alloc_ep.param.connect_evd_handle; + new_ep.param.pz = alloc_ep.param.pz; + new_ep.param.recv_evd = alloc_ep.param.recv_evd; + new_ep.param.request_evd = alloc_ep.param.request_evd; + new_ep.param.connect_evd = alloc_ep.param.connect_evd; /* Deal with each of the allocation fields. */ if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS @@ -1778,50 +1715,50 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand if (DAPL_QP_STATE_UNATTACHED != new_ep.qp_state) { dat_status = dapl_ib_qp_modify(ia, ep2, &ep_attr2); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep2->header.lock, - ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, + ep2->common.flags); goto bail; } } *ep2 = new_ep; - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); /* * Modify reference counts, incrementing new ones * and then decrementing old ones (so if they're the same * the refcount never drops to zero). */ - tmp_pz = (struct dapl_pz *)new_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)new_ep.param.pz; if (tmp_pz) atomic_inc(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.request_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); /* decreament the old reference counts */ - tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz; if (tmp_pz) atomic_dec(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); @@ -1869,7 +1806,7 @@ bail: return dat_status; } -u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1880,12 +1817,12 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_read (%p, %d, %p, %p, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1900,7 +1837,7 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE return dat_status; } -u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1910,12 +1847,12 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_write (%p, %d, %p, %P, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1930,7 +1867,7 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL return dat_status; } -u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1941,15 +1878,15 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_recv (%p, %d, %p, %P, %x)\n", - ep_handle, num_segments, local_iov, user_cookie.as_64, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *) ep_handle; + ep_ptr = (struct dapl_ep *) ep; /* * Synchronization ok since this buffer is only used for receive @@ -1979,7 +1916,7 @@ bail: return status; } -u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1989,11 +1926,11 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_send (%p, %d, %p, %P, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -2007,21 +1944,21 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h return dat_status; } -u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) +u32 dapl_ep_query(struct dat_ep *ep, struct dat_ep_param *ep_param) { struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_query (%p, %p)\n", - ep_handle, ep_param); + ep, ep_param); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } @@ -2037,12 +1974,10 @@ u32 dapl_ep_query(DAT_EP_HANDLE ep_handl * N.B. Just slam all values into the user structure, there * is nothing to be gained by checking for each bit. */ - if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { + if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) /* obtain the remote IP address */ - status = - dapl_ib_cm_remote_addr((DAT_HANDLE) ep_handle, - &ep_ptr->remote_ia_address); - } + status = dapl_ib_cm_remote_addr(ep, &ep_ptr->remote_ia_address); + ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *)&ep_ptr->remote_ia_address; *ep_param = ep_ptr->param; @@ -2051,21 +1986,21 @@ bail: return status; } -u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, int *nbufs_allocate, +u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocate, int *bufs_alloc_span) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", - ep_handle, nbufs_allocate, bufs_alloc_span); + ep, nbufs_allocate, bufs_alloc_span); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2078,17 +2013,17 @@ bail: } -u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_reset(struct dat_ep *ep) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2110,7 +2045,7 @@ bail: return dat_status; } -u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -2118,14 +2053,14 @@ u32 dapl_ep_set_watermark(DAT_EP_HANDLE u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", - ep_handle, soft_high_watermark, hard_high_watermark); + ep, soft_high_watermark, hard_high_watermark); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; Index: linux-kernel/dat-provider/dapl_rsp.c =================================================================== --- linux-kernel/dat-provider/dapl_rsp.c (revision 2572) +++ linux-kernel/dat-provider/dapl_rsp.c (working copy) @@ -44,13 +44,13 @@ * event * * Input: - * ia_handle + * ia * conn_qual - * ep_handle - * evd_handle + * ep + * evd * * Output: - * rsp_handle + * rsp * * Returns: * DAT_SUCCESS @@ -59,9 +59,9 @@ * DAT_INVALID_STATE * DAT_CONN_QUAL_IN_USE */ -u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle) +u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -70,39 +70,39 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free conn_qual: %x EP: %p\n", - conn_qual, ep_handle); + conn_qual, ep); - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (rsp_handle == NULL) { + if (rsp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -126,11 +126,11 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Fill out the RSP args */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = 0; - sp_ptr->ep_handle = ep_handle; + sp_ptr->ep = ep; /* * Take a reference on the EVD handle @@ -162,7 +162,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han * the state to FREE, so we know the call failed. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; sp_ptr->state = DAPL_SP_STATE_FREE; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -178,7 +178,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *rsp_handle = (DAT_RSP_HANDLE)sp_ptr; + *rsp = (struct dat_sp *)sp_ptr; bail: return status; @@ -190,7 +190,7 @@ bail: * Destroy a specific instance of a Reserved Service Point. * * Input: - * rsp_handle + * rsp * * Output: * none @@ -199,50 +199,49 @@ bail: * DAT_SUCCESS * DAT_INVALID_HANDLE */ -u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) +u32 dapl_rsp_free(struct dat_sp *rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp_handle); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_RSP)) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp); + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if there are no connections. If * we defer removing the sp it becomes something of a zombie * container until disconnection, after which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * Make sure we don't leave a dangling EP. If the state is still * RESERVED then the RSP still owns it. */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -258,8 +257,8 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han /* This RSP has never been used. Clean it up */ sp_ptr->listening = FALSE; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -275,20 +274,20 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); } bail: return status; } -u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) +u32 dapl_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *rsp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { + if (!rsp) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } @@ -298,15 +297,15 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha goto bail; } - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Fill in the RSP params */ - rsp_param->ia_handle = sp_ptr->ia_handle; + rsp_param->ia = sp_ptr->ia; rsp_param->conn_qual = sp_ptr->conn_qual; - rsp_param->evd_handle = sp_ptr->evd_handle; - rsp_param->ep_handle = sp_ptr->ep_handle; + rsp_param->evd = sp_ptr->evd; + rsp_param->ep = sp_ptr->ep; status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_set_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_set_consumer_context.c (revision 2572) +++ linux-kernel/dat-provider/dapl_set_consumer_context.c (working copy) @@ -1,74 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -/* - * dapl_set_consumer_context - * - * Set a consumer context in the provided dat_handle - * - * Input: - * dat_handle - * context - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - */ -u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, union dat_context context) -{ - u32 dat_status = DAT_SUCCESS; - struct dapl_header *header; - - header = (struct dapl_header *)dat_handle; - if (((header) == NULL) || - ((unsigned long)(header) & 3) || - (header->magic != DAPL_MAGIC_IA && - header->magic != DAPL_MAGIC_EVD && - header->magic != DAPL_MAGIC_EP && - header->magic != DAPL_MAGIC_LMR && - header->magic != DAPL_MAGIC_RMR && - header->magic != DAPL_MAGIC_PZ && - header->magic != DAPL_MAGIC_PSP && - header->magic != DAPL_MAGIC_RSP && - header->magic != DAPL_MAGIC_CR)) { - dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - header->user_context = context; - - bail: - return dat_status; -} Index: linux-kernel/dat-provider/dapl_ia_open.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_open.c (revision 2572) +++ linux-kernel/dat-provider/dapl_ia_open.c (working copy) @@ -45,18 +45,18 @@ * and calls this function. */ u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr) + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr) { u32 dat_status = DAT_SUCCESS; struct dat_provider *provider; struct dapl_hca *hca_ptr = NULL; - struct dapl_ia *ia_ptr = NULL; - struct dapl_evd *evd_ptr; + struct dapl_ia *ia = NULL; + struct dapl_evd *evd; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_open (%s, %d, %p, %p)\n", - name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); + name, async_evd_qlen, async_evd_ptr, ia_ptr); dat_status = dapl_provider_list_search(name, &provider); if (DAT_SUCCESS != dat_status) { @@ -64,25 +64,25 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - /* ia_handle_ptr and async_evd_handle_ptr cannot be NULL */ - if (ia_handle_ptr == NULL) { + /* ia_ptr and async_evd_ptr cannot be NULL */ + if (ia_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } - if (async_evd_handle_ptr == NULL) { + if (async_evd_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; } /* initialize the caller's param */ - *ia_handle_ptr = DAT_HANDLE_NULL; + *ia_ptr = NULL; hca_ptr = (struct dapl_hca *)provider->extension; atomic_inc(&hca_ptr->handle_ref_count); /* Allocate and initialize ia structure */ - ia_ptr = dapl_ia_alloc(provider, hca_ptr); - if (!ia_ptr) { + ia = dapl_ia_alloc(provider, hca_ptr); + if (!ia) { atomic_dec(&hca_ptr->handle_ref_count); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -93,17 +93,16 @@ u32 dapl_ia_open(const char *name, int a * use the one passed in (if non-NULL) or create one */ - evd_ptr = (struct dapl_evd *)*async_evd_handle_ptr; - if (evd_ptr) { - if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || - !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { + evd = (struct dapl_evd *)*async_evd_ptr; + if (evd) { + if (!(evd->evd_flags & DAT_EVD_ASYNC_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - if (evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle != + if (evd->common.owner_ia->hca_ptr->ib_hca_handle != hca_ptr->ib_hca_handle) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, @@ -111,8 +110,8 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - ia_ptr->cleanup_async_error_evd = FALSE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = FALSE; + ia->async_error_evd = evd; } else { /* Verify we have >0 length, and let the provider check the size */ if (async_evd_qlen <= 0) { @@ -120,32 +119,33 @@ u32 dapl_ia_open(const char *name, int a DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } - dat_status = dapl_evd_internal_create(ia_ptr, async_evd_qlen, + dat_status = dapl_evd_internal_create(ia, + async_evd_qlen, DAT_UPCALL_DISABLE, NULL, DAT_EVD_ASYNC_FLAG, - &evd_ptr); + &evd); if (dat_status != DAT_SUCCESS) goto bail; - atomic_inc(&evd_ptr->evd_ref_count); + atomic_inc(&evd->evd_ref_count); /* Register the handlers associated with the async EVD. */ - dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); + dat_status = dapl_ia_setup_callbacks(ia, evd); /* Assign the EVD so it gets cleaned up */ - ia_ptr->cleanup_async_error_evd = TRUE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = TRUE; + ia->async_error_evd = evd; if (dat_status != DAT_SUCCESS) goto bail; } dat_status = DAT_SUCCESS; - *ia_handle_ptr = ia_ptr; - *async_evd_handle_ptr = evd_ptr; + *ia_ptr = (struct dat_ia *)ia; + *async_evd_ptr = (struct dat_evd *)evd; - bail: - if (dat_status != DAT_SUCCESS) { - if (ia_ptr) - dapl_ia_close(ia_ptr, DAT_CLOSE_ABRUPT_FLAG); - } +bail: + if (dat_status != DAT_SUCCESS) + if (ia) + dapl_ia_close((struct dat_ia *)ia, + DAT_CLOSE_ABRUPT_FLAG); dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_ia_open () returns 0x%x\n", dat_status); Index: linux-kernel/dat-provider/dapl_mr_util.c =================================================================== --- linux-kernel/dat-provider/dapl_mr_util.c (revision 2572) +++ linux-kernel/dat-provider/dapl_mr_util.c (working copy) @@ -59,7 +59,7 @@ u64 dapl_mr_get_address(DAT_REGION_DESCR case DAT_MEM_TYPE_VIRTUAL: return (u64) (unsigned long) desc.for_va; case DAT_MEM_TYPE_LMR: - lmr = (struct dapl_lmr *)desc.for_lmr_handle; + lmr = (struct dapl_lmr *)desc.for_lmr; /* Since this function is recoursive we cannot inline it */ return dapl_mr_get_address(lmr->param.region_desc, Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2572) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -51,7 +51,7 @@ static void dapl_evd_upcall_trigger(stru } for (;;) { - status = dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); + status = dapl_evd_dequeue((struct dat_evd *)evd, &event); if (DAT_SUCCESS != status) return; @@ -157,17 +157,12 @@ struct dapl_evd *dapl_evd_alloc(struct d /* zero the structure */ memset(evd, 0, sizeof *evd); - /* - * initialize the header - */ - evd->header.provider = ia->header.provider; - evd->header.magic = DAPL_MAGIC_EVD; - evd->header.handle_type = DAT_HANDLE_TYPE_EVD; - evd->header.owner_ia = ia; - evd->header.user_context.as_64 = 0; - evd->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&evd->header.ia_list_entry); - spin_lock_init(&evd->header.lock); + evd->evd.provider = ia->ia.provider; + evd->common.owner_ia = ia; + evd->common.user_context.as_64 = 0; + evd->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&evd->common.ia_list_entry); + spin_lock_init(&evd->common.lock); /* * Initialize the body @@ -249,7 +244,6 @@ static u32 dapl_evd_dealloc(struct dapl_ u32 status = DAT_SUCCESS; struct dapl_ia *ia; - dapl_os_assert(evd->header.magic == DAPL_MAGIC_EVD); dapl_os_assert(atomic_read(&evd->evd_ref_count) == 0); /* @@ -257,7 +251,7 @@ static u32 dapl_evd_dealloc(struct dapl_ * up from it. */ if (evd->cq != NULL) { - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; status = dapl_ib_cq_free(evd); if (status != DAT_SUCCESS) { @@ -265,12 +259,6 @@ static u32 dapl_evd_dealloc(struct dapl_ } } - /* - * We should now be safe to invalidate the EVD; reset the - * magic to prevent reuse. - */ - evd->header.magic = DAPL_MAGIC_INVALID; - /* If the ring buffer allocation failed, then the dapl_rbuf_destroy */ /* function will detect that the ring buffer's internal data (ex. base */ /* pointer) are invalid and will handle the situation appropriately */ @@ -314,14 +302,14 @@ static struct dat_event *dapl_evd_get_ev struct dat_event *event; if (evd->evd_producer_locking_needed) { - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); } event = (struct dat_event *) dapl_rbuf_remove(&evd->free_event_queue); /* Release the lock if it was taken and the call failed. */ if (!event && evd->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); return event; } @@ -348,8 +336,8 @@ static void dapl_evd_post_event(struct d dapl_os_assert(evd->evd_state == DAPL_EVD_STATE_OPEN); if (evd->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd->header.lock, - evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, + evd->common.flags); dapl_evd_upcall_trigger(evd); } @@ -363,12 +351,12 @@ static void dapl_evd_format_overflow_eve { struct dapl_ia *ia; - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; event->event_number = DAT_ASYNC_ERROR_EVD_OVERFLOW; event->event_data.asynch_error_event_data.dat_handle = - (DAT_HANDLE) ia; + (struct dat_ia *)ia; } /* @@ -409,9 +397,9 @@ static struct dat_event *dapl_evd_get_an event = dapl_evd_get_event(evd); if (!event) dapl_evd_post_overflow_event( - evd->header.owner_ia->async_error_evd, evd); + evd->common.owner_ia->async_error_evd, evd); else { - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; event->event_number = evno; } @@ -419,10 +407,10 @@ static struct dat_event *dapl_evd_get_an } u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp, struct sockaddr *ia_address, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle) + struct dat_cr *cr) { struct dat_event *event; event = dapl_evd_get_and_init_event(evd, DAT_CONNECTION_REQUEST_EVENT); @@ -436,11 +424,11 @@ u32 dapl_evd_post_cr_arrival_event(struc return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event->event_data.cr_arrival_event_data.sp_handle = sp_handle; + event->event_data.cr_arrival_event_data.sp = sp; event->event_data.cr_arrival_event_data.local_ia_address_ptr = ia_address; event->event_data.cr_arrival_event_data.conn_qual = conn_qual; - event->event_data.cr_arrival_event_data.cr_handle = cr_handle; + event->event_data.cr_arrival_event_data.cr = cr; dapl_evd_post_event(evd, event); @@ -449,7 +437,7 @@ u32 dapl_evd_post_cr_arrival_event(struc u32 dapl_evd_post_connection_event(struct dapl_evd *evd, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, int private_data_size, void *private_data) { @@ -466,7 +454,7 @@ u32 dapl_evd_post_connection_event(struc DAT_RESOURCE_MEMORY); } - event->event_data.connect_event_data.ep_handle = ep_handle; + event->event_data.connect_event_data.ep = ep; event->event_data.connect_event_data.private_data_size = private_data_size; event->event_data.connect_event_data.private_data = private_data; @@ -492,7 +480,7 @@ static u32 dapl_evd_post_async_error_eve return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event->event_data.asynch_error_event_data.dat_handle = (DAT_HANDLE) ia; + event->event_data.asynch_error_event_data.dat_handle = ia; dapl_evd_post_event(evd, event); @@ -539,12 +527,8 @@ static void dapl_evd_wc_to_event(struct ep = cookie->ep; dapl_os_assert(NULL != ep); - if (ep->header.magic != DAPL_MAGIC_EP) { - /* ep may have been freed, just return */ - return; - } - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; switch (cookie->type) { case DAPL_COOKIE_TYPE_DTO: @@ -563,7 +547,7 @@ static void dapl_evd_wc_to_event(struct } event->event_number = DAT_DTO_COMPLETION_EVENT; - event_data->ep_handle = cookie->ep; + event_data->ep = (struct dat_ep *)cookie->ep; event_data->user_cookie = cookie->val.dto.cookie; event_data->status = dto_status; @@ -606,7 +590,7 @@ static void dapl_evd_wc_to_event(struct event->event_number = DAT_RMR_BIND_COMPLETION_EVENT; event_data = &event->event_data.rmr_completion_event_data; - event_data->rmr_handle = cookie->val.rmr.rmr; + event_data->rmr = (struct dat_rmr *)cookie->val.rmr.rmr; event_data->user_cookie = cookie->val.rmr.cookie; if (dto_status == DAT_DTO_SUCCESS) { @@ -654,7 +638,7 @@ void dapl_evd_qp_async_error_callback(st u32 status; ep = (struct dapl_ep *)context; - async_evd = (struct dapl_evd *)ep->header.owner_ia->async_error_evd; + async_evd = (struct dapl_evd *)ep->common.owner_ia->async_error_evd; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback: " @@ -679,7 +663,7 @@ void dapl_evd_qp_async_error_callback(st * we are not interested in. */ (void)dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback () returns\n"); @@ -698,11 +682,11 @@ void dapl_evd_cq_async_error_callback(st if (!evd) dapl_os_panic("NULL == context\n"); - async_evd = evd->header.owner_ia->async_error_evd; + async_evd = evd->common.owner_ia->async_error_evd; status = dapl_evd_post_async_error_event(async_evd, DAT_ASYNC_ERROR_EVD_OVERFLOW, - async_evd->header.owner_ia); + async_evd->common.owner_ia); if (status != DAT_SUCCESS) { dapl_os_panic("async EVD overflow\n"); @@ -738,7 +722,7 @@ void dapl_evd_un_async_error_callback(st */ dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_un_async_error_callback () returns\n"); @@ -764,14 +748,14 @@ void dapl_evd_connection_callback(struct "ep: %p\n", cm_handle, event, ep); - evd = (struct dapl_evd *)ep->param.connect_evd_handle; + evd = (struct dapl_evd *)ep->param.connect_evd; private_data_size = 0; /* * All operations effect the EP, so lock it once and unlock * when necessary */ - spin_lock_irqsave(&ep->header.lock, ep->header.flags); + spin_lock_irqsave(&ep->common.lock, ep->common.flags); switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: @@ -784,13 +768,13 @@ void dapl_evd_connection_callback(struct /* If someone pulled the plug on the connection, just * exit */ - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); status = DAT_SUCCESS; break; } ep->param.ep_state = DAT_EP_STATE_CONNECTED; - ep->cm_handle = cm_handle; + ep->cm = cm_handle; if (private_data == NULL) { private_data_size = 0; } else { @@ -798,7 +782,7 @@ void dapl_evd_connection_callback(struct memcpy(ep->private_data, private_data, private_data_size); } - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -811,8 +795,8 @@ void dapl_evd_connection_callback(struct */ ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); break; } @@ -820,7 +804,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -828,7 +812,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -836,7 +820,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -844,14 +828,14 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, FALSE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } case DAT_CONNECTION_REQUEST_EVENT: default: { - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); evd = NULL; dapl_os_assert(0); /* shouldn't happen */ @@ -865,7 +849,7 @@ void dapl_evd_connection_callback(struct */ if (evd != NULL) { status = dapl_evd_post_connection_event(evd, event, - (DAT_HANDLE) ep, + (struct dat_ep *)ep, private_data_size, /* 0 except on CONNECT */ ep->private_data); @@ -877,10 +861,10 @@ void dapl_evd_connection_callback(struct * expect a callback on an ABRUPT disconnect. */ dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); - spin_lock_irqsave(&ep->header.lock, ep->header.flags); + spin_lock_irqsave(&ep->common.lock, ep->common.flags); ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); } } @@ -901,7 +885,6 @@ void dapl_evd_dto_callback(struct ib_cq evd = (struct dapl_evd *)user_context; dapl_os_assert(evd->cq == cq); - dapl_os_assert(evd->header.magic == DAPL_MAGIC_EVD); /* Read once. */ state = *(volatile DAPL_EVD_STATE *)&evd->evd_state; @@ -927,9 +910,9 @@ void dapl_evd_dto_callback(struct ib_cq status = dapl_ib_completion_notify(evd); if (DAT_SUCCESS != status) (void)dapl_evd_post_async_error_event( - evd->header.owner_ia->async_error_evd, + evd->common.owner_ia->async_error_evd, DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR, - evd->header.owner_ia); + evd->common.owner_ia); dapl_evd_upcall_trigger(evd); } @@ -1008,10 +991,10 @@ bail: return status; } -u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int min_qlen, +u32 dapl_evd_kcreate(struct dat_ia *ia_handle, int min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object *upcall, - enum dat_evd_flags flags, DAT_EVD_HANDLE *evd_handle) + enum dat_evd_flags flags, struct dat_evd **evd_handle) { struct dapl_ia *ia; struct dapl_evd *evd; @@ -1025,7 +1008,7 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha evd = NULL; *evd_handle = NULL; - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -1053,17 +1036,17 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha if (status != DAT_SUCCESS) goto bail; - *evd_handle = (DAT_EVD_HANDLE) evd; + *evd_handle = (struct dat_evd *)evd; bail: if (status != DAT_SUCCESS) if (evd) - dapl_evd_free(evd); + dapl_evd_free((struct dat_evd *)evd); return status; } -u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) +u32 dapl_evd_free(struct dat_evd *evd_handle) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; @@ -1072,7 +1055,7 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1083,11 +1066,11 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han goto bail; } - dapl_ia_unlink_evd(evd->header.owner_ia, evd); + dapl_ia_unlink_evd(evd->common.owner_ia, evd); status = dapl_evd_dealloc(evd); if (status != DAT_SUCCESS) { - dapl_ia_link_evd(evd->header.owner_ia, evd); + dapl_ia_link_evd(evd->common.owner_ia, evd); } bail: @@ -1097,7 +1080,7 @@ bail: return status; } -u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) +u32 dapl_evd_kquery(struct dat_evd *evd_handle, struct dat_evd_param *evd_param) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; @@ -1108,11 +1091,11 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h } /* Note: the spec. allows for events to be directed to a NULL EVD */ - /* with handle of type DAT_HANDLE_NULL. See 6.3.1 */ - if (DAT_HANDLE_NULL == evd_handle) { + /* with handle of type NULL. See 6.3.1 */ + if (NULL == evd_handle) { memset(evd_param, 0, sizeof *evd_param); } else { - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1128,7 +1111,7 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h * about reading the state variable atomically when we add * in waitable/unwaitable. */ - evd_param->ia_handle = evd->header.owner_ia; + evd_param->ia = (struct dat_ia *)evd->common.owner_ia; evd_param->evd_qlen = evd->qlen; evd_param->evd_flags = evd->evd_flags; evd_param->upcall_policy = evd->upcall_policy; @@ -1139,7 +1122,7 @@ bail: return status; } -u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +u32 dapl_evd_modify_upcall(struct dat_evd *evd_handle, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object *upcall) { @@ -1150,7 +1133,7 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1162,14 +1145,14 @@ bail: return status; } -u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) +u32 dapl_evd_post_se(struct dat_evd *evd_handle, const struct dat_event *event) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1196,7 +1179,7 @@ u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_ return status; } -u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) +u32 dapl_evd_dequeue(struct dat_evd *evd_handle, struct dat_event *event) { struct dapl_evd *evd; struct dat_event *local_event; @@ -1207,7 +1190,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1222,7 +1205,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ * restricts us from having multiple threads in CQ poll, and the * DAT API allows multiple threads in dat_evd_dequeue() */ - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); /* * Make sure there are no other waiters and the evd is active. @@ -1230,7 +1213,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ */ if (evd->evd_state != DAPL_EVD_STATE_OPEN || evd->catastrophic_overflow) { - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); status = DAT_ERROR(DAT_INVALID_STATE, 0); goto bail; } @@ -1249,7 +1232,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ status = DAT_ERROR(DAT_QUEUE_EMPTY, 0); } - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_dequeue () returns 0x%x\n", status); @@ -1257,7 +1240,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ return status; } -u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) +u32 dapl_evd_resize(struct dat_evd *evd_handle, int evd_qlen) { struct dapl_ia *ia; struct dapl_evd *evd; @@ -1267,13 +1250,13 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", evd_handle, evd_qlen); - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } evd = (struct dapl_evd *)evd_handle; - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; if (evd_qlen == evd->qlen) { status = DAT_SUCCESS; @@ -1285,7 +1268,7 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; } - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); pend_cnt = dapl_rbuf_count(&evd->pending_event_queue); if (pend_cnt > evd_qlen) { @@ -1302,6 +1285,6 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; bail: - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); return status; } Index: linux-kernel/dat-provider/dapl_ia_query.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_query.c (revision 2572) +++ linux-kernel/dat-provider/dapl_ia_query.c (working copy) @@ -38,12 +38,12 @@ * Provide the consumer with Interface Adapter and Provider parameters. * * Input: - * ia_handle + * ia * ia_mask * provider_mask * * Output: - * async_evd_handle + * async_evd * ia_parameters * provider_parameters * @@ -51,7 +51,7 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE *async_evd_handle, +u32 dapl_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -66,21 +66,21 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_query (%p, %p, %p, %p)\n", - ia_handle, - async_evd_handle, + ia, + async_evd, ia_attr, provider_attr); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (NULL != async_evd_handle) { - *async_evd_handle = ia_ptr->async_error_evd; + if (NULL != async_evd) { + *async_evd = (struct dat_evd *)ia_ptr->async_error_evd; } if (NULL != ia_attr) @@ -88,7 +88,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl if (NULL != provider_attr) { strncpy(provider_attr->provider_name, - ia_ptr->header.provider->device_name, + ia_ptr->ia.provider->device_name, DAT_NAME_MAX_LENGTH); provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; Index: linux-kernel/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel/dat-provider/dapl_sp_util.c (revision 2572) +++ linux-kernel/dat-provider/dapl_sp_util.c (working copy) @@ -65,22 +65,16 @@ struct dapl_sp *dapl_sp_alloc(struct dap /* zero the structure */ memset(sp_ptr, 0, sizeof *sp_ptr); - /* - * initialize the header - */ - sp_ptr->header.provider = ia_ptr->header.provider; - if (is_psp) { - sp_ptr->header.magic = DAPL_MAGIC_PSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_PSP; - } else { - sp_ptr->header.magic = DAPL_MAGIC_RSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_RSP; - } - sp_ptr->header.owner_ia = ia_ptr; - sp_ptr->header.user_context.as_64 = 0; - sp_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&sp_ptr->header.ia_list_entry); - spin_lock_init(&sp_ptr->header.lock); + sp_ptr->sp.provider = ia_ptr->ia.provider; + if (is_psp) + sp_ptr->sp.type = DAT_SP_TYPE_PSP; + else + sp_ptr->sp.type = DAT_SP_TYPE_RSP; + sp_ptr->common.owner_ia = ia_ptr; + sp_ptr->common.user_context.as_64 = 0; + sp_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&sp_ptr->common.ia_list_entry); + spin_lock_init(&sp_ptr->common.lock); /* * Initialize the Body (set to NULL above) @@ -107,12 +101,8 @@ struct dapl_sp *dapl_sp_alloc(struct dap */ void dapl_sp_dealloc(struct dapl_sp *sp_ptr) { - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); dapl_os_assert(dapl_llist_is_empty(&sp_ptr->cr_list_head)); - sp_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - kfree(sp_ptr); } @@ -134,11 +124,11 @@ void dapl_sp_dealloc(struct dapl_sp *sp_ */ void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) { - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_llist_add_tail(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry, cr_ptr); + &cr_ptr->common.ia_list_entry, cr_ptr); sp_ptr->cr_list_count++; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, sp_ptr->common.flags); } /* @@ -174,7 +164,7 @@ struct dapl_cr *dapl_sp_search_cr(struct if (cr_ptr->ib_cm_handle == ib_cm_handle) return cr_ptr; - cr_ptr = cr_ptr->header.ia_list_entry.flink->data; + cr_ptr = cr_ptr->common.ia_list_entry.flink->data; } while ((void *)cr_ptr != (void *)sp_ptr->cr_list_head->data); return NULL; @@ -208,7 +198,7 @@ void dapl_sp_remove_cr(struct dapl_sp *s } dapl_llist_remove_entry(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); sp_ptr->cr_list_count--; } @@ -238,13 +228,13 @@ void dapl_sp_remove_ep(struct dapl_ep *e if (cr_ptr != NULL) { sp_ptr = cr_ptr->sp_ptr; - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* Remove the CR from the queue */ dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); Index: linux-kernel/dat-provider/dapl_evd.h =================================================================== --- linux-kernel/dat-provider/dapl_evd.h (revision 2572) +++ linux-kernel/dat-provider/dapl_evd.h (working copy) @@ -60,14 +60,14 @@ extern void dapl_evd_un_async_error_call */ extern u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp_handle, struct sockaddr *ia_address, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle); + struct dat_cr *cr_handle); extern u32 dapl_evd_post_connection_event(struct dapl_evd *evd, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep_handle, int private_data_size, void *private_data); Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2572) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -46,23 +46,6 @@ * * *********************************************************************/ -typedef enum dapl_magic { - /* magic number values for verification & debug */ - DAPL_MAGIC_IA = 0xCafeF00d, - DAPL_MAGIC_EVD = 0xFeedFace, - DAPL_MAGIC_EP = 0xDeadBabe, - DAPL_MAGIC_LMR = 0xBeefCafe, - DAPL_MAGIC_RMR = 0xABadCafe, - DAPL_MAGIC_PZ = 0xDeafBeef, - DAPL_MAGIC_PSP = 0xBeadeD0c, - DAPL_MAGIC_RSP = 0xFab4Feed, - DAPL_MAGIC_SRQ = 0xC001Babe, - DAPL_MAGIC_CR = 0xBe12Cee1, - DAPL_MAGIC_CR_DESTROYED = 0xB12bDead, - DAPL_MAGIC_CNO = 0xDeadF00d, - DAPL_MAGIC_INVALID = 0xFFFFFFFF -} DAPL_MAGIC; - typedef enum dapl_evd_state { DAPL_EVD_STATE_OPEN, DAPL_EVD_STATE_DEAD = 0xDEAD @@ -92,23 +75,6 @@ typedef enum dapl_evd_completion { /********************************************************************* * * - * Macros * - * * - *********************************************************************/ - -/* - * Simple macro to verify a handle is bad. Conditions: - * - pointer is NULL - * - pointer is not word aligned - * - pointer's magic number is wrong - */ -#define DAPL_BAD_HANDLE(h, magicNum) ( \ - ((h) == NULL) || \ - ((unsigned long)(h) & 3) || \ - (((struct dapl_header *)(h))->magic != (magicNum))) - -/********************************************************************* - * * * Typedefs * * * *********************************************************************/ @@ -170,11 +136,8 @@ struct dapl_hca { struct dat_ep_attr ep_attr; }; -/* DAPL Objects always have the following header */ -struct dapl_header { - struct dat_provider *provider; /* required by DAT - must be first */ - DAPL_MAGIC magic; /* magic number for verification */ - enum dat_handle_type handle_type; +/* DAPL Objects always have the following items */ +struct dapl_common { struct dapl_ia *owner_ia; struct dapl_llist_entry ia_list_entry; union dat_context user_context; /* user context - opaque to DAPL */ @@ -182,9 +145,9 @@ struct dapl_header { unsigned long flags; /* saved lock flag values */ }; -/* struct dapl_ia maps to DAT_IA_HANDLE */ struct dapl_ia { - struct dapl_header header; /* WARNING: must be first */ + struct dat_ia ia; + struct dapl_common common; struct dapl_hca *hca_ptr; struct dapl_evd *async_error_evd; boolean_t cleanup_async_error_evd; @@ -200,9 +163,9 @@ struct dapl_ia { struct dapl_llist_entry *srq_list_head; /* SRQ queue */ }; -/* struct dapl_evd maps to DAT_EVD_HANDLE */ struct dapl_evd { - struct dapl_header header; /* WARNING: must be first */ + struct dat_evd evd; + struct dapl_common common; DAPL_EVD_STATE evd_state; enum dat_evd_flags evd_flags; @@ -230,9 +193,9 @@ struct dapl_evd { struct dat_upcall_object upcall; }; -/* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { - struct dapl_header header; + struct dat_ep ep; + struct dapl_common common; /* What the DAT Consumer asked for */ struct dat_ep_param param; @@ -241,7 +204,7 @@ struct dapl_ep { enum ib_qp_state qp_state; /* communications manager handle */ - struct dapl_cm_id *cm_handle; + struct dapl_cm_id *cm; /* store the remote IA address here, reference from the param * struct which only has a pointer, no storage */ @@ -267,33 +230,33 @@ struct dapl_ep { int send_iov_num; }; -/* struct dapl_srq maps to DAT_SRQ_HANDLE */ struct dapl_srq { - struct dapl_header header; /* WARNING: must be first */ + struct dat_srq srq; + struct dapl_common common; struct dat_srq_param param; atomic_t srq_ref_count; struct dapl_cookie_buffer recv_buffer; atomic_t recv_count; }; -/* struct dapl_pz maps to DAT_PZ_HANDLE */ struct dapl_pz { - struct dapl_header header; /* WARNING: must be first */ + struct dat_pz pz; + struct dapl_common common; struct ib_pd *pd; atomic_t pz_ref_count; }; -/* struct dapl_lmr maps to DAT_LMR_HANDLE */ struct dapl_lmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_lmr lmr; + struct dapl_common common; struct dat_lmr_param param; struct ib_mr *mr; atomic_t lmr_ref_count; }; -/* struct dapl_rmr maps to DAT_RMR_HANDLE */ struct dapl_rmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_rmr rmr; + struct dapl_common common; struct dat_rmr_param param; struct dapl_ep *ep; struct dapl_pz *pz; @@ -310,17 +273,17 @@ typedef enum dapl_sp_state { DAPL_SP_STATE_RSP_PENDING } DAPL_SP_STATE; -/* struct dapl_sp maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ struct dapl_sp { - struct dapl_header header; /* WARNING: must be first */ + struct dat_sp sp; + struct dapl_common common; DAPL_SP_STATE state; /* type and queue of the SP */ /* PSP/RSP PARAM fields */ - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; /* maintenance fields */ boolean_t listening; /* PSP is registered & active */ @@ -329,9 +292,9 @@ struct dapl_sp { int cr_list_count; /* count of CRs on queue */ }; -/* struct dapl_cr maps to DAT_CR_HANDLE */ struct dapl_cr { - struct dapl_header header; + struct dat_cr cr; + struct dapl_common common; /* for convenience the data is kept as a struct dat_cr_param. * however, the "local_endpoint" field is always NULL * so this wastes a pointer. This is probably ok to @@ -396,91 +359,88 @@ struct dapl_cookie { */ extern u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr); + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr); -extern u32 dapl_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dapl_ia_close(struct dat_ia *, enum dat_close_flags); -extern u32 dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, +extern u32 dapl_ia_query(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); /* helper functions */ -extern u32 dapl_set_consumer_context(DAT_HANDLE handle, +extern u32 dapl_set_consumer_context(void *handle, union dat_context context); -extern u32 dapl_get_consumer_context(DAT_HANDLE handle, +extern u32 dapl_get_consumer_context(void *handle, union dat_context *context); -extern u32 dapl_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type); - /* CR Functions */ -extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); +extern u32 dapl_cr_query(struct dat_cr *, struct dat_cr_param *); -extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +extern u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data); -extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); +extern u32 dapl_cr_reject(struct dat_cr *cr); -extern u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL handoff); +extern u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff); /* EVD Functions */ -extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, +extern u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, u64 *suggested_alignment); -extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, +extern u32 dapl_evd_kcreate(struct dat_ia *ia, int evd_min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE * evd_handle); + struct dat_evd ** evd); -extern u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_kquery(struct dat_evd *evd, struct dat_evd_param *evd_args); -extern u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall); -extern u32 dapl_evd_enable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_enable(struct dat_evd *evd); -extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_disable(struct dat_evd *evd); -extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen); +extern u32 dapl_evd_resize(struct dat_evd *evd, int evd_qlen); -extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_post_se(struct dat_evd *evd, const struct dat_event *event); -extern u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event); +extern u32 dapl_evd_dequeue(struct dat_evd *evd, struct dat_event *event); -extern u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_free(struct dat_evd *evd); -extern u32 dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_set_unwaitable(struct dat_evd *evd); -extern u32 dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_clear_unwaitable(struct dat_evd *evd); /* EP functions */ -extern u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE in_dto_completion_evd_handle, - DAT_EVD_HANDLE out_dto_completion_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, +extern u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_dto_completion_evd, + struct dat_evd *out_dto_completion_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *ep_parameters, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); +extern u32 dapl_ep_query(struct dat_ep *, struct dat_ep_param *); -extern u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask ep_args_mask, const struct dat_ep_param *ep_args); -extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, @@ -489,161 +449,161 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); -extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, +extern u32 dapl_ep_dup_connect(struct dat_ep *ep, + struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); -extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags close_flags); -extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle); -extern u32 dapl_ep_free(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_free(struct dat_ep *ep); -extern u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_reset(struct dat_ep *ep); -extern u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_ep_create_with_srq(struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq_handle, const struct dat_ep_attr *ep_attributes, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocated, int *bufs_alloc_span); -extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark); /* LMR functions */ -extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +extern u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address); -extern u32 dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); +extern u32 dapl_lmr_query(struct dat_lmr *, struct dat_lmr_param *); -extern u32 dapl_lmr_free(DAT_LMR_HANDLE); +extern u32 dapl_lmr_free(struct dat_lmr *); -extern u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); -extern u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); /* RMR Functions */ -extern u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle); +extern u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr); -extern u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_query(struct dat_rmr *rmr, struct dat_rmr_param *rmr_args); -extern u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_bind(struct dat_rmr *rmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *context); -extern u32 dapl_rmr_free(DAT_RMR_HANDLE); +extern u32 dapl_rmr_free(struct dat_rmr *rmr); /* PSP Functions */ -extern u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, +extern u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, +extern u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); +extern u32 dapl_psp_query(struct dat_sp *, struct dat_psp_param *); -extern u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle); +extern u32 dapl_psp_free(struct dat_sp *psp); /* RSP Functions */ -extern u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle); +extern u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp); -extern u32 dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); +extern u32 dapl_rsp_query(struct dat_sp *, struct dat_rsp_param *); -extern u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle); +extern u32 dapl_rsp_free(struct dat_sp *rsp); /* PZ Functions */ -extern u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle); +extern u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz); -extern u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_args); +extern u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_args); -extern u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle); +extern u32 dapl_pz_free(struct dat_pz *pz); /* SRQ functions */ -extern u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, +extern u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *srq_attr, - DAT_SRQ_HANDLE *srq_handle); + struct dat_srq **srq); -extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); +extern u32 dapl_srq_free(struct dat_srq *srq); -extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +extern u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie); -extern u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param); -extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto); -extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark); +extern u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark); /* * DAPL internal utility function prototypes Index: linux-kernel/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_util.c (revision 2572) +++ linux-kernel/dat-provider/dapl_ia_util.c (working copy) @@ -65,17 +65,12 @@ struct dapl_ia *dapl_ia_alloc(struct dat /* zero the structure */ memset(ia_ptr, 0, sizeof *ia_ptr); - /* - * initialize the header - */ - ia_ptr->header.provider = provider; - ia_ptr->header.magic = DAPL_MAGIC_IA; - ia_ptr->header.handle_type = DAT_HANDLE_TYPE_IA; - ia_ptr->header.owner_ia = ia_ptr; - ia_ptr->header.user_context.as_64 = 0; - ia_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&ia_ptr->header.ia_list_entry); - spin_lock_init(&ia_ptr->header.lock); + ia_ptr->ia.provider = provider; + ia_ptr->common.owner_ia = ia_ptr; + ia_ptr->common.user_context.as_64 = 0; + ia_ptr->common.user_context.as_ptr = NULL; + dapl_llist_init_entry(&ia_ptr->common.ia_list_entry); + spin_lock_init(&ia_ptr->common.lock); /* * initialize the body @@ -127,9 +122,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rmr_list_head)); while (rmr_ptr != NULL) { next_rmr_ptr = dapl_llist_next_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header. + &rmr_ptr->common. ia_list_entry); - dat_status = dapl_rmr_free(rmr_ptr); + dat_status = dapl_rmr_free((struct dat_rmr *)rmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rmr_free(%p) returns %x\n", @@ -141,9 +136,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rsp_list_head)); while (sp_ptr != NULL) { next_sp_ptr = dapl_llist_next_entry(&ia_ptr->rsp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); - dat_status = dapl_rsp_free(sp_ptr); + dat_status = dapl_rsp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rsp_free(%p) returns %x\n", @@ -155,7 +150,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->ep_list_head)); while (ep_ptr != NULL) { next_ep_ptr = dapl_llist_next_entry(&ia_ptr->ep_list_head, - &ep_ptr->header. + &ep_ptr->common. ia_list_entry); /* * Issue a disconnect if the EP needs it @@ -163,21 +158,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - || ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING - || ep_ptr->param.ep_state == - DAT_EP_STATE_DISCONNECT_PENDING) { - dat_status = - dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + DAT_EP_STATE_ACTIVE_CONNECTION_PENDING || + ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { + dat_status = dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_disconnect(%p) returns %x\n", ep_ptr, dat_status); } /* force the EP into error state to force flush all posted DTOs. */ - (void) dapl_modify_qp_state_to_error(ep_ptr->qp); + (void)dapl_modify_qp_state_to_error(ep_ptr->qp); - dat_status = dapl_ep_free(ep_ptr); + dat_status = dapl_ep_free((struct dat_ep *)ep_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_free(%p) returns %x\n", @@ -189,9 +182,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->lmr_list_head)); while (lmr_ptr != NULL) { next_lmr_ptr = dapl_llist_next_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header. + &lmr_ptr->common. ia_list_entry); - dat_status = dapl_lmr_free(lmr_ptr); + dat_status = dapl_lmr_free((struct dat_lmr *)lmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): lmr_free(%p) returns %x\n", @@ -213,7 +206,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status); next_sp_ptr = dapl_llist_next_entry(&ia_ptr->psp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); /* Remove CR's from this PSP and clean them up */ @@ -222,18 +215,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia while (cr_ptr != NULL) { next_cr_ptr = dapl_llist_next_entry( &sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); /* Remove the CR from the queue & cleanup */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); cr_ptr = next_cr_ptr; } - dat_status = dapl_psp_free(sp_ptr); + dat_status = dapl_psp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): psp_free(%p) returns %x\n", @@ -246,9 +240,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->pz_list_head)); while (pz_ptr != NULL) { next_pz_ptr = dapl_llist_next_entry(&ia_ptr->pz_list_head, - &pz_ptr->header. + &pz_ptr->common. ia_list_entry); - dat_status = dapl_pz_free(pz_ptr); + dat_status = dapl_pz_free((struct dat_pz *)pz_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): pz_free(%p) returns %x\n", @@ -266,11 +260,11 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->evd_list_head)); while (evd_ptr != NULL) { next_evd_ptr = dapl_llist_next_entry(&ia_ptr->evd_list_head, - &evd_ptr->header. + &evd_ptr->common. ia_list_entry); if (evd_ptr != ia_ptr->async_error_evd) { /* it isn't the async EVD; delete it. */ - dat_status = dapl_evd_free(evd_ptr); + dat_status = dapl_evd_free((struct dat_evd *)evd_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): evd_free(%p) returns %x\n", @@ -289,7 +283,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status = dapl_ia_teardown_callbacks(ia_ptr); atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - dat_status = dapl_evd_free(ia_ptr->async_error_evd); + dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != dat_status) dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -404,7 +398,7 @@ u32 dapl_ia_graceful_close(struct dapl_i if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - cur_dat_status = dapl_evd_free(ia_ptr->async_error_evd); + cur_dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; @@ -424,8 +418,6 @@ bail: */ void dapl_ia_free(struct dapl_ia *ia_ptr) { - dapl_os_assert(ia_ptr->header.magic == DAPL_MAGIC_IA); - dapl_os_assert(ia_ptr->async_error_evd == NULL); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->lmr_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rmr_list_head)); @@ -434,12 +426,8 @@ void dapl_ia_free(struct dapl_ia *ia_ptr dapl_os_assert(dapl_llist_is_empty(&ia_ptr->psp_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rsp_list_head)); - /* - * deinitialize the header - */ dapl_hca_unlink_ia(ia_ptr->hca_ptr, ia_ptr); - ia_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - /* no need to destroy ia_ptr->header.lock */ + /* no need to destroy ia_ptr->common.lock */ kfree(ia_ptr); } @@ -449,10 +437,10 @@ void dapl_ia_free(struct dapl_ia *ia_ptr */ void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry, ep_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry, ep_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -460,10 +448,10 @@ void dapl_ia_link_ep(struct dapl_ia *ia_ */ void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -471,10 +459,10 @@ void dapl_ia_unlink_ep(struct dapl_ia *i */ void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry, srq_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry, srq_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -482,10 +470,10 @@ void dapl_ia_link_srq(struct dapl_ia *ia */ void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -493,10 +481,10 @@ void dapl_ia_unlink_srq(struct dapl_ia * */ void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry, lmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry, lmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -504,10 +492,10 @@ void dapl_ia_link_lmr(struct dapl_ia *ia */ void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -515,10 +503,10 @@ void dapl_ia_unlink_lmr(struct dapl_ia * */ void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry, rmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry, rmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -526,10 +514,10 @@ void dapl_ia_link_rmr(struct dapl_ia *ia */ void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -537,10 +525,10 @@ void dapl_ia_unlink_rmr(struct dapl_ia * */ void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry, pz_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry, pz_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -548,10 +536,10 @@ void dapl_ia_link_pz(struct dapl_ia *ia_ */ void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -559,10 +547,10 @@ void dapl_ia_unlink_pz(struct dapl_ia *i */ void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry, evd_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry, evd_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -570,10 +558,10 @@ void dapl_ia_link_evd(struct dapl_ia *ia */ void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -581,10 +569,10 @@ void dapl_ia_unlink_evd(struct dapl_ia * */ void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->psp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -594,17 +582,17 @@ void dapl_ia_unlink_sp(struct dapl_ia *i { struct dapl_llist_entry **list_head; - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_PSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_PSP) { list_head = &ia_ptr->psp_list_head; } else { - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == + DAT_SP_TYPE_RSP); list_head = &ia_ptr->rsp_list_head; } - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); - dapl_llist_remove_entry(list_head, &sp_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); + dapl_llist_remove_entry(list_head, &sp_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -634,7 +622,7 @@ struct dapl_sp *dapl_ia_sp_search(struct else list_head = &ia_ptr->rsp_list_head; - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); sp_ptr = (dapl_llist_is_empty(list_head) ? NULL : dapl_llist_peek_head(list_head)); @@ -644,10 +632,10 @@ struct dapl_sp *dapl_ia_sp_search(struct break; sp_ptr = dapl_llist_next_entry(list_head, - &sp_ptr->header.ia_list_entry); + &sp_ptr->common.ia_list_entry); } - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); return sp_ptr; } @@ -670,10 +658,10 @@ struct dapl_sp *dapl_ia_sp_search(struct */ void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rsp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, Index: linux-kernel/dat-provider/dapl_cr_query.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_query.c (revision 2572) +++ linux-kernel/dat-provider/dapl_cr_query.c (working copy) @@ -34,16 +34,16 @@ #include "dapl.h" #include "dapl_openib_util.h" -u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) +u32 dapl_cr_query(struct dat_cr *cr, struct dat_cr_param *cr_param) { struct dapl_cr *cr_ptr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", - cr_handle, cr_param); + cr, cr_param); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (cr == NULL) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } @@ -53,10 +53,10 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* obtain the remote IP address */ - (void) dapl_ib_cm_remote_addr((DAT_HANDLE) cr_handle, + (void) dapl_ib_cm_remote_addr((void *) cr, &cr_ptr->remote_ia_address); /* since the arguments are easily accessible, ignore the mask */ Index: linux-kernel/dat-provider/dapl_provider.c =================================================================== --- linux-kernel/dat-provider/dapl_provider.c (revision 2572) +++ linux-kernel/dat-provider/dapl_provider.c (working copy) @@ -71,9 +71,8 @@ static struct dat_provider g_dapl_provid .ia_close_func = &dapl_ia_close, .ia_memtype_hint_func = &dapl_ia_memtype_hint, - .set_consumer_context_func = &dapl_set_consumer_context, - .get_consumer_context_func = &dapl_get_consumer_context, - .get_handle_type_func = &dapl_get_handle_type, + .set_consumer_context_func = NULL, + .get_consumer_context_func = NULL, .cr_query_func = &dapl_cr_query, .cr_accept_func = &dapl_cr_accept, Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2572) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -70,7 +70,7 @@ static void dapl_destroy_cm_id(struct da conn, conn->cm_id); ib_destroy_cm_id(conn->cm_id); if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } } @@ -172,7 +172,7 @@ static int dapl_cm_active_cb_handler(str spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -285,7 +285,7 @@ static int dapl_cm_passive_cb_handler(st spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -413,7 +413,7 @@ error: * Initiate a connection with the passive listener on another node * * Input: - * ep_handle + * ep * remote_ia_address * remote_conn_qual * private_data_size size of private data and structure @@ -428,7 +428,7 @@ error: * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data) @@ -438,7 +438,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han struct dapl_cm_id *conn; int status; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->qp == NULL) { printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); return DAT_INTERNAL_ERROR; @@ -460,9 +460,9 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han kfree(conn); return DAT_INTERNAL_ERROR; } - conn->ep->cm_handle = conn; + conn->ep->cm = conn; - ia_ptr = conn->ep->header.owner_ia; + ia_ptr = conn->ep->common.owner_ia; /* Setup QP/CM parameters */ memset(&conn->param, 0, sizeof conn->param); @@ -507,7 +507,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han * Disconnect an EP * * Input: - * ep_handle + * ep * close_flags * * Output: @@ -521,7 +521,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han */ u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags) { - struct dapl_cm_id *conn = ep_ptr->cm_handle; + struct dapl_cm_id *conn = ep_ptr->cm; int status; dapl_dbg_log(DAPL_DBG_TYPE_CM, @@ -537,7 +537,7 @@ u32 dapl_ib_disconnect(struct dapl_ep *e status = ib_send_cm_dreq(conn->cm_id, NULL, 0); if (status) printk(KERN_ERR "dapl_ib_disconnect: CM ID 0x%p " - "status %d\n", ep_ptr->cm_handle, status); + "status %d\n", ep_ptr->cm, status); } return DAT_SUCCESS; @@ -607,14 +607,14 @@ void dapl_ib_reinit_ep(struct dapl_ep *e dapl_dbg_log(DAPL_DBG_TYPE_CM, " >>> dapl_ib_reinit_ep: EP %p\n", ep_ptr); - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* * What if QP is connected? Do we need to destroy the cm_id * here as well, send a DREQ, REJ? What cm_id state are we in? * we may not care if we just destroy the cm_id - * ib_destroy_cm_id(ep_ptr->cm_handle); - * supporting this call may require setting the cm_handle to + * ib_destroy_cm_id(ep_ptr->cm); + * supporting this call may require setting the cm to * something like NULL wherever it is destroyed */ @@ -685,7 +685,7 @@ u32 dapl_ib_setup_conn_listener(struct d * Have the CM remove a connection listener. * * Input: - * ia_handle IA handle + * ia IA handle * ServiceID IB Channel Service ID * * Output: @@ -719,7 +719,7 @@ u32 dapl_ib_remove_conn_listener(struct * Perform necessary steps to reject a connection * * Input: - * cm_handle + * cm * * Output: * none @@ -730,24 +730,24 @@ u32 dapl_ib_remove_conn_listener(struct * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle) +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm) { int status; - if (cm_handle == NULL) { + if (cm == NULL) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_reject_connection: invalid cm handle\n"); return DAT_SUCCESS; } - status = ib_send_cm_rej(cm_handle->cm_id, IB_CM_REJ_CONSUMER_DEFINED, + status = ib_send_cm_rej(cm->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_reject_connection: " "ib_send_cm_rej failed: %d\n", status); return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); } - dapl_destroy_cm_id(cm_handle); + dapl_destroy_cm_id(cm); return DAT_SUCCESS; } @@ -757,8 +757,8 @@ u32 dapl_ib_reject_connection(struct dap * Perform necessary steps to accept a connection * * Input: - * cr_handle - * ep_handle + * cr + * ep * private_data_size * private_data * @@ -771,7 +771,7 @@ u32 dapl_ib_reject_connection(struct dap * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *prd_ptr) { @@ -783,9 +783,9 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN struct ib_cm_rep_param passive_params; struct dapl_cm_id *conn; - cr_ptr = (struct dapl_cr *)cr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; - ia_ptr = ep_ptr->header.owner_ia; + cr_ptr = (struct dapl_cr *)cr; + ep_ptr = (struct dapl_ep *)ep; + ia_ptr = ep_ptr->common.owner_ia; conn = cr_ptr->ib_cm_handle; /* Obtain size of private data structure & contents */ @@ -810,8 +810,8 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN } } - cr_ptr->param.local_ep_handle = ep_handle; - ep_ptr->cm_handle = conn; + cr_ptr->param.local_ep = ep; + ep_ptr->cm = conn; conn->ep = ep_ptr; memset(&passive_params, 0, sizeof passive_params); @@ -851,7 +851,7 @@ reject: /* * ib_cm_get_remote_gid */ -static int ib_cm_get_remote_gid(struct dapl_cm_id *cm_handle, u8 *remote_gid) +static int ib_cm_get_remote_gid(struct dapl_cm_id *cm, u8 *remote_gid) { return DAT_INTERNAL_ERROR; /* for now!!! */ } @@ -862,7 +862,7 @@ static int ib_cm_get_remote_gid(struct d * Obtain the remote IP address given a connection * * Input: - * cr_handle + * cr * private data structure handle (only if not using ATS) * * Output: @@ -874,7 +874,7 @@ static int ib_cm_get_remote_gid(struct d * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address) { struct dapl_ia *ia_ptr; @@ -883,7 +883,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da struct dapl_cr *cr_ptr = (struct dapl_cr *)dat_handle; - ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia_handle; + ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia; rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); if (rc < 0) Index: linux-kernel/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel/patches/alt_dat_provider_makefile (revision 2572) +++ linux-kernel/patches/alt_dat_provider_makefile (working copy) @@ -21,8 +21,6 @@ PROVIDER_MODULES := \ dapl_cr_util \ dapl_ep \ dapl_evd \ - dapl_get_consumer_context \ - dapl_get_handle_type \ dapl_hash \ dapl_hca_util \ dapl_ia_close \ @@ -39,7 +37,6 @@ PROVIDER_MODULES := \ dapl_ring_buffer_util \ dapl_rmr \ dapl_rsp \ - dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ dapl_util Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2572) +++ linux-kernel/dat/api.c (working copy) @@ -350,7 +350,7 @@ static u32 dat_strerror_minor(u32 value, } u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, + struct dat_evd **async_event_handle, struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; @@ -387,13 +387,12 @@ u32 dat_ia_openv(const char *name, int a return status; } - return ia_open_func(name, async_event_qlen, async_event_handle, - ia_handle); + return ia_open_func(name, async_event_qlen, async_event_handle, ia); } EXPORT_SYMBOL(dat_ia_openv); -u32 dat_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags flags) +u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; @@ -401,17 +400,16 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle const char *ia_name; dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "%s(ia:%x, flags:%x)\n", __func__, ia_handle, flags); + "%s(ia:%x, flags:%x)\n", __func__, ia, flags); - provider = DAT_HANDLE_TO_PROVIDER(ia_handle); + provider = DAT_HANDLE_TO_PROVIDER(ia); ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia_handle, NULL, NULL, + if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); - else if (DAT_SUCCESS != (status = provider->ia_close_func(ia_handle, - flags))) + else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "close function for IA %s failed\n", ia_name); else { Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2572) +++ linux-kernel/dat/dat.h (working copy) @@ -333,20 +333,6 @@ enum dat_return_subtype { #define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) #define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) -/* dat handle types */ -enum dat_handle_type { - DAT_HANDLE_TYPE_CR, - DAT_HANDLE_TYPE_EP, - DAT_HANDLE_TYPE_EVD, - DAT_HANDLE_TYPE_IA, - DAT_HANDLE_TYPE_LMR, - DAT_HANDLE_TYPE_PSP, - DAT_HANDLE_TYPE_PZ, - DAT_HANDLE_TYPE_RMR, - DAT_HANDLE_TYPE_RSP, - DAT_HANDLE_TYPE_SRQ -}; - #define DAT_NAME_MAX_LENGTH 256 struct dat_named_attr { @@ -385,21 +371,6 @@ enum dat_completion_flags { #define DAT_TIMEOUT_MAX ULONG_MAX -/* dat handles */ -typedef void *DAT_HANDLE; -typedef DAT_HANDLE DAT_CR_HANDLE; -typedef DAT_HANDLE DAT_EP_HANDLE; -typedef DAT_HANDLE DAT_EVD_HANDLE; -typedef DAT_HANDLE DAT_IA_HANDLE; -typedef DAT_HANDLE DAT_LMR_HANDLE; -typedef DAT_HANDLE DAT_PSP_HANDLE; -typedef DAT_HANDLE DAT_PZ_HANDLE; -typedef DAT_HANDLE DAT_RMR_HANDLE; -typedef DAT_HANDLE DAT_RSP_HANDLE; -typedef DAT_HANDLE DAT_SRQ_HANDLE; - -#define DAT_HANDLE_NULL ((DAT_HANDLE)NULL) - typedef u64 DAT_CONN_QUAL; typedef u64 DAT_PORT_QUAL; @@ -541,8 +512,8 @@ enum dat_mem_priv_flags { /* RMR Arguments */ struct dat_rmr_param { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; struct dat_lmr_triplet lmr_triplet; enum dat_mem_priv_flags mem_priv; DAT_RMR_CONTEXT rmr_context; @@ -588,17 +559,17 @@ enum dat_ep_state { }; struct dat_ep_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_ep_state ep_state; struct sockaddr *local_ia_address_ptr; DAT_PORT_QUAL local_port_qual; struct sockaddr *remote_ia_address_ptr; DAT_PORT_QUAL remote_port_qual; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EVD_HANDLE request_evd_handle; - DAT_EVD_HANDLE connect_evd_handle; - DAT_SRQ_HANDLE srq_handle; + struct dat_pz *pz; + struct dat_evd *recv_evd; + struct dat_evd *request_evd; + struct dat_evd *connect_evd; + struct dat_srq *srq; struct dat_ep_attr ep_attr; }; @@ -654,9 +625,9 @@ struct dat_srq_attr { }; struct dat_srq_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_srq_state srq_state; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; int max_recv_dtos; int max_recv_iov; int low_watermark; @@ -667,25 +638,25 @@ struct dat_srq_param { /* PZ Parameters */ struct dat_pz_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; }; /* PSP Parameters */ struct dat_psp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; }; /* RSP Parameters */ struct dat_rsp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_ep *ep; }; /* Connection Request Parameters. @@ -712,11 +683,11 @@ struct dat_cr_param { /* The local Endpoint provided by the Service Point for the requested * connection. It is the only Endpoint that can accept a Connection - * Request on this Service Point. The value DAT_HANDLE_NULL + * Request on this Service Point. The value NULL * represents that there is no associated local Endpoint for the * requested connection. */ - DAT_EP_HANDLE local_ep_handle; + struct dat_ep *local_ep; }; /************************** Events ******************************/ @@ -753,7 +724,7 @@ enum dat_dto_completion_status { /* DTO completion event data */ /* transfered_length is not defined if status is not DAT_SUCCESS */ struct dat_dto_completion_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; DAT_DTO_COOKIE user_cookie; enum dat_dto_completion_status status; u64 transfered_length; @@ -761,25 +732,20 @@ struct dat_dto_completion_event_data { /* RMR bind completion event data */ struct dat_rmr_bind_completion_event_data { - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr *rmr; DAT_RMR_COOKIE user_cookie; enum dat_dto_completion_status status; }; -typedef union dat_sp_handle { - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; -} DAT_SP_HANDLE; - /* Connection Request Arrival event data */ struct dat_cr_arrival_event_data { /* Handle to the Service Point that received the Connection Request * from the remote side. If the Service Point was Reserved, sp is - * DAT_HANDLE_NULL because the reserved Service Point is + * NULL because the reserved Service Point is * automatically destroyed upon generating this event. Can be PSP * or RSP. */ - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; /* Address of the IA on which the Connection Request arrived. */ struct sockaddr *local_ia_address_ptr; @@ -791,30 +757,30 @@ struct dat_cr_arrival_event_data { /* The Connection Request instance created by a Provider for the * arrived Connection Request. Consumers can find out private_data - * passed by a remote Consumer from cr_handle. It is up to a Consumer + * passed by a remote Consumer from cr. It is up to a Consumer * to dat_cr_accept or dat_cr_reject of the Connection Request. */ - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; }; /* Connection event data */ struct dat_connection_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; int private_data_size; void *private_data; }; /* Async Error event data */ -/* For unaffiliated asynchronous event dat_handle is ia_handle. For - * Endpoint affiliated asynchronous event dat_handle is ep_handle. For - * EVD affiliated asynchronous event dat_handle is evd_handle. For SRQ +/* For unaffiliated asynchronous event dat_handle is ia. For + * Endpoint affiliated asynchronous event dat_handle is ep. For + * EVD affiliated asynchronous event dat_handle is evd. For SRQ * affiliated asynchronous event dat_handle is srq_handle. For Memory - * affiliated asynchronous event dat_handle is either lmr_handle, - * rmr_handle or pz_handle. + * affiliated asynchronous event dat_handle is either lmr, + * rmr_handle or pz. */ struct dat_asynch_error_event_data { - DAT_HANDLE dat_handle; /* either IA, EP, EVD, SRQ, */ + void *dat_handle; /* either IA, EP, EVD, SRQ, */ /* LMR, RMR, or PZ handle */ int reason; /* object specific */ }; @@ -894,7 +860,7 @@ typedef union dat_event_data { struct dat_event { enum dat_event_number event_number; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; DAT_EVENT_DATA event_data; }; @@ -934,7 +900,7 @@ struct dat_provider_info { }; struct dat_evd_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; int evd_qlen; enum dat_upcall_policy upcall_policy; struct dat_upcall_object upcall; @@ -960,7 +926,7 @@ enum dat_mem_optimize_flags { typedef union dat_region_description { void *for_va; - DAT_LMR_HANDLE for_lmr_handle; + struct dat_lmr *for_lmr; void *for_pointer; /* For kdapl only */ void *for_array; /* For kdapl only */ u64 for_pa; /* For kdapl only */ @@ -968,11 +934,11 @@ typedef union dat_region_description { /* LMR Arguments */ struct dat_lmr_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_mem_type mem_type; DAT_REGION_DESCRIPTION region_desc; u64 length; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; enum dat_mem_priv_flags mem_priv; DAT_LMR_CONTEXT lmr_context; DAT_RMR_CONTEXT rmr_context; @@ -1083,178 +1049,176 @@ struct dat_provider_attr { * ****************************************************************/ -typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *); +typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **); -typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *, u32, u32, boolean_t); +typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **, u32, u32, boolean_t); -typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum dat_close_flags); -typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_HANDLE, DAT_EVD_HANDLE *, +typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); - -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(void *, union dat_context); -typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, enum dat_handle_type *); +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(void *, union dat_context *); -typedef u32 (*DAT_CR_QUERY_FUNC)(DAT_CR_HANDLE, struct dat_cr_param *); +typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct dat_cr_param *); -typedef u32 (*DAT_CR_ACCEPT_FUNC)(DAT_CR_HANDLE, DAT_EP_HANDLE, int, +typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct dat_ep *, int, const void*); -typedef u32 (*DAT_CR_REJECT_FUNC)(DAT_CR_HANDLE); +typedef u32 (*DAT_CR_REJECT_FUNC)(struct dat_cr *); -typedef u32 (*DAT_CR_HANDOFF_FUNC)(DAT_CR_HANDLE, DAT_CONN_QUAL); +typedef u32 (*DAT_CR_HANDOFF_FUNC)(struct dat_cr *, DAT_CONN_QUAL); -typedef u32 (*DAT_EVD_RESIZE_FUNC)(DAT_EVD_HANDLE, int); +typedef u32 (*DAT_EVD_RESIZE_FUNC)(struct dat_evd *, int); -typedef u32 (*DAT_EVD_POST_SE_FUNC)(DAT_EVD_HANDLE, const struct dat_event *); +typedef u32 (*DAT_EVD_POST_SE_FUNC)(struct dat_evd *, const struct dat_event *); -typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(DAT_EVD_HANDLE, struct dat_event *); +typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(struct dat_evd *, struct dat_event *); -typedef u32 (*DAT_EVD_FREE_FUNC)(DAT_EVD_HANDLE); +typedef u32 (*DAT_EVD_FREE_FUNC)(struct dat_evd *); -typedef u32 (*DAT_EVD_QUERY_FUNC)(DAT_EVD_HANDLE, struct dat_evd_param *); +typedef u32 (*DAT_EVD_QUERY_FUNC)(struct dat_evd *, struct dat_evd_param *); -typedef u32 (*DAT_EP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - const struct dat_ep_attr *, DAT_EP_HANDLE *); +typedef u32 (*DAT_EP_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, struct dat_evd *, + struct dat_evd *, struct dat_evd *, + const struct dat_ep_attr *, struct dat_ep **); -typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_SRQ_HANDLE, +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_evd *, struct dat_evd *, + struct dat_evd *, struct dat_srq *, const struct dat_ep_attr *, - DAT_EP_HANDLE *); + struct dat_ep **); -typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_HANDLE, struct dat_ep_param *); +typedef u32 (*DAT_EP_QUERY_FUNC)(struct dat_ep *, struct dat_ep_param *); -typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, enum dat_ep_param_mask, +typedef u32 (*DAT_EP_MODIFY_FUNC)(struct dat_ep *, enum dat_ep_param_mask, const struct dat_ep_param *); -typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, +typedef u32 (*DAT_EP_CONNECT_FUNC)(struct dat_ep *, struct sockaddr *, DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); -typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(struct dat_ep *, struct dat_ep *, unsigned long timeout, int, const void *, enum dat_qos); -typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_EP_DISCONNECT_FUNC)(struct dat_ep *, enum dat_close_flags); -typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_SEND_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RECV_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RECV_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, enum dat_ep_state *, +typedef u32 (*DAT_EP_GET_STATUS_FUNC)(struct dat_ep *, enum dat_ep_state *, boolean_t *, boolean_t *); -typedef u32 (*DAT_EP_FREE_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_FREE_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RESET_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_RESET_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(DAT_EP_HANDLE, int *, int *); +typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(struct dat_ep *, int *, int *); -typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(DAT_EP_HANDLE, int, int); +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(struct dat_ep *, int, int); -typedef u32 (*DAT_LMR_FREE_FUNC)(DAT_LMR_HANDLE); +typedef u32 (*DAT_LMR_FREE_FUNC)(struct dat_lmr *); -typedef u32 (*DAT_LMR_QUERY_FUNC)(DAT_LMR_HANDLE, struct dat_lmr_param *); +typedef u32 (*DAT_LMR_QUERY_FUNC)(struct dat_lmr *, struct dat_lmr_param *); -typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_PZ_HANDLE, DAT_RMR_HANDLE *); +typedef u32 (*DAT_RMR_CREATE_FUNC)(struct dat_pz *, struct dat_rmr * *); -typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RMR_HANDLE, struct dat_rmr_param *); +typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); -typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, - enum dat_mem_priv_flags, DAT_EP_HANDLE, +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, + enum dat_mem_priv_flags, struct dat_ep *, DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); -typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); +typedef u32 (*DAT_RMR_FREE_FUNC)(struct dat_rmr *); -typedef u32 (*DAT_PSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, - enum dat_psp_flags, DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_evd *, + enum dat_psp_flags, struct dat_sp **); -typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL *, - DAT_EVD_HANDLE, enum dat_psp_flags, - DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(struct dat_ia *, DAT_CONN_QUAL *, + struct dat_evd *, enum dat_psp_flags, + struct dat_sp **); -typedef u32 (*DAT_PSP_QUERY_FUNC)(DAT_PSP_HANDLE, struct dat_psp_param *); +typedef u32 (*DAT_PSP_QUERY_FUNC)(struct dat_sp *, struct dat_psp_param *); -typedef u32 (*DAT_PSP_FREE_FUNC)(DAT_PSP_HANDLE); +typedef u32 (*DAT_PSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_RSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, - DAT_EVD_HANDLE, DAT_RSP_HANDLE *); +typedef u32 (*DAT_RSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_ep *, + struct dat_evd *, struct dat_sp **); -typedef u32 (*DAT_RSP_QUERY_FUNC)(DAT_RSP_HANDLE, struct dat_rsp_param *); +typedef u32 (*DAT_RSP_QUERY_FUNC)(struct dat_sp *, struct dat_rsp_param *); -typedef u32 (*DAT_RSP_FREE_FUNC)(DAT_RSP_HANDLE); +typedef u32 (*DAT_RSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_PZ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE *); +typedef u32 (*DAT_PZ_CREATE_FUNC)(struct dat_ia *, struct dat_pz **); -typedef u32 (*DAT_PZ_QUERY_FUNC)(DAT_PZ_HANDLE, struct dat_pz_param *); +typedef u32 (*DAT_PZ_QUERY_FUNC)(struct dat_pz *, struct dat_pz_param *); -typedef u32 (*DAT_PZ_FREE_FUNC)(DAT_PZ_HANDLE); +typedef u32 (*DAT_PZ_FREE_FUNC)(struct dat_pz *); -typedef u32 (*DAT_SRQ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - struct dat_srq_attr *, DAT_SRQ_HANDLE *); +typedef u32 (*DAT_SRQ_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_srq_attr *, struct dat_srq **); -typedef u32 (*DAT_SRQ_SET_LW_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_SET_LW_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_FREE_FUNC)(DAT_SRQ_HANDLE); +typedef u32 (*DAT_SRQ_FREE_FUNC)(struct dat_srq *); -typedef u32 (*DAT_SRQ_QUERY_FUNC)(DAT_SRQ_HANDLE, struct dat_srq_param *); +typedef u32 (*DAT_SRQ_QUERY_FUNC)(struct dat_srq *, struct dat_srq_param *); -typedef u32 (*DAT_SRQ_RESIZE_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_RESIZE_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DAT_SRQ_HANDLE, int, +typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(struct dat_srq *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, +typedef u32 (*DAT_LMR_KCREATE_FUNC)(struct dat_ia *, enum dat_mem_type, DAT_REGION_DESCRIPTION, u64, - DAT_PZ_HANDLE, enum dat_mem_priv_flags, + struct dat_pz *, enum dat_mem_priv_flags, enum dat_mem_optimize_flags, - DAT_LMR_HANDLE *, + struct dat_lmr **, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, u64 *, u64 *); -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, u64, +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(struct dat_ia *, enum dat_mem_type, u64, enum dat_mem_optimize_flags, u64 *, u64 *); -typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, enum dat_upcall_policy, +typedef u32 (*DAT_EVD_KCREATE_FUNC) (struct dat_ia *, int, enum dat_upcall_policy, const struct dat_upcall_object *, - enum dat_evd_flags, DAT_EVD_HANDLE *); + enum dat_evd_flags, struct dat_evd **); -typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(struct dat_evd *, enum dat_upcall_policy, const struct dat_upcall_object *); @@ -1269,7 +1233,6 @@ struct dat_provider { DAT_SET_CONSUMER_CONTEXT_FUNC set_consumer_context_func; DAT_GET_CONSUMER_CONTEXT_FUNC get_consumer_context_func; - DAT_GET_HANDLE_TYPE_FUNC get_handle_type_func; DAT_CR_QUERY_FUNC cr_query_func; DAT_CR_ACCEPT_FUNC cr_accept_func; @@ -1338,6 +1301,52 @@ struct dat_provider { DAT_SRQ_SET_LW_FUNC srq_set_lw_func; }; +struct dat_ep { + struct dat_provider *provider; +}; + +struct dat_ia { + struct dat_provider *provider; +}; + +struct dat_evd { + struct dat_provider *provider; +}; + +struct dat_cno { + struct dat_provider *provider; +}; + +struct dat_srq { + struct dat_provider *provider; +}; + +struct dat_pz { + struct dat_provider *provider; +}; + +struct dat_lmr { + struct dat_provider *provider; +}; + +struct dat_rmr { + struct dat_provider *provider; +}; + +enum dat_sp_type { + DAT_SP_TYPE_RSP, + DAT_SP_TYPE_PSP +}; + +struct dat_sp { + struct dat_provider *provider; + enum dat_sp_type type; +}; + +struct dat_cr { + struct dat_provider *provider; +}; + /*************************************************************** * * FUNCTION PROTOTYPES @@ -1359,8 +1368,8 @@ extern u32 dat_registry_remove_provider( * DAT registry functions for consumers */ extern u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, - DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, + struct dat_evd **async_event_handle, + struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety); #define dat_ia_open(name, qlen, async_evd, ia) \ @@ -1368,7 +1377,7 @@ extern u32 dat_ia_openv(const char *name DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ DAT_THREADSAFE) -extern u32 dat_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); extern u32 dat_strerror(u32, const char **, const char **); @@ -1383,27 +1392,21 @@ extern u32 dat_registry_list_providers(i #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) -static inline u32 dat_set_consumer_context(DAT_HANDLE handle, +static inline u32 dat_set_consumer_context(void *handle, union dat_context context) { return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, context); } -static inline u32 dat_get_consumer_context(DAT_HANDLE handle, +static inline u32 dat_get_consumer_context(void *handle, union dat_context *context) { return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, context); } -static inline u32 dat_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type) -{ - return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); -} - -static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, +static inline u32 dat_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimize, @@ -1415,7 +1418,7 @@ static inline u32 dat_ia_memtype_hint(DA preferred_alignment); } -static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE *async_evd, +static inline u32 dat_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -1423,49 +1426,49 @@ static inline u32 dat_ia_query(DAT_IA_HA ia_query_func, ia, async_evd, ia_attr, provider_attr); } -static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, +static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data) { return DAT_CALL_PROVIDER_FUNC( cr_accept_func, cr, ep, private_data_size, private_data); } -static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) +static inline u32 dat_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff) { return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); } -static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) +static inline u32 dat_cr_query(struct dat_cr *cr, struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } -static inline u32 dat_cr_reject(DAT_CR_HANDLE cr) +static inline u32 dat_cr_reject(struct dat_cr *cr) { return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); } -static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) +static inline u32 dat_evd_dequeue(struct dat_evd *evd, struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } -static inline u32 dat_evd_free(DAT_EVD_HANDLE evd) +static inline u32 dat_evd_free(struct dat_evd *evd) { return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); } -static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, +static inline u32 dat_evd_kcreate(struct dat_ia *ia, int qlen, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall, enum dat_evd_flags flags, - DAT_EVD_HANDLE * evd) + struct dat_evd ** evd) { return DAT_CALL_PROVIDER_FUNC(evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); } -static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall) { @@ -1473,23 +1476,23 @@ static inline u32 dat_evd_modify_upcall( upcall); } -static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_post_se(struct dat_evd *evd, const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } -static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) +static inline u32 dat_evd_query(struct dat_evd *evd, struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } -static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, int qlen) +static inline u32 dat_evd_resize(struct dat_evd *evd, int qlen) { return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, +static inline u32 dat_ep_connect(struct dat_ep *ep, struct sockaddr *ia_addr, DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, @@ -1500,36 +1503,36 @@ static inline u32 dat_ep_connect(DAT_EP_ qos, flags); } -static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, +static inline u32 dat_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, struct dat_evd *out_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_func, ia, pz, in_evd, out_evd, connect_evd, attr, ep); } -static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, - DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, - DAT_SRQ_HANDLE srq, +static inline u32 dat_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, + struct dat_evd *out_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_with_srq_func, ia, pz, in_evd, out_evd, connect_evd, srq, attr, ep); } -static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_disconnect(struct dat_ep *ep, enum dat_close_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } -static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, +static inline u32 dat_ep_dup_connect(struct dat_ep *ep, struct dat_ep *dup_ep, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) @@ -1538,25 +1541,25 @@ static inline u32 dat_ep_dup_connect(DAT private_data_size, private_data, qos); } -static inline u32 dat_ep_free(DAT_EP_HANDLE ep) +static inline u32 dat_ep_free(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, enum dat_ep_state *state, +static inline u32 dat_ep_get_status(struct dat_ep *ep, enum dat_ep_state *state, boolean_t *recv_idle, boolean_t *req_idle) { return DAT_CALL_PROVIDER_FUNC(ep_get_status_func, ep, state, recv_idle, req_idle); } -static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, enum dat_ep_param_mask mask, +static inline u32 dat_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } -static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_read(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1566,7 +1569,7 @@ static inline u32 dat_ep_post_rdma_read( local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_write(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1576,7 +1579,7 @@ static inline u32 dat_ep_post_rdma_write local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_recv(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, enum dat_completion_flags flags) @@ -1585,7 +1588,7 @@ static inline u32 dat_ep_post_recv(DAT_E cookie, flags); } -static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_send(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1595,24 +1598,24 @@ static inline u32 dat_ep_post_send(DAT_E ep_post_send_func, ep, size, local_iov, cookie, flags); } -static inline u32 dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) +static inline u32 dat_ep_query(struct dat_ep *ep, struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } -static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, int *bufs_alloc, +static inline u32 dat_ep_recv_query(struct dat_ep *ep, int *bufs_alloc, int *bufs_avail) { return DAT_CALL_PROVIDER_FUNC(ep_recv_query_func, ep, bufs_alloc, bufs_avail); } -static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) +static inline u32 dat_ep_reset(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); } -static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, +static inline u32 dat_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -1620,17 +1623,17 @@ static inline u32 dat_ep_set_watermark(D soft_high_watermark, hard_high_watermark); } -static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) +static inline u32 dat_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } -static inline u32 dat_lmr_free(DAT_LMR_HANDLE lmr) +static inline u32 dat_lmr_free(struct dat_lmr *lmr) { return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); } -static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1638,7 +1641,7 @@ static inline u32 dat_lmr_sync_rdma_read num_iovs); } -static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1646,20 +1649,20 @@ static inline u32 dat_lmr_sync_rdma_writ num_iovs); } -static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE *rmr) +static inline u32 dat_rmr_create(struct dat_pz *pz, struct dat_rmr * *rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline u32 dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) +static inline u32 dat_rmr_query(struct dat_rmr * rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, +static inline u32 dat_rmr_bind(struct dat_rmr * rmr, const struct dat_lmr_triplet *iov, enum dat_mem_priv_flags mem_flags, - DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, + struct dat_ep *ep, DAT_RMR_COOKIE cookie, enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) { @@ -1667,84 +1670,84 @@ static inline u32 dat_rmr_bind(DAT_RMR_H cookie, comp_flags, context); } -static inline u32 dat_rmr_free(DAT_RMR_HANDLE rmr) +static inline u32 dat_rmr_free(struct dat_rmr * rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) +static inline u32 dat_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd, +static inline u32 dat_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_any_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) +static inline u32 dat_psp_query(struct dat_sp *psp, struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } -static inline u32 dat_psp_free(DAT_PSP_HANDLE psp) +static inline u32 dat_psp_free(struct dat_sp *psp) { return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); } -static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, - DAT_RSP_HANDLE *rsp) +static inline u32 dat_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_create_func, ia, conn_qual, ep, evd, rsp); } -static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) +static inline u32 dat_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } -static inline u32 dat_rsp_free(DAT_RSP_HANDLE rsp) +static inline u32 dat_rsp_free(struct dat_sp *rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); } -static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE *pz) +static inline u32 dat_pz_create(struct dat_ia *ia, struct dat_pz **pz) { return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) +static inline u32 dat_pz_query(struct dat_pz *pz, struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } -static inline u32 dat_pz_free(DAT_PZ_HANDLE pz) +static inline u32 dat_pz_free(struct dat_pz *pz) { return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); } -static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, +static inline u32 dat_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *attr, - DAT_SRQ_HANDLE *srq) + struct dat_srq **srq) { return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); } -static inline u32 dat_srq_free(DAT_SRQ_HANDLE srq) +static inline u32 dat_srq_free(struct dat_srq *srq) { return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); } -static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, int num_iovs, +static inline u32 dat_srq_post_recv(struct dat_srq *srq, int num_iovs, struct dat_lmr_triplet *iovs, DAT_DTO_COOKIE cookie) { @@ -1752,34 +1755,34 @@ static inline u32 dat_srq_post_recv(DAT_ cookie); } -static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) +static inline u32 dat_srq_query(struct dat_srq *srq, struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } -static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, int max_recv_dtos) +static inline u32 dat_srq_resize(struct dat_srq *srq, int max_recv_dtos) { return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); } -static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, int low_watermark) +static inline u32 dat_srq_set_lw(struct dat_srq *srq, int low_watermark) { return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, enum dat_mem_type type, +static inline u32 dat_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type type, DAT_REGION_DESCRIPTION region, u64 len, - DAT_PZ_HANDLE pz, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { return DAT_CALL_PROVIDER_FUNC(lmr_kcreate_func, ia, type, region, len, - pz, privileges, optimization, lmr_handle, + pz, privileges, optimization, lmr, lmr_context, rmr_context, registered_length, registered_address); } From mshefty at ichips.intel.com Wed Jun 8 11:10:03 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 08 Jun 2005 11:10:03 -0700 Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <1118253907.31223.2.camel@duffman> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> Message-ID: <42A7347B.8060100@ichips.intel.com> Tom Duffy wrote: > On Wed, 2005-06-08 at 12:35 -0400, James Lentini wrote: > >>Committed in revision 2572. > > Updated patch against 2572. > > Signed-off-by: Tom Duffy James, is there a specific hold-up for this patch? - Sean From sakasitashakoujjj at cocoa.ocn.ne.jp Wed Jun 8 09:08:04 2005 From: sakasitashakoujjj at cocoa.ocn.ne.jp (=?ISO-2022-JP?B?GyRCOmQyPDt2TDM9ahsoQiA=?=) Date: Thu, 09 Jun 2005 01:08:04 +0900 Subject: [openib-general] =?iso-2022-jp?b?GyRCIic6ZDI8PFI4cjt2TDMbKEI=?= =?iso-2022-jp?b?GyRCPWokKyRpNS5KfU1NJFgkNE8iTW0kRyQ5IicbKEI=?= Message-ID: <20050608.1608040609@sakasitashakoujjj-cocoa.ocn.ne.jp> 女性会員登録条件→既婚未婚問わず、年収1,000万以上 の当社交場から貴方様に個人的指名が入りました。 【指名者】 村山理沙 【プロフィール】 29歳・162cm48kg・年収2,500万・未婚 【コメント】 以前お付き合いしてた方がとても暴力的で辛かったです。 会社をいくつか運営しているので金銭的な事だけが目的だったようなので仕方ありませんが、もし少しでも可愛がってくれるって気持ちがあるのなら一度お食事でもご一緒したいです。 もう前の人とは完全に縁を切っています。彼に使ったお金を貴方に使えるなら 私は少しの優しさをいただければそれで幸せです。良いお返事待ってます。 ↓こちらからご登録（無料）をして、お待ち下さい。↓ http://pure1107.net/ 直接、村山様からご連絡がきます。 From libor at topspin.com Wed Jun 8 12:46:50 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 8 Jun 2005 12:46:50 -0700 Subject: [openib-general] shutdown issue with SDP In-Reply-To: <78d18e20506081029183adcfc@mail.gmail.com>; from bjordan.ics@gmail.com on Wed, Jun 08, 2005 at 01:29:42PM -0400 References: <78d18e20506081029183adcfc@mail.gmail.com> Message-ID: <20050608124650.A28180@topspin.com> On Wed, Jun 08, 2005 at 01:29:42PM -0400, William Jordan wrote: > I'm trying to run netperf over SDP using libsdp. I'm having > intermittent success, but usually, I get a failure. The client does a > shutdown (SHUT_WR), but the server gets an ECONNRESET, "Connection > reset by peer", on the recv instead of a zero length recv. > > I'm running 2568. > > Anyone seen similar/different results? Bill, Can you try the patch I sent out yesterday? It significantly simplifies the connection state machine within SDP. I ran it successfully with a number of the netperf tests. (TCP_CC, TCP_CRR, TCP_RR, TCP_STREAM) Thanks. -Libor From rep.nop at aon.at Wed Jun 8 12:48:32 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Wed, 8 Jun 2005 21:48:32 +0200 Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <42A7347B.8060100@ichips.intel.com> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> <42A7347B.8060100@ichips.intel.com> Message-ID: <20050608194831.GB5412@aon.at> On Wed, Jun 08, 2005 at 11:10:03AM -0700, Sean Hefty wrote: >Tom Duffy wrote: >>On Wed, 2005-06-08 at 12:35 -0400, James Lentini wrote: >> >>>Committed in revision 2572. >> >>Updated patch against 2572. >> >>Signed-off-by: Tom Duffy > >James, is there a specific hold-up for this patch? > Tom, just curious.. Do you plan to iterate over those disturbing 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally would appreciate it, since these variants are harder to read for me. But then i also tend to prefer '* 4711e-0815' over '/ 0x100000' From jlentini at netapp.com Wed Jun 8 12:48:52 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 15:48:52 -0400 (EDT) Subject: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: <1118101725.30742.2.camel@duffman> References: <1118093694.5735.13.camel@duffman> <1118101725.30742.2.camel@duffman> Message-ID: I like this update Tom. I'd propse a few changes: - remove dat_cno because it is not part of the kDAPL API - remove DAT_HANDLE_TO_PROVIDER - we can provide the functionality of dat_set_consumer_context and dat_get_consumer_context very easily by creating a dat_common structure (see my patch). In that case we can remove the user_context member from the dapl_common structure. - I've added the symbols removed from dat.h to kdat.h for legacy support. Let me know what you think. james > Rebased against r2564. > > James, you think you will apply this? Does it need more work? > > Signed-off-by: Tom Duffy -------------- next part -------------- Index: test/dapltest/test/dapl_performance_util.c =================================================================== --- test/dapltest/test/dapl_performance_util.c (revision 2572) +++ test/dapltest/test/dapl_performance_util.c (working copy) @@ -33,7 +33,7 @@ boolean_t DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -61,10 +61,10 @@ DT_Performance_Test_Create ( test_ptr->remote_ia_addr = remote_ia_addr; test_ptr->is_remote_little_endian = is_remote_little_endian; test_ptr->base_port = (DAT_CONN_QUAL) pt_ptr->Server_Info.first_port_number; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; - ret = dat_ia_query (test_ptr->ia_handle, + ret = dat_ia_query (test_ptr->ia, NULL, &test_ptr->ia_attr, NULL); @@ -95,17 +95,17 @@ DT_Performance_Test_Create ( test_ptr->creq_evd_length = DT_PERF_DFLT_EVD_LENGTH; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; return FALSE; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->recv_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG, @@ -114,11 +114,11 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; return FALSE; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->reqt_evd_length, test_ptr->cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -127,7 +127,7 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: 0x%x dat_evd_create (request) error: %s\n", test_ptr->reqt_evd_length,test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; return FALSE; } @@ -135,30 +135,30 @@ DT_Performance_Test_Create ( if ( is_server ) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->creq_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CR_FLAG, &test_ptr->creq_evd_hdl); /* cr */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; return FALSE; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->conn_evd_length, - DAT_HANDLE_NULL, + NULL, DAT_EVD_CONNECTION_FLAG, &test_ptr->conn_evd_hdl); /* conn */ if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; return FALSE; } @@ -176,18 +176,18 @@ DT_Performance_Test_Create ( test_ptr->ep_context.ep_attr.max_request_dtos = pipeline_len; /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &test_ptr->ep_context.ep_attr, /* EP attrs */ - &test_ptr->ep_context.ep_handle); + &test_ptr->ep_context.ep); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->ep_context.ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context.ep = NULL; return FALSE; } @@ -197,10 +197,10 @@ DT_Performance_Test_Create ( */ test_ptr->ep_context.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, + NULL, /* rmr */ DT_PERF_SYNC_BUFF_SIZE, 2, /* 2 RMIs */ 255, /* FIXME should query for this */ @@ -226,7 +226,7 @@ DT_Performance_Test_Create ( * Post recv and sync buffers */ if ( !DT_post_recv_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_RECV_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE) ) @@ -269,13 +269,13 @@ DT_Performance_Test_Destroy ( boolean_t is_server) { u32 ret; - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; DT_Tdep_Print_Head *phead; int status = 0; phead = pt_ptr->Params.phead; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ if (test_ptr->ep_context.op.bp) @@ -310,9 +310,9 @@ DT_Performance_Test_Destroy ( * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context.ep_handle) + if (test_ptr->ep_context.ep) { - ret = dat_ep_disconnect (test_ptr->ep_context.ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context.ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -323,7 +323,7 @@ DT_Performance_Test_Destroy ( status = 1; } else if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -331,10 +331,10 @@ DT_Performance_Test_Destroy ( } } - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free error: %s\n", @@ -394,9 +394,9 @@ DT_Performance_Test_Destroy ( } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -417,7 +417,7 @@ DT_Performance_Test_Destroy ( boolean_t DT_performance_post_rdma_op ( Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats) { unsigned int j; @@ -456,7 +456,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -473,7 +473,7 @@ DT_performance_post_rdma_op ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -502,17 +502,17 @@ DT_performance_post_rdma_op ( unsigned int DT_performance_reap ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Mode_Type mode, Performance_Stats_t *stats) { if ( BLOCKING_MODE == mode ) { - return DT_performance_wait (phead, evd_handle, stats); + return DT_performance_wait (phead, evd, stats); } else { - return DT_performance_poll (phead, evd_handle, stats); + return DT_performance_poll (phead, evd, stats); } } @@ -520,7 +520,7 @@ DT_performance_reap ( unsigned int DT_performance_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { int i; @@ -537,7 +537,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_wait ( evd_handle, + ret = DT_Tdep_evd_wait ( evd, DAT_TIMEOUT_MAX, &event); @@ -571,7 +571,7 @@ DT_performance_wait ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); @@ -611,7 +611,7 @@ DT_performance_wait ( unsigned int DT_performance_poll ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd * evd, Performance_Stats_t *stats) { u32 ret; @@ -626,7 +626,7 @@ DT_performance_poll ( pre_ctxt_num = DT_Mdep_GetContextSwitchNum (); pre_ts = DT_Mdep_GetTimeStamp (); - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: test/dapltest/test/dapl_performance_client.c =================================================================== --- test/dapltest/test/dapl_performance_client.c (revision 2572) +++ test/dapltest/test/dapl_performance_client.c (working copy) @@ -34,7 +34,7 @@ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr * remote_ia_addr) { Performance_Test_t *test_ptr = NULL; @@ -47,7 +47,7 @@ DT_Performance_Test_Client ( DT_Tdep_PT_Debug (1,(phead,"Client: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - ia_handle, + ia, remote_ia_addr, FALSE, pt_ptr->Server_Info.is_little_endian, @@ -102,7 +102,7 @@ DT_Performance_Test_Client_Connect ( test_ptr->base_port, test_ptr->ep_context.port)); retry: - ret = dat_ep_connect (test_ptr->ep_context.ep_handle, + ret = dat_ep_connect (test_ptr->ep_context.ep, test_ptr->remote_ia_addr, test_ptr->ep_context.port, DAT_TIMEOUT_MAX, @@ -119,7 +119,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -296,7 +296,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_write (ep_context->ep_handle, + ret = dat_ep_post_rdma_write (ep_context->ep, op->num_segs, iov, cookie, @@ -307,7 +307,7 @@ DT_Performance_Test_Client_Phase2 ( { pre_ts = DT_Mdep_GetTimeStamp (); - ret = dat_ep_post_rdma_read (ep_context->ep_handle, + ret = dat_ep_post_rdma_read (ep_context->ep, op->num_segs, iov, cookie, @@ -374,9 +374,9 @@ DT_Performance_Test_Client_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -405,7 +405,7 @@ DT_Performance_Test_Client_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) @@ -460,7 +460,7 @@ DT_Performance_Test_Client_Exchange ( DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Sending Sync Msg\n", test_ptr->base_port)); if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -477,7 +477,7 @@ DT_Performance_Test_Client_Exchange ( if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Client_Sync_Send")) Index: test/dapltest/test/dapl_cnxn.c =================================================================== --- test/dapltest/test/dapl_cnxn.c (revision 2532) +++ test/dapltest/test/dapl_cnxn.c (working copy) @@ -29,7 +29,7 @@ /****************************************************************************/ int -get_ep_connection_state (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle) +get_ep_connection_state (DT_Tdep_Print_Head *phead, struct dat_ep *ep) { enum dat_ep_state ep_state; boolean_t in_dto_idle; @@ -39,7 +39,7 @@ get_ep_connection_state (DT_Tdep_Print_H char *req_status = "Idle"; - ret = dat_ep_get_status (ep_handle, &ep_state, &in_dto_idle, + ret = dat_ep_get_status (ep, &ep_state, &in_dto_idle, &out_dto_idle); if (ret != 0) { Index: test/dapltest/test/dapl_server.c =================================================================== --- test/dapltest/test/dapl_server.c (revision 2532) +++ test/dapltest/test/dapl_server.c (working copy) @@ -103,7 +103,7 @@ DT_cs_Server (Params_t * params_ptr) ret = dat_ia_open (Server_Cmd->dapl_name, DFLT_QLEN, &ps_ptr->async_evd_hdl, - &ps_ptr->ia_handle); + &ps_ptr->ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -111,28 +111,28 @@ DT_cs_Server (Params_t * params_ptr) module, Server_Cmd->dapl_name, DT_RetToString (ret)); - ps_ptr->ia_handle = DAT_HANDLE_NULL; + ps_ptr->ia = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: IA %s opened\n", module, Server_Cmd->dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ps_ptr->ia_handle, &ps_ptr->pz_handle); + ret = dat_pz_create (ps_ptr->ia, &ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->pz_handle = DAT_HANDLE_NULL; + ps_ptr->pz = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: PZ created\n", module)); /* Create 4 events - recv, request, connection-request, connect */ - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -142,11 +142,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->recv_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -156,11 +156,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->reqt_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CR_FLAG, @@ -170,11 +170,11 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (cr) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->creq_evd_hdl = NULL; status = 1; goto server_exit; } - ret = DT_Tdep_evd_create (ps_ptr->ia_handle, + ret = DT_Tdep_evd_create (ps_ptr->ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -184,44 +184,44 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - ps_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + ps_ptr->conn_evd_hdl = NULL; status = 1; goto server_exit; } /* Create the EP */ - ret = dat_ep_create (ps_ptr->ia_handle, /* IA */ - ps_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (ps_ptr->ia, /* IA */ + ps_ptr->pz, /* PZ */ ps_ptr->recv_evd_hdl, /* recv */ ps_ptr->reqt_evd_hdl, /* request */ ps_ptr->conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ps_ptr->ep_handle); + &ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->ep_handle = DAT_HANDLE_NULL; + ps_ptr->ep = NULL; status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: EP created\n", module)); /* Create PSP */ - ret = dat_psp_create (ps_ptr->ia_handle, + ret = dat_psp_create (ps_ptr->ia, SERVER_PORT_NUMBER, ps_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &ps_ptr->psp_handle); + &ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_create error: %s\n", module, DT_RetToString (ret)); - ps_ptr->psp_handle = DAT_HANDLE_NULL; + ps_ptr->psp = NULL; status = 1; goto server_exit; } @@ -233,10 +233,10 @@ DT_cs_Server (Params_t * params_ptr) */ ps_ptr->bpool = DT_BpoolAlloc (NULL, phead, - ps_ptr->ia_handle, - ps_ptr->pz_handle, - ps_ptr->ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ps_ptr->ia, + ps_ptr->pz, + ps_ptr->ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ 256, /* FIXME query for this value */ @@ -275,7 +275,7 @@ DT_cs_Server (Params_t * params_ptr) */ for (;/* EVER */;) { - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; @@ -299,7 +299,7 @@ DT_cs_Server (Params_t * params_ptr) /* Server_Info, Client_Info, Params set up below */ /* Gather whatever info we want about defaults */ - if (!DT_query (pt_ptr, ps_ptr->ia_handle, ps_ptr->ep_handle)) + if (!DT_query (pt_ptr, ps_ptr->ia, ps_ptr->ep)) { status = 1; goto server_exit; @@ -308,7 +308,7 @@ DT_cs_Server (Params_t * params_ptr) /* Post recv buffers for ClientInfo and Transaction_Cmd_t */ DT_Tdep_PT_Debug (1,(phead,"%s: Posting 2 recvs\n", module)); if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 0, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0))) @@ -320,7 +320,7 @@ DT_cs_Server (Params_t * params_ptr) goto server_exit; } if (!DT_post_recv_buffer (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->bpool, 1, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1))) @@ -343,9 +343,9 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_cr_event_wait (phead, ps_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check (phead, &cr_stat, - ps_ptr->psp_handle, + ps_ptr->psp, SERVER_PORT_NUMBER, - &cr_handle, + &cr, module)) { @@ -357,7 +357,7 @@ DT_cs_Server (Params_t * params_ptr) } DT_Tdep_PT_Debug (1,(phead,"%s: Accepting Connection Request\n", module)); - ret = dat_cr_accept (cr_handle, ps_ptr->ep_handle, 0, (void *)0); + ret = dat_cr_accept (cr, ps_ptr->ep, 0, (void *)0); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -370,7 +370,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Debug (1,(phead,"%s: Awaiting connection ...\n", module)); if (!DT_conn_event_wait (phead, - ps_ptr->ep_handle, + ps_ptr->ep, ps_ptr->conn_evd_hdl, &event_num)) { @@ -384,7 +384,7 @@ DT_cs_Server (Params_t * params_ptr) if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ps_ptr->ep_handle); + get_ep_connection_state (phead, ps_ptr->ep); } /* Wait for Client_Info */ @@ -395,7 +395,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 0), dto_cookie, "Client_Info_Recv")) @@ -420,7 +420,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 1), dto_cookie, "Client_Cmd_Recv")) @@ -585,7 +585,7 @@ DT_cs_Server (Params_t * params_ptr) /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -601,7 +601,7 @@ DT_cs_Server (Params_t * params_ptr) if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) @@ -619,7 +619,7 @@ DT_cs_Server (Params_t * params_ptr) /* we passed the pt_ptr to the thread and must now 'forget' it */ pt_ptr = NULL; - ret = dat_ep_disconnect (ps_ptr->ep_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect fails: %s\n", @@ -635,7 +635,7 @@ DT_cs_Server (Params_t * params_ptr) } /* reset the EP to get back into the game */ - dat_ep_reset (ps_ptr->ep_handle); + dat_ep_reset (ps_ptr->ep); DT_Tdep_PT_Debug (1,(phead,"%s: Waiting for another client...\n", module)); } /* end loop accepting connections */ @@ -669,9 +669,9 @@ server_exit: * we also get here on error, hence abrupt closure to * flush any lingering buffers posted. */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_disconnect (ps_ptr->ep_handle, + ret = dat_ep_disconnect (ps_ptr->ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -704,9 +704,9 @@ server_exit: } /* Free the PSP */ - if (ps_ptr->psp_handle) + if (ps_ptr->psp) { - ret = dat_psp_free (ps_ptr->psp_handle); + ret = dat_psp_free (ps_ptr->psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_psp_free error: %s\n", @@ -718,9 +718,9 @@ server_exit: } /* Free the EP */ - if (ps_ptr->ep_handle) + if (ps_ptr->ep) { - ret = dat_ep_free (ps_ptr->ep_handle); + ret = dat_ep_free (ps_ptr->ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free error: %s\n", @@ -781,9 +781,9 @@ server_exit: } /* Free the PZ */ - if (ps_ptr->pz_handle) + if (ps_ptr->pz) { - ret = dat_pz_free (ps_ptr->pz_handle); + ret = dat_pz_free (ps_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free error: %s\n", @@ -794,17 +794,17 @@ server_exit: } /* Close the IA */ - if (ps_ptr->ia_handle) + if (ps_ptr->ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) error: %s\n", module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ps_ptr->ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -871,7 +871,7 @@ send_control_data ( sizeof (Server_Info_t)); DT_Server_Info_Endian ((Server_Info_t *) buffp); - if (!DT_post_send_buffer ( phead, ps_ptr->ep_handle, + if (!DT_post_send_buffer ( phead, ps_ptr->ep, ps_ptr->bpool, 2, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) @@ -886,7 +886,7 @@ send_control_data ( if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - ps_ptr->ep_handle, + ps_ptr->ep, DT_Bpool_GetBuffSize (ps_ptr->bpool, 2), dto_cookie, "Server_Info_Send")) Index: test/dapltest/test/dapl_fft_util.c =================================================================== --- test/dapltest/test/dapl_fft_util.c (revision 2532) +++ test/dapltest/test/dapl_fft_util.c (working copy) @@ -55,22 +55,22 @@ void DT_assert_fail (DT_Tdep_Print_Head } /* helper function to open an IA */ -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle) +int DT_ia_open (char *dev_name, struct dat_ia **ia) { - DAT_EVD_HANDLE evd_handle; - evd_handle = DAT_HANDLE_NULL; - return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, ia_handle); + struct dat_evd *evd; + evd = NULL; + return dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, ia); } /* helper function to create an endpoint and its associated EVDs */ int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle) + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep) { u32 status; DT_Tdep_Print_Head *phead; @@ -80,7 +80,7 @@ int DT_ep_create (Params_t *params_ptr, *cr_evd = NULL; phead = params_ptr->phead; - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CR_FLAG, cr_evd); if (status != DAT_SUCCESS) { @@ -88,7 +88,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_CONNECTION_FLAG, conn_evd); if (status != DAT_SUCCESS) { @@ -96,7 +96,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, send_evd); if (status != DAT_SUCCESS) @@ -105,7 +105,7 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, + status = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, recv_evd); if (status != DAT_SUCCESS) { @@ -113,8 +113,8 @@ int DT_ep_create (Params_t *params_ptr, return status; } - status = dat_ep_create (ia_handle, pz_handle, *recv_evd, - *send_evd, *conn_evd, NULL, ep_handle); + status = dat_ep_create (ia, pz, *recv_evd, + *send_evd, *conn_evd, NULL, ep); if (status != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "dat_ep_create failed %s\n", DT_RetToString (status)); @@ -125,15 +125,15 @@ int DT_ep_create (Params_t *params_ptr, /* function that initializes the connection struct */ void DT_fft_init_conn_struct (FFT_Connection_t *conn) { - conn->ia_handle = NULL; - conn->pz_handle = NULL; - conn->psp_handle = NULL; - conn->ep_handle = NULL; + conn->ia = NULL; + conn->pz = NULL; + conn->psp = NULL; + conn->ep = NULL; conn->cr_evd = NULL; conn->send_evd = NULL; conn->conn_evd = NULL; conn->recv_evd = NULL; - conn->cr_handle = NULL; + conn->cr = NULL; conn->remote_netaddr = NULL; conn->bpool = NULL; conn->pt_ptr = NULL; @@ -154,27 +154,27 @@ void DT_fft_init_client (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); if (rc != DAT_SUCCESS) { /* make sure the handle has an invalid value */ - conn->ia_handle = NULL; + conn->ia = NULL; } DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* if a server name is given, allocate memory for a net address and set it @@ -198,11 +198,11 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_Print_Head *phead; phead = params_ptr->phead; - if (conn->ep_handle) + if (conn->ep) { if (conn->connected) { - rc = dat_ep_disconnect (conn->ep_handle, DAT_CLOSE_DEFAULT); + rc = dat_ep_disconnect (conn->ep, DAT_CLOSE_DEFAULT); DT_assert_clean (phead, rc == DAT_SUCCESS); if (!DT_disco_event_wait ( phead, conn->cr_evd, NULL )) @@ -211,16 +211,16 @@ int DT_fft_destroy_conn_struct (Params_t DT_Tdep_PT_Printf (phead, "DT_fft_destroy_conn_struct: bad disconnect event\n"); } } - rc = dat_ep_free (conn->ep_handle); + rc = dat_ep_free (conn->ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->bpool) { DT_Bpool_Destroy (NULL, phead, conn->bpool); } - if (conn->psp_handle) + if (conn->psp) { - rc = dat_psp_free (conn->psp_handle); + rc = dat_psp_free (conn->psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn->cr_evd) @@ -247,14 +247,14 @@ int DT_fft_destroy_conn_struct (Params_t { DT_Free_Per_Test_Data (conn->pt_ptr); } - if (conn->pz_handle) + if (conn->pz) { - rc = dat_pz_free (conn->pz_handle); + rc = dat_pz_free (conn->pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn->ia_handle) + if (conn->ia) { - rc = dat_ia_close (conn->ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (conn->ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return rc; @@ -272,31 +272,31 @@ void DT_fft_init_server (Params_t *param DT_fft_init_conn_struct (conn); /* open the IA */ - rc = DT_ia_open (cmd->device_name, &conn->ia_handle); + rc = DT_ia_open (cmd->device_name, &conn->ia); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PZ */ - rc = dat_pz_create (conn->ia_handle, &conn->pz_handle); + rc = dat_pz_create (conn->ia, &conn->pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create an EP and its EVDs */ rc =DT_ep_create (params_ptr, - conn->ia_handle, - conn->pz_handle, + conn->ia, + conn->pz, &conn->cr_evd, &conn->conn_evd, &conn->send_evd, &conn->recv_evd, - &conn->ep_handle); + &conn->ep); DT_assert_dat (phead, rc == DAT_SUCCESS); /* create a PSP */ - rc = dat_psp_create (conn->ia_handle, SERVER_PORT_NUMBER, conn->cr_evd, - DAT_PSP_CONSUMER_FLAG, &conn->psp_handle); + rc = dat_psp_create (conn->ia, SERVER_PORT_NUMBER, conn->cr_evd, + DAT_PSP_CONSUMER_FLAG, &conn->psp); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate memory for buffers */ - conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia_handle, conn->pz_handle, + conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia, conn->pz, NULL, NULL, 8192, 2, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, conn->bpool); @@ -316,19 +316,19 @@ void DT_fft_listen (Params_t *params_ptr DT_assert_dat (phead, DT_cr_event_wait (phead, conn->cr_evd, &conn->cr_stat) && DT_cr_check (phead, &conn->cr_stat, - conn->psp_handle, + conn->psp, SERVER_PORT_NUMBER, - &conn->cr_handle, + &conn->cr, "DT_fft_listen")); /* accept the connection */ - rc =dat_cr_accept (conn->cr_handle, conn->ep_handle, 0, (void *)0); + rc =dat_cr_accept (conn->cr, conn->ep, 0, (void *)0); DT_assert_dat (phead, rc == DAT_SUCCESS); /* wait on a conn event via the conn EVD */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); conn->connected = TRUE; @@ -351,7 +351,7 @@ int DT_fft_connect (Params_t *params_ptr DT_Tdep_PT_Printf (phead, "Connection to server, attempt #%d\n", wait_count+1); /* attempt to connect, timeout = 10 secs */ - rc = dat_ep_connect (conn->ep_handle, conn->remote_netaddr, + rc = dat_ep_connect (conn->ep, conn->remote_netaddr, SERVER_PORT_NUMBER, 10*1000000, 0, (void *)0, DAT_QOS_BEST_EFFORT, DAT_CONNECT_DEFAULT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); @@ -359,7 +359,7 @@ int DT_fft_connect (Params_t *params_ptr /* wait on conn event */ DT_assert (phead, DT_conn_event_wait (phead, - conn->ep_handle, + conn->ep, conn->conn_evd, &conn->event_num) == TRUE); Index: test/dapltest/test/dapl_fft_mem.c =================================================================== --- test/dapltest/test/dapl_fft_mem.c (revision 2532) +++ test/dapltest/test/dapl_fft_mem.c (working copy) @@ -37,28 +37,28 @@ int DT_mem_generic (Params_t *params_ptr FFT_Connection_t conn; DAT_REGION_DESCRIPTION region; u64 reg_size; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; DAT_LMR_CONTEXT lmr_context; u64 reg_addr; unsigned char *alloc_ptr; int res; u64 buffer_size; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; rc = 0; expect = 0; res = 1; - lmr_handle = NULL; + lmr = NULL; lmr_context = 0; reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; DT_fft_init_client (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); if (flag == 2) { @@ -76,24 +76,24 @@ int DT_mem_generic (Params_t *params_ptr memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - ia_handle = conn.ia_handle; + ia = conn.ia; if (flag != 3) { - pz_handle = conn.pz_handle; + pz = conn.pz; } if (flag != 4) { DT_Tdep_PT_Printf (phead, "Registering memory\n"); - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -110,21 +110,21 @@ int DT_mem_generic (Params_t *params_ptr } if (flag == 1) { - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_dat (phead, rc == DAT_SUCCESS); } - lmr_handle = NULL; + lmr = NULL; - rc = dat_lmr_kcreate (conn.ia_handle, + rc = dat_lmr_kcreate (conn.ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - conn.pz_handle, + conn.pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, @@ -133,9 +133,9 @@ int DT_mem_generic (Params_t *params_ptr } cleanup: - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (alloc_ptr) @@ -197,7 +197,7 @@ int DT_mem_case4 ( Params_t *params_ptr, DT_Tdep_Print_Head *phead; phead = params_ptr->phead; DT_Tdep_PT_Printf (phead, "\ - Description: Try to deregister memory with null lmr_handle\n"); + Description: Try to deregister memory with null lmr\n"); return DT_mem_generic (params_ptr, cmd, 4); } Index: test/dapltest/test/dapl_limit.c =================================================================== --- test/dapltest/test/dapl_limit.c (revision 2532) +++ test/dapltest/test/dapl_limit.c (working copy) @@ -33,13 +33,13 @@ static boolean_t more_handles (DT_Tdep_Print_Head *phead, - DAT_HANDLE **old_ptrptr, /* pointer to current pointer */ + void ***old_ptrptr, /* pointer to current pointer */ unsigned int *old_count, /* number pointed to */ unsigned int size) /* size of one datum */ { unsigned int count = *old_count; - DAT_HANDLE *old_handles = *old_ptrptr; - DAT_HANDLE *handle_tmp = DT_Mdep_Malloc (count * 2 * size); + void * *old_handles = *old_ptrptr; + void * *handle_tmp = DT_Mdep_Malloc (count * 2 * size); if (!handle_tmp) { @@ -74,19 +74,19 @@ limit_test ( DT_Tdep_Print_Head *phead, Limit_Cmd_t *cmd, Limit_Index depth) { - DAT_EVD_HANDLE conn_handle; + struct dat_evd * conn_handle; typedef struct obj_set { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; - DAT_LMR_HANDLE lmr_handle; + struct dat_evd * evd; + struct dat_ep * ep; + struct dat_lmr * lmr; char * lmr_buffer; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; } Obj_Set; @@ -135,19 +135,19 @@ limit_test ( DT_Tdep_Print_Head *phead, for (w = 0; w < cmd->width; w++) { /* Specify that we want to get back an async EVD. */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdl_sets[w].ia_async_handle, - &hdl_sets[w].ia_handle); + &hdl_sets[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", module, cmd->device_name, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ia_async_handle = DAT_HANDLE_NULL; - hdl_sets[w].ia_handle = DAT_HANDLE_NULL; + hdl_sets[w].ia_async_handle = NULL; + hdl_sets[w].ia = NULL; goto clean_up_now; } } @@ -159,8 +159,8 @@ limit_test ( DT_Tdep_Print_Head *phead, */ typedef struct _ia { - DAT_IA_HANDLE ia_handle; - DAT_EVD_HANDLE ia_async_handle; + struct dat_ia * ia; + struct dat_evd * ia_async_handle; } OneOpen; unsigned int count = START_COUNT; @@ -178,7 +178,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -187,11 +187,11 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } /* Specify that we want to get back an async EVD. */ - hdlptr[w].ia_async_handle = DAT_HANDLE_NULL; + hdlptr[w].ia_async_handle = NULL; ret = dat_ia_open (cmd->device_name, DFLT_QLEN, &hdlptr[w].ia_async_handle, - &hdlptr[w].ia_handle); + &hdlptr[w].ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", @@ -209,14 +209,14 @@ limit_test ( DT_Tdep_Print_Head *phead, for (tmp = 0; tmp < w; tmp++) { DT_Mdep_Schedule(); - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) fails: %s\n", module, DT_RetToString (ret)); retval = FALSE; - ret = dat_ia_close (hdlptr[tmp].ia_handle, + ret = dat_ia_close (hdlptr[tmp].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -245,14 +245,14 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_pz_create (hdl_sets[w].ia_handle, - &hdl_sets[w].pz_handle); + ret = dat_pz_create (hdl_sets[w].ia, + &hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].pz_handle = DAT_HANDLE_NULL; + hdl_sets[w].pz = NULL; goto clean_up_now; } } @@ -263,7 +263,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PZs we can create */ unsigned int count = START_COUNT; - DAT_PZ_HANDLE *hdlptr = (DAT_PZ_HANDLE *) + struct dat_pz **hdlptr = (struct dat_pz **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PZ Exhaustion test loop */ @@ -279,7 +279,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -287,7 +287,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_pz_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_pz_create (hdl_sets[w % cmd->width].ia, &hdlptr[w]); if (ret != DAT_SUCCESS) { @@ -335,13 +335,13 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_cno_create (hdl_sets[w].ia_handle, + ret = dat_cno_create (hdl_sets[w].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdl_sets[w].cno_handle); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { DT_Tdep_PT_Printf (phead, "%s: dat_cno_create unimplemented\n", module); - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; /* ignore this error */ break; } @@ -350,7 +350,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_cno_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].cno_handle = DAT_HANDLE_NULL; + hdl_sets[w].cno_handle = NULL; goto clean_up_now; } } @@ -377,7 +377,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -385,7 +385,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_cno_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_cno_create (hdl_sets[w % cmd->width].ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) @@ -446,7 +446,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -456,22 +456,22 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; goto clean_up_now; } for (w = 0; w < cmd->width; w++) { - ret = DT_Tdep_evd_create (hdl_sets[w].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w].ia, DFLT_QLEN, hdl_sets[w].cno_handle, flags, - &hdl_sets[w].evd_handle); + &hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].evd_handle = DAT_HANDLE_NULL; + hdl_sets[w].evd = NULL; goto clean_up_now; } } @@ -482,7 +482,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EVDs we can create */ unsigned int count = START_COUNT; - DAT_EVD_HANDLE *hdlptr = (DAT_EVD_HANDLE *) + struct dat_evd **hdlptr = (struct dat_evd **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); enum dat_evd_flags flags = ( DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG @@ -500,7 +500,7 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * First create a connection EVD to be used for EP creation */ - ret = DT_Tdep_evd_create (hdl_sets[0].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[0].ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -510,13 +510,13 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: conn dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->maximum; w++) { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -524,7 +524,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia_handle, + ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia, DFLT_QLEN, hdl_sets[w % cmd->width].cno_handle, flags, @@ -542,10 +542,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; /* EVD Cleanup loop */ - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (tmp = 0; tmp < w; tmp++) { @@ -578,19 +578,19 @@ limit_test ( DT_Tdep_Print_Head *phead, cmd->width)); for (w = 0; w < cmd->width; w++) { - ret = dat_ep_create (hdl_sets[w].ia_handle, - hdl_sets[w].pz_handle, - hdl_sets[w].evd_handle, /* recv */ - hdl_sets[w].evd_handle, /* request */ + ret = dat_ep_create (hdl_sets[w].ia, + hdl_sets[w].pz, + hdl_sets[w].evd, /* recv */ + hdl_sets[w].evd, /* request */ conn_handle, /* connect */ (struct dat_ep_attr *) NULL, - &hdl_sets[w].ep_handle); + &hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].ep_handle = DAT_HANDLE_NULL; + hdl_sets[w].ep = NULL; goto clean_up_now; } } @@ -601,7 +601,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many EPs we can create */ unsigned int count = START_COUNT; - DAT_EP_HANDLE *hdlptr = (DAT_EP_HANDLE *) + struct dat_ep **hdlptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* EP Exhaustion test loop */ @@ -615,7 +615,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -623,10 +623,10 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, /* connect */ (struct dat_ep_attr *) NULL, &hdlptr[w]); @@ -672,9 +672,9 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many RSPs we can create */ unsigned int count = START_COUNT; - DAT_RSP_HANDLE *hdlptr = (DAT_RSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); - DAT_EP_HANDLE *epptr = (DAT_EP_HANDLE *) + struct dat_ep **epptr = (struct dat_ep **) DT_Mdep_Malloc (count * sizeof (*epptr)); /* RSP Exhaustion test loop */ @@ -692,7 +692,7 @@ limit_test ( DT_Tdep_Print_Head *phead, unsigned int count1 = count; unsigned int count2 = count; - if (!more_handles (phead, (DAT_HANDLE **) &hdlptr, + if (!more_handles (phead, (void ***) &hdlptr, &count1, sizeof (*hdlptr))) { @@ -700,7 +700,7 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - if (!more_handles (phead, (DAT_HANDLE **) &epptr, + if (!more_handles (phead, (void ***) &epptr, &count2, sizeof (*epptr))) { @@ -723,10 +723,10 @@ limit_test ( DT_Tdep_Print_Head *phead, /* * Each RSP needs a unique EP, so create one first */ - ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].evd_handle, - hdl_sets[w % cmd->width].evd_handle, + ret = dat_ep_create (hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].evd, + hdl_sets[w % cmd->width].evd, conn_handle, (struct dat_ep_attr *) NULL, &epptr[w]); @@ -738,10 +738,10 @@ limit_test ( DT_Tdep_Print_Head *phead, break; } - ret = dat_rsp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_rsp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, epptr[w], - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, &hdlptr[w]); if (DAT_GET_TYPE (ret) == DAT_NOT_IMPLEMENTED) { @@ -808,7 +808,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * See how many PSPs we can create */ unsigned int count = START_COUNT; - DAT_PSP_HANDLE *hdlptr = (DAT_PSP_HANDLE *) + struct dat_sp **hdlptr = (struct dat_sp **) DT_Mdep_Malloc (count * sizeof (*hdlptr)); /* PSP Exhaustion test loop */ @@ -822,7 +822,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -830,9 +830,9 @@ limit_test ( DT_Tdep_Print_Head *phead, retval = TRUE; break; } - ret = dat_psp_create (hdl_sets[w % cmd->width].ia_handle, + ret = dat_psp_create (hdl_sets[w % cmd->width].ia, CONN_QUAL0 + w, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].evd, DAT_PSP_CONSUMER_FLAG, &hdlptr[w]); if (ret != DAT_SUCCESS) @@ -840,7 +840,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_psp_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); retval = TRUE; - hdlptr[w] = DAT_HANDLE_NULL; + hdlptr[w] = NULL; break; } } @@ -897,14 +897,14 @@ limit_test ( DT_Tdep_Print_Head *phead, memset (®ion, 0, sizeof (region)); region.for_va = hdl_sets[w].lmr_buffer; - ret = dat_lmr_kcreate (hdl_sets[w].ia_handle, + ret = dat_lmr_kcreate (hdl_sets[w].ia, DAT_MEM_TYPE_VIRTUAL, region, DFLT_BUFFSZ, - hdl_sets[w].pz_handle, + hdl_sets[w].pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &hdl_sets[w].lmr_handle, + &hdl_sets[w].lmr, &hdl_sets[w].lmr_context, NULL, /* FIXME */ ®_size, ®_addr); @@ -913,7 +913,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_lmr_kcreate #%d fails: %s\n", module, w+1, DT_RetToString (ret)); /* handle contents undefined on failure */ - hdl_sets[w].lmr_handle = DAT_HANDLE_NULL; + hdl_sets[w].lmr = NULL; goto clean_up_now; } if ((uintptr_t)reg_addr > (uintptr_t)hdl_sets[w].lmr_buffer @@ -949,7 +949,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, sizeof (*hdlptr))) { @@ -966,10 +966,10 @@ limit_test ( DT_Tdep_Print_Head *phead, */ hdlptr[w] = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[w % cmd->width].ia_handle, - hdl_sets[w % cmd->width].pz_handle, - hdl_sets[w % cmd->width].ep_handle, - hdl_sets[w % cmd->width].evd_handle, + hdl_sets[w % cmd->width].ia, + hdl_sets[w % cmd->width].pz, + hdl_sets[w % cmd->width].ep, + hdl_sets[w % cmd->width].evd, DFLT_BUFFSZ, 1, 256, /* FIXME should query for this */ @@ -1026,7 +1026,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Mdep_Schedule(); if (w == count - && !more_handles (phead, (DAT_HANDLE **) &hdlptr, + && !more_handles (phead, (void ***) &hdlptr, &count, cmd->width * sizeof (*hdlptr))) { @@ -1054,7 +1054,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_ep_post_recv #%d\n", module, w * cmd->width + i + 1); - ret = dat_ep_post_recv (hdl_sets[i].ep_handle, + ret = dat_ep_post_recv (hdl_sets[i].ep, 1, iovp, cookie, @@ -1091,7 +1091,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * outstanding recv DTOs in error, and otherwise * be a no-op. */ - ret = dat_ep_reset (hdl_sets[i].ep_handle); + ret = dat_ep_reset (hdl_sets[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect (abrupt) fails: %s\n", @@ -1107,7 +1107,7 @@ limit_test ( DT_Tdep_Print_Head *phead, */ do { - ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd_handle, + ret = DT_Tdep_evd_dequeue ( hdl_sets[i].evd, &event); } while (ret == DAT_SUCCESS); } @@ -1130,10 +1130,10 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Mdep_Schedule(); test_bpool = DT_BpoolAlloc ((Per_Test_Data_t *)0, phead, - hdl_sets[0].ia_handle, - hdl_sets[0].pz_handle, - hdl_sets[0].ep_handle, - hdl_sets[0].evd_handle, + hdl_sets[0].ia, + hdl_sets[0].pz, + hdl_sets[0].ep, + hdl_sets[0].evd, test_size, 1, 256, /* FIXME should query for this */ @@ -1185,9 +1185,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].lmr_handle) + if (hdl_sets[w].lmr) { - ret = dat_lmr_free (hdl_sets[w].lmr_handle); + ret = dat_lmr_free (hdl_sets[w].lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_lmr_free fails: %s\n", @@ -1220,9 +1220,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ep_handle) + if (hdl_sets[w].ep) { - ret = dat_ep_free (hdl_sets[w].ep_handle); + ret = dat_ep_free (hdl_sets[w].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free fails: %s\n", @@ -1237,16 +1237,16 @@ clean_up_now: { unsigned int w; - if (conn_handle != DAT_HANDLE_NULL) + if (conn_handle != NULL) { ret = DT_Tdep_evd_free (conn_handle); - conn_handle = DAT_HANDLE_NULL; + conn_handle = NULL; } for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].evd_handle) + if (hdl_sets[w].evd) { - ret = DT_Tdep_evd_free (hdl_sets[w].evd_handle); + ret = DT_Tdep_evd_free (hdl_sets[w].evd); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_evd_free fails: %s\n", @@ -1284,9 +1284,9 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].pz_handle) + if (hdl_sets[w].pz) { - ret = dat_pz_free (hdl_sets[w].pz_handle); + ret = dat_pz_free (hdl_sets[w].pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free fails: %s\n", @@ -1303,10 +1303,10 @@ clean_up_now: for (w = 0; w < cmd->width; w++) { - if (hdl_sets[w].ia_handle) + if (hdl_sets[w].ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { @@ -1318,7 +1318,7 @@ clean_up_now: * we may as well try the largest hammer we have. */ retval = FALSE; - ret = dat_ia_close (hdl_sets[w].ia_handle, + ret = dat_ia_close (hdl_sets[w].ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { Index: test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- test/dapltest/test/dapl_fft_queryinfo.c (revision 2532) +++ test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -37,31 +37,31 @@ int DT_queryinfo_basic (Params_t *params u32 result_wanted) { char *dev_name; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; struct dat_ia_attr ia_attributes; struct dat_provider_attr provider_attributes; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE cr_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *cr_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; + struct dat_ep *ep; struct dat_ep_param ep_param; DAT_CNO_HANDLE cno_handle; #ifndef __KDAPLTEST__ DAT_CNO_PARAM cno_param; #endif struct dat_evd_param evd_param; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * psp; struct dat_psp_param psp_param; - DAT_RSP_HANDLE rsp_handle; + struct dat_sp * rsp; struct dat_rsp_param rsp_param; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; struct dat_pz_param pz_param; - DAT_LMR_HANDLE lmr_handle; + struct dat_lmr *lmr; struct dat_lmr_param lmr_param; DAT_LMR_CONTEXT lmr_context; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; struct dat_rmr_param rmr_param; DAT_REGION_DESCRIPTION region; u64 reg_size; @@ -77,30 +77,30 @@ int DT_queryinfo_basic (Params_t *params reg_addr = 0; alloc_ptr = NULL; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - lmr_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + lmr = NULL; rmr_handle = NULL; - pz_handle = NULL; - psp_handle = NULL; - rsp_handle = NULL; + pz = NULL; + psp = NULL; + rsp = NULL; cno_handle = NULL; - evd_handle = DAT_HANDLE_NULL; - conn_evd_handle = DAT_HANDLE_NULL; - cr_evd_handle = DAT_HANDLE_NULL; - recv_evd_handle = DAT_HANDLE_NULL; - send_evd_handle = DAT_HANDLE_NULL; + evd = NULL; + conn_evd = NULL; + cr_evd = NULL; + recv_evd = NULL; + send_evd = NULL; dev_name = cmd->device_name; - /* All functions require an ia_handle to be created */ + /* All functions require an ia to be created */ rc = dat_ia_open ((const char *)dev_name, DEFAULT_QUEUE_LEN, - &evd_handle, - &ia_handle); + &evd, + &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - /* These functions require a pz_handle to be created */ + /* These functions require a pz to be created */ if ( (object_to_query == QUERY_EVD) || (object_to_query == QUERY_RMR) || (object_to_query == QUERY_LMR) || @@ -108,43 +108,43 @@ int DT_queryinfo_basic (Params_t *params (object_to_query == QUERY_RSP) || (object_to_query == QUERY_PZ) ) { - rc = dat_pz_create (ia_handle, - &pz_handle); + rc = dat_pz_create (ia, + &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - /* These functions require a ep_handle to be created */ + /* These functions require a ep to be created */ if ( (object_to_query == QUERY_EP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CONNECTION_FLAG, - &conn_evd_handle); + &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, - pz_handle, - recv_evd_handle, - send_evd_handle, - conn_evd_handle, + rc = dat_ep_create (ia, + pz, + recv_evd, + send_evd, + conn_evd, NULL, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -152,11 +152,11 @@ int DT_queryinfo_basic (Params_t *params if ( (object_to_query == QUERY_PSP) || (object_to_query == QUERY_RSP) ) { - rc = DT_Tdep_evd_create (ia_handle, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, cno_handle, DAT_EVD_CR_FLAG, - &cr_evd_handle); + &cr_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -165,8 +165,8 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -177,15 +177,15 @@ int DT_queryinfo_basic (Params_t *params * NULL out ia_attr and for the DAT_IA_ATTR_MASK to * have values */ - rc = dat_ia_query (ia_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, NULL, &provider_attributes); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_ia_query (evd_handle, - &evd_handle, + rc = dat_ia_query (ia, + &evd, &ia_attributes, &provider_attributes); } @@ -197,11 +197,11 @@ int DT_queryinfo_basic (Params_t *params #ifndef __KDAPLTEST__ #if defined(WIN32) - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, NULLPROXY, &cno_handle); #else - rc = dat_cno_create (ia_handle, + rc = dat_cno_create (ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &cno_handle); #endif @@ -220,7 +220,7 @@ int DT_queryinfo_basic (Params_t *params } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_cno_query (ia_handle, + rc = dat_cno_query (ia, &cno_param); } #endif @@ -230,17 +230,17 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, &evd_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_evd_query (evd_handle, + rc = dat_evd_query (evd, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_evd_query (ia_handle, + rc = dat_evd_query (evd, &evd_param); } } @@ -248,25 +248,25 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_psp_query function */ else if (object_to_query == QUERY_PSP) { - rc = dat_psp_create (ia_handle, + rc = dat_psp_create (ia, SERVER_PORT_NUMBER, - cr_evd_handle, + cr_evd, DAT_PSP_PROVIDER_FLAG, - &psp_handle); + &psp); DT_assert_dat (phead, rc == DAT_SUCCESS); if (result_wanted == DAT_SUCCESS) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, &psp_param); } else if (result_wanted == DAT_INVALID_PARAMETER) { - rc = dat_psp_query (psp_handle, + rc = dat_psp_query (psp, NULL); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_psp_query (evd_handle, + rc = dat_psp_query (psp, &psp_param); } } @@ -274,13 +274,13 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_rsp_query function */ else if (object_to_query == QUERY_RSP) { - rc = dat_rsp_create (ia_handle, + rc = dat_rsp_create (ia, SERVER_PORT_NUMBER, - ep_handle, - cr_evd_handle, - &rsp_handle); + ep, + cr_evd, + &rsp); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_rsp_query (rsp_handle, + rc = dat_rsp_query (rsp, &rsp_param); } @@ -294,14 +294,14 @@ int DT_queryinfo_basic (Params_t *params /* Test dat_ep_query function */ else if (object_to_query == QUERY_EP) { - rc = dat_ep_query (ep_handle, + rc = dat_ep_query (ep, &ep_param); } /* Test dat_pz_query function */ else if (object_to_query == QUERY_PZ) { - rc = dat_pz_query (pz_handle, + rc = dat_pz_query (pz, &pz_param); } @@ -312,27 +312,27 @@ int DT_queryinfo_basic (Params_t *params DT_assert (phead, alloc_ptr); memset (®ion, 0, sizeof (region)); region.for_va = alloc_ptr; - rc = dat_lmr_kcreate (ia_handle, + rc = dat_lmr_kcreate (ia, DAT_MEM_TYPE_VIRTUAL, region, buffer_size, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &lmr_handle, + &lmr, &lmr_context, NULL, /* FIXME */ ®_size, ®_addr); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_lmr_query (lmr_handle, + rc = dat_lmr_query (lmr, &lmr_param); } /* Test dat_rmr_query function */ else if (object_to_query == QUERY_RMR) { - rc = dat_rmr_create (pz_handle, + rc = dat_rmr_create (pz, &rmr_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_rmr_query (rmr_handle, @@ -342,39 +342,39 @@ int DT_queryinfo_basic (Params_t *params DT_assert_dat (phead, DAT_GET_TYPE (rc) == result_wanted); cleanup: - if (rsp_handle) + if (rsp) { - rc = dat_rsp_free (rsp_handle); + rc = dat_rsp_free (rsp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (lmr_handle) + if (lmr) { - rc = dat_lmr_free (lmr_handle); + rc = dat_lmr_free (lmr); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -391,27 +391,27 @@ cleanup: DT_assert_clean (phead, rc == DAT_SUCCESS); } #endif - if (psp_handle) + if (psp) { - rc = dat_psp_free (psp_handle); + rc = dat_psp_free (psp); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (cr_evd_handle) + if (cr_evd) { - rc = DT_Tdep_evd_free (cr_evd_handle); + rc = DT_Tdep_evd_free (cr_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: test/dapltest/test/dapl_transaction_test.c =================================================================== --- test/dapltest/test/dapl_transaction_test.c (revision 2532) +++ test/dapltest/test/dapl_transaction_test.c (working copy) @@ -47,7 +47,7 @@ /****************************************************************************/ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr) { Transaction_Cmd_t *cmd = &pt_ptr->Params.u.Transaction_Cmd; @@ -68,7 +68,7 @@ DT_Transaction_Test_Client (Per_Test_Dat DT_Tdep_PT_Debug (1,(phead,"Client: Starting Client side of test\n")); if (!DT_Transaction_Create_Test (pt_ptr, - ia_handle, + ia, FALSE, port_num, pt_ptr->Server_Info.is_little_endian, @@ -110,7 +110,7 @@ DT_Transaction_Test_Server (void *params + i * cmd->eps_per_thread; if (!DT_Transaction_Create_Test (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, TRUE, port_num, pt_ptr->Client_Info.is_little_endian, @@ -156,7 +156,7 @@ DT_Transaction_Test_Server (void *params boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE *ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -181,7 +181,7 @@ DT_Transaction_Create_Test (Per_Test_Dat test_ptr->remote_is_little_endian = remote_is_little_endian; test_ptr->is_server = is_server; test_ptr->pt_ptr = pt_ptr; - test_ptr->ia_handle = ia_handle; + test_ptr->ia = ia; test_ptr->base_port = (DAT_CONN_QUAL) port_num; test_ptr->cmd = &pt_ptr->Params.u.Transaction_Cmd; test_ptr->time_out = DFLT_TMO * 1000; /* DFLT_TMO seconds */ @@ -244,18 +244,18 @@ DT_Transaction_Main (void *param) private_data_str = "DAPL and RDMA rule! Test 4321."; /* create a protection zone */ - ret = dat_pz_create (test_ptr->ia_handle, &test_ptr->pz_handle); + ret = dat_pz_create (test_ptr->ia, &test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->pz_handle = DAT_HANDLE_NULL; + test_ptr->pz = NULL; status = 1; goto test_failure; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG, @@ -264,12 +264,12 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + test_ptr->recv_evd_hdl = NULL; status = 1; goto test_failure; } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -278,7 +278,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (request) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + test_ptr->reqt_evd_hdl = NULL; status = 1; goto test_failure; } @@ -286,7 +286,7 @@ DT_Transaction_Main (void *param) if (pt_ptr->local_is_server) { /* Client-side doesn't need CR events */ - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CR_FLAG, @@ -295,13 +295,13 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + test_ptr->creq_evd_hdl = NULL; status = 1; goto test_failure; } } - ret = DT_Tdep_evd_create (test_ptr->ia_handle, + ret = DT_Tdep_evd_create (test_ptr->ia, test_ptr->evd_length, NULL, DAT_EVD_CONNECTION_FLAG, @@ -310,7 +310,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + test_ptr->conn_evd_hdl = NULL; status = 1; goto test_failure; } @@ -363,18 +363,18 @@ DT_Transaction_Main (void *param) } /* Create EP */ - ret = dat_ep_create (test_ptr->ia_handle, /* IA */ - test_ptr->pz_handle, /* PZ */ + ret = dat_ep_create (test_ptr->ia, /* IA */ + test_ptr->pz, /* PZ */ test_ptr->recv_evd_hdl, /* recv */ test_ptr->reqt_evd_hdl, /* request */ test_ptr->conn_evd_hdl, /* connect */ &ep_attr, /* EP attrs */ - &test_ptr->ep_context[i].ep_handle); + &test_ptr->ep_context[i].ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - test_ptr->ep_context[i].ep_handle = DAT_HANDLE_NULL; + test_ptr->ep_context[i].ep = NULL; status = 1; goto test_failure; } @@ -385,10 +385,10 @@ DT_Transaction_Main (void *param) */ test_ptr->ep_context[i].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ buff_size, 4, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -423,7 +423,7 @@ DT_Transaction_Main (void *param) * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -433,7 +433,7 @@ DT_Transaction_Main (void *param) goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -457,11 +457,11 @@ DT_Transaction_Main (void *param) * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->creq_evd_hdl, - &test_ptr->ep_context[i].rsp_handle); + &test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create #%d error: %s\n", @@ -472,11 +472,11 @@ DT_Transaction_Main (void *param) } else { - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context[i].ia_port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &test_ptr->ep_context[i].psp_handle); + &test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create #%d error: %s\n", @@ -519,7 +519,7 @@ DT_Transaction_Main (void *param) RemoteMemoryInfo *RemoteMemInfo; struct dat_dto_completion_event_data dto_stat; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; /* * Establish the connection @@ -538,16 +538,16 @@ DT_Transaction_Main (void *param) &cr_stat) || !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].rsp_handle, + test_ptr->ep_context[i].rsp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error:(%x) %s\n", @@ -571,21 +571,21 @@ DT_Transaction_Main (void *param) } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, + ret = dat_cr_accept (cr, NULL, /* NULL for RSP */ 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -594,7 +594,7 @@ DT_Transaction_Main (void *param) goto test_failure; } /* throw away single-use PSP */ - ret = dat_rsp_free (test_ptr->ep_context[i].rsp_handle); + ret = dat_rsp_free (test_ptr->ep_context[i].rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free #%d error: %s\n", @@ -622,16 +622,16 @@ DT_Transaction_Main (void *param) if ( !DT_cr_check ( phead, &cr_stat, - test_ptr->ep_context[i].psp_handle, + test_ptr->ep_context[i].psp, test_ptr->ep_context[i].ia_port, - &cr_handle, + &cr, "Server") ) { status = 1; goto test_failure; } - ret = dat_cr_query (cr_handle, &cr_param); + ret = dat_cr_query (cr, &cr_param); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error: %s\n", @@ -656,33 +656,33 @@ DT_Transaction_Main (void *param) /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context[i].ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context[i].ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); - /* cr_handle consumed on failure */ - (void) dat_psp_free (test_ptr->ep_context[i].psp_handle); + /* cr consumed on failure */ + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { /* error message printed by DT_cr_event_wait */ - (void) dat_psp_free (&test_ptr->ep_context[i].psp_handle); + (void) dat_psp_free (test_ptr->ep_context[i].psp); status = 1; goto test_failure; } /* throw away single-use PSP */ - ret = dat_psp_free (test_ptr->ep_context[i].psp_handle); + ret = dat_psp_free (test_ptr->ep_context[i].psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free #%d error: %s\n", @@ -709,7 +709,7 @@ DT_Transaction_Main (void *param) test_ptr->ep_context[i].ia_port)); retry: - ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_connect (test_ptr->ep_context[i].ep, test_ptr->remote_ia_addr, test_ptr->ep_context[i].ia_port, DAT_TIMEOUT_MAX, @@ -726,7 +726,7 @@ retry: /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -742,7 +742,7 @@ retry: struct dat_event event; int drained = 0; - dat_ep_reset (test_ptr->ep_context[i].ep_handle); + dat_ep_reset (test_ptr->ep_context[i].ep); do { ret = DT_Tdep_evd_dequeue ( test_ptr->recv_evd_hdl, @@ -756,7 +756,7 @@ retry: * Post recv and sync buffers */ if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID, buff_size)) @@ -766,7 +766,7 @@ retry: goto test_failure; } if (!DT_post_recv_buffer ( phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_RECV_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -835,9 +835,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -871,9 +871,9 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, @@ -906,10 +906,10 @@ retry: test_ptr->ep_context[i].op[j].bp = DT_BpoolAlloc (pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context[i].ep_handle, - DAT_HANDLE_NULL, /* rmr */ + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context[i].ep, + NULL, /* rmr */ test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, pt_ptr->provider_attr.optimal_buffer_alignment, @@ -976,7 +976,7 @@ retry: /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, RMI_SEND_BUFFER_ID, buff_size)) @@ -993,7 +993,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Send" @@ -1017,7 +1017,7 @@ retry: if (!DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, buff_size, dto_cookie, test_ptr->is_server ? "Client_Mem_Info_Recv" @@ -1082,9 +1082,9 @@ test_failure: /* Foreach EP */ for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; - ep_handle = DAT_HANDLE_NULL; + ep = NULL; /* Free the per-op buffers */ for (j = 0; j < test_ptr->cmd->num_ops; j++) @@ -1122,9 +1122,9 @@ test_failure: * bailed out mid-setup, or ran to completion * normally, so we use abrupt closure. */ - if (test_ptr->ep_context[i].ep_handle) + if (test_ptr->ep_context[i].ep) { - ret = dat_ep_disconnect (test_ptr->ep_context[i].ep_handle, + ret = dat_ep_disconnect (test_ptr->ep_context[i].ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -1148,7 +1148,7 @@ test_failure: { if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, - &ep_handle)) + &ep)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); @@ -1164,16 +1164,16 @@ test_failure: */ for (j = 0; j < test_ptr->cmd->eps_per_thread; j++) { - if ( test_ptr->ep_context[j].ep_handle == ep_handle ) + if ( test_ptr->ep_context[j].ep == ep ) { - test_ptr->ep_context[j].ep_handle = NULL; + test_ptr->ep_context[j].ep = NULL; } } } } else /* !success - QP may be in error state */ { - ep_handle = test_ptr->ep_context[i].ep_handle; + ep = test_ptr->ep_context[i].ep; } /* @@ -1182,7 +1182,7 @@ test_failure: * disconnected as we are racing with the remote side * disconnects. */ - if ( DAT_HANDLE_NULL != ep_handle) + if ( NULL != ep) { struct dat_event event; /* @@ -1195,7 +1195,7 @@ test_failure: &event); } while (ret == DAT_SUCCESS); /* Destroy the EP */ - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free #%d error: %s\n", @@ -1259,9 +1259,9 @@ test_failure: } /* clean up the PZ */ - if (test_ptr->pz_handle) + if (test_ptr->pz) { - ret = dat_pz_free (test_ptr->pz_handle); + ret = dat_pz_free (test_ptr->pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", @@ -1357,7 +1357,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1432,7 +1432,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { if (!DT_post_send_buffer (phead, - test_ptr->ep_context[i].ep_handle, + test_ptr->ep_context[i].ep, test_ptr->ep_context[i].bp, SYNC_SEND_BUFFER_ID, SYNC_BUFF_SIZE)) @@ -1923,8 +1923,8 @@ DT_Print_Transaction_Test (DT_Tdep_Print test_ptr->remote_is_little_endian); DT_Tdep_PT_Printf (phead, "TransTest.base_port : " F64x "\n", test_ptr->base_port); - DT_Tdep_PT_Printf (phead, "TransTest.pz_handle : %p\n", - test_ptr->pz_handle); + DT_Tdep_PT_Printf (phead, "TransTest.pz : %p\n", + test_ptr->pz); /* statistics */ DT_Tdep_PT_Printf (phead, "TransTest.bytes_send : %d\n", test_ptr->stats.stat_bytes_send); Index: test/dapltest/test/dapl_performance_server.c =================================================================== --- test/dapltest/test/dapl_performance_server.c (revision 2532) +++ test/dapltest/test/dapl_performance_server.c (working copy) @@ -41,7 +41,7 @@ DT_Performance_Test_Server ( DT_Tdep_PT_Debug (1,(phead,"Server: Starting performance test\n")); if ( !DT_Performance_Test_Create (pt_ptr, - pt_ptr->ps_ptr->ia_handle, + pt_ptr->ps_ptr->ia, (struct sockaddr *) 0, TRUE, pt_ptr->Client_Info.is_little_endian, @@ -97,15 +97,15 @@ DT_Performance_Test_Server_Connect ( { u32 ret; boolean_t status; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; struct dat_cr_arrival_event_data cr_stat; - DAT_CR_HANDLE cr_handle; + struct dat_cr * cr; enum dat_event_number event_num; - rsp_handle = DAT_HANDLE_NULL; - psp_handle = DAT_HANDLE_NULL; + rsp = NULL; + psp = NULL; #if 0 /* FIXME */ if (test_ptr->cmd->use_rsp) { @@ -113,11 +113,11 @@ DT_Performance_Test_Server_Connect ( * Server - create a single-use RSP and * await a connection for this EP */ - ret = dat_rsp_create (test_ptr->ia_handle, + ret = dat_rsp_create (test_ptr->ia, test_ptr->ep_context.port, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->creq_evd_hdl, - &rsp_handle); + &rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create error: %s\n", @@ -132,9 +132,9 @@ DT_Performance_Test_Server_Connect ( /* wait for the connection request */ if (!DT_cr_event_wait (test_ptr->conn_evd_hdl, &cr_stat) || !DT_cr_check ( &cr_stat, - DAT_HANDLE_NULL, + NULL, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -142,20 +142,20 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ - if (!DT_conn_event_wait ( test_ptr->ep_context.ep_handle, + if (!DT_conn_event_wait ( test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num)) { @@ -173,17 +173,17 @@ DT_Performance_Test_Server_Connect ( */ status = TRUE; - ret = dat_psp_create (test_ptr->ia_handle, + ret = dat_psp_create (test_ptr->ia, test_ptr->ep_context.port, test_ptr->creq_evd_hdl, DAT_PSP_CONSUMER_FLAG, - &psp_handle); + &psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); status = FALSE; - psp_handle = DAT_HANDLE_NULL; + psp = NULL; return (status); } @@ -204,9 +204,9 @@ DT_Performance_Test_Server_Connect ( if (!DT_cr_event_wait (phead, test_ptr->creq_evd_hdl, &cr_stat) || !DT_cr_check ( phead, &cr_stat, - psp_handle, + psp, test_ptr->ep_context.port, - &cr_handle, + &cr, "Server") ) { status = FALSE; @@ -214,22 +214,22 @@ DT_Performance_Test_Server_Connect ( } /* what, me query? just try to accept the connection */ - ret = dat_cr_accept (cr_handle, - test_ptr->ep_context.ep_handle, + ret = dat_cr_accept (cr, + test_ptr->ep_context.ep, 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - /* cr_handle consumed on failure */ + /* cr consumed on failure */ status = FALSE; goto psp_free; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ if (!DT_conn_event_wait (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->conn_evd_hdl, &event_num ) ) { @@ -241,10 +241,10 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_PT_Debug (1, (phead,"Server[" F64x "]: Accept on port 0x" F64x "\n", test_ptr->base_port, test_ptr->ep_context.port)); psp_free: - if ( DAT_HANDLE_NULL != psp_handle ) + if ( NULL != psp ) { /* throw away single-use PSP */ - ret = dat_psp_free (psp_handle); + ret = dat_psp_free (psp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free error: %s\n", @@ -252,10 +252,10 @@ psp_free: status = FALSE; } } - if ( DAT_HANDLE_NULL != rsp_handle ) + if ( NULL != rsp ) { /* throw away single-use PSP */ - ret = dat_rsp_free (rsp_handle); + ret = dat_rsp_free (rsp); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free error: %s\n", @@ -282,9 +282,9 @@ DT_Performance_Test_Server_Exchange ( test_ptr->ep_context.op.bp = DT_BpoolAlloc (test_ptr->pt_ptr, phead, - test_ptr->ia_handle, - test_ptr->pz_handle, - test_ptr->ep_context.ep_handle, + test_ptr->ia, + test_ptr->pz, + test_ptr->ep_context.ep, test_ptr->reqt_evd_hdl, test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, @@ -340,7 +340,7 @@ DT_Performance_Test_Server_Exchange ( /* post the send buffer */ if (!DT_post_send_buffer (phead, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID, DT_PERF_SYNC_BUFF_SIZE)) @@ -358,7 +358,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Send Sync_Msg") ) @@ -380,7 +380,7 @@ DT_Performance_Test_Server_Exchange ( if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - test_ptr->ep_context.ep_handle, + test_ptr->ep_context.ep, DT_PERF_SYNC_BUFF_SIZE, dto_cookie, "Recieve Sync_Msg") ) Index: test/dapltest/test/dapl_fft_dataxfer_client.c =================================================================== --- test/dapltest/test/dapl_fft_dataxfer_client.c (revision 2532) +++ test/dapltest/test/dapl_fft_dataxfer_client.c (working copy) @@ -40,7 +40,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ u32 rc=0; DT_fft_init_client (phead, cmd, &conn); - DT_assert_dat (phead, conn.ia_handle != NULL) + DT_assert_dat (phead, conn.ia != NULL) DT_assert (phead, DT_fft_connect (phead, &conn)); @@ -58,8 +58,8 @@ int DT_dataxfer_client_generic (DT_Tdep_ { conn.bpool = DT_BpoolAlloc (0, phead, - conn.ia_handle, - conn.pz_handle, + conn.ia, + conn.pz, NULL, NULL, 4096, @@ -68,7 +68,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ FALSE, FALSE); DT_assert (phead, conn.bpool != 0); - rc = DT_post_send_buffer (phead, conn.ep_handle, conn.bpool, 0, + rc = DT_post_send_buffer (phead, conn.ep, conn.bpool, 0, DT_Bpool_GetBuffSize (conn.bpool, 0)); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = dat_evd_wait (conn.send_evd, 10*1000000, 1, &conn.event, @@ -80,11 +80,11 @@ int DT_dataxfer_client_generic (DT_Tdep_ /* cleanup */ cleanup: - if (conn.ep_handle) + if (conn.ep) { /* disconnect */ DT_Tdep_PT_Printf (phead, "Disconnect\n"); - rc = dat_ep_disconnect (conn.ep_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ep_disconnect (conn.ep, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } rc = DT_fft_destroy_conn_struct (phead, &conn); Index: test/dapltest/test/dapl_fft_connmgt.c =================================================================== --- test/dapltest/test/dapl_fft_connmgt.c (revision 2532) +++ test/dapltest/test/dapl_fft_connmgt.c (working copy) @@ -39,7 +39,7 @@ int DT_connmgt_case0 (Params_t *params_p Description: Ensure time in dat_evd_wait works correctly\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 10000, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); @@ -62,7 +62,7 @@ int DT_connmgt_case1 (Params_t *params_p Description: Attempt to use timeout of 0 in dat_evd_wait\n"); DT_fft_init_server (params_ptr, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); rc = DT_Tdep_evd_wait (conn.cr_evd, 0, &conn.event); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_TIMEOUT_EXPIRED); Index: test/dapltest/test/dapl_bpool.c =================================================================== --- test/dapltest/test/dapl_bpool.c (revision 2532) +++ test/dapltest/test/dapl_bpool.c (working copy) @@ -86,10 +86,10 @@ Bpool * DT_BpoolAlloc ( Per_Test_Data_t *pt_ptr, DT_Tdep_Print_Head *phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia * ia, + struct dat_pz * pz, + struct dat_ep * ep, + struct dat_evd * rmr_evd, int seg_size, int num_segs, int alignment, @@ -131,16 +131,16 @@ DT_BpoolAlloc ( bpool_ptr->alloc_ptr = alloc_ptr; bpool_ptr->alloc_size = alloc_size; - bpool_ptr->pz_handle = pz_handle; + bpool_ptr->pz = pz; bpool_ptr->num_segs = num_segs; - bpool_ptr->ep_handle = ep_handle; + bpool_ptr->ep = ep; bpool_ptr->buffer_size = seg_size * num_segs; bpool_ptr->buffer_start = DT_AlignPtr (alloc_ptr, alignment) + GG_ALLOC_OFFSET; bpool_ptr->tripl_start = (struct dat_lmr_triplet *) (bpool_ptr + 1); bpool_ptr->seg_size = seg_size; bpool_ptr->enable_rdma_write = enable_rdma_write; bpool_ptr->enable_rdma_read = enable_rdma_read; - bpool_ptr->rmr_evd_handle = rmr_evd_handle; + bpool_ptr->rmr_evd = rmr_evd; DT_Tdep_PT_Debug (3, (phead, "lmr_create [%p, " F64x "]\n", @@ -160,14 +160,14 @@ DT_BpoolAlloc ( region.for_pa = virt_to_phys(region.for_va); } - ret = dat_lmr_kcreate (ia_handle, + ret = dat_lmr_kcreate (ia, DT_mem_type, region, bp_len, - pz_handle, + pz, DAT_MEM_PRIV_ALL_FLAG, DAT_MEM_OPTIMIZE_DONT_CARE, - &bpool_ptr->lmr_handle, + &bpool_ptr->lmr, &bpool_ptr->lmr_context, &bpool_ptr->rmr_context, &bpool_ptr->reg_size, @@ -213,7 +213,7 @@ DT_BpoolAlloc ( struct dat_rmr_bind_completion_event_data rmr_stat; /* create the RMR */ - ret = dat_rmr_create (pz_handle, &bpool_ptr->rmr_handle); + ret = dat_rmr_create (pz, &bpool_ptr->rmr_handle); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_rmr_create failed %s\n", @@ -238,7 +238,7 @@ DT_BpoolAlloc ( ret = dat_rmr_bind ( bpool_ptr->rmr_handle, &iov, mflags, - bpool_ptr->ep_handle, + bpool_ptr->ep, cookie, DAT_COMPLETION_DEFAULT_FLAG, &bpool_ptr->rmr_context); @@ -253,7 +253,7 @@ DT_BpoolAlloc ( /* await the bind result */ if (!DT_rmr_event_wait (phead, - bpool_ptr->rmr_evd_handle, + bpool_ptr->rmr_evd, &rmr_stat) || !DT_rmr_check (phead, &rmr_stat, @@ -290,9 +290,9 @@ err: DT_RetToString (ret)); } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - ret = dat_lmr_free (bpool_ptr->lmr_handle); + ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -357,9 +357,9 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p } } - if (bpool_ptr->lmr_handle) + if (bpool_ptr->lmr) { - u32 ret = dat_lmr_free (bpool_ptr->lmr_handle); + u32 ret = dat_lmr_free (bpool_ptr->lmr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -433,8 +433,8 @@ DT_Bpool_print (DT_Tdep_Print_Head *phea "BPOOL alloc_size %x\n", (int) bpool_ptr->alloc_size); DT_Tdep_PT_Printf (phead, - "BPOOL pz_handle %p\n", - bpool_ptr->pz_handle); + "BPOOL pz %p\n", + bpool_ptr->pz); DT_Tdep_PT_Printf (phead, "BPOOL num_segs %x\n", (int) bpool_ptr->num_segs); Index: test/dapltest/test/dapl_test_util.c =================================================================== --- test/dapltest/test/dapl_test_util.c (revision 2532) +++ test/dapltest/test/dapl_test_util.c (working copy) @@ -32,11 +32,11 @@ */ boolean_t DT_query ( Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle) + struct dat_ia * ia, + struct dat_ep * ep) { unsigned char *module = "DT_query"; - DAT_EVD_HANDLE async_evd_hdl; /* not used */ + struct dat_evd * async_evd_hdl; /* not used */ struct dat_ep_param ep_params; u32 ret; DT_Tdep_Print_Head *phead; @@ -44,7 +44,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, phead = pt_ptr->Params.phead; /* Query the IA */ - ret = dat_ia_query (ia_handle, + ret = dat_ia_query (ia, &async_evd_hdl, &pt_ptr->ia_attr, &pt_ptr->provider_attr); @@ -57,7 +57,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, } /* Query the EP */ - ret = dat_ep_query (ep_handle, &ep_params); + ret = dat_ep_query (ep, &ep_params); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_query error: %s\n", @@ -160,7 +160,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, */ boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -184,7 +184,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Recv #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_recv (ep_handle, + ret = dat_ep_post_recv (ep, 1, iov, cookie, @@ -205,7 +205,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head */ boolean_t DT_post_send_buffer (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size) @@ -228,7 +228,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Debug (3, (phead, "Post-Send #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ - ret = dat_ep_post_send (ep_handle, + ret = dat_ep_post_send (ep, 1, iov, cookie, @@ -249,7 +249,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head */ boolean_t DT_cr_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p) { int err_cnt; @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -306,8 +306,8 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p */ boolean_t DT_conn_event_wait (DT_Tdep_Print_Head *phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep * ep, + struct dat_evd *evd, enum dat_event_number *event_number) { @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -341,7 +341,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * { /* * Could return struct dat_connection_event_data and verify: - * event.event_data.connect_event_data.ep_handle + * event.event_data.connect_event_data.ep * event.event_data.connect_event_data.private_data_size * event.event_data.connect_event_data.private_data */ @@ -361,15 +361,15 @@ DT_conn_event_wait (DT_Tdep_Print_Head * */ boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ) + struct dat_evd *evd, + struct dat_ep * *ep ) { for (;;) { u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -392,9 +392,9 @@ DT_disco_event_wait ( DT_Tdep_Print_Head if (event.event_number == DAT_CONNECTION_EVENT_DISCONNECTED) { - if ( ep_handle != NULL ) + if ( ep != NULL ) { - *ep_handle = event.event_data.connect_event_data.ep_handle; + *ep = event.event_data.connect_event_data.ep; } return (TRUE); } @@ -412,17 +412,17 @@ DT_disco_event_wait ( DT_Tdep_Print_Head */ boolean_t DT_dto_event_reap (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dto_statusp) { if (poll) { - return DT_dto_event_poll (phead, evd_handle, dto_statusp); + return DT_dto_event_poll (phead, evd, dto_statusp); } else { - return DT_dto_event_wait (phead, evd_handle, dto_statusp); + return DT_dto_event_wait (phead, evd, dto_statusp); } } @@ -432,7 +432,7 @@ DT_dto_event_reap (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_poll (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -440,7 +440,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_dequeue ( evd_handle, + ret = DT_Tdep_evd_dequeue ( evd, &event); if (DAT_GET_TYPE (ret) == DAT_QUEUE_EMPTY) @@ -460,7 +460,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -484,7 +484,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p */ boolean_t DT_dto_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dto_statusp) { for (;;) @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -505,7 +505,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p { /* * Pass back all the useful bits if requested: - * ep_handle, user_cookie.as_ptr + * ep, user_cookie.as_ptr * status, transfered_length */ if (dto_statusp) @@ -528,7 +528,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p */ boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head *phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_statusp) { for (;;) @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_MAX, &event); + ret = DT_Tdep_evd_wait (evd, DAT_TIMEOUT_MAX, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", @@ -572,12 +572,12 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p boolean_t DT_dto_check ( DT_Tdep_Print_Head *phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message ) { - if ( ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) || dto_p->transfered_length != len_expected || dto_p->user_cookie.as_64 != cookie_expected.as_64 || dto_p->status != DAT_DTO_SUCCESS ) @@ -588,10 +588,10 @@ DT_dto_check ( DT_Tdep_Print_Head *phead : (dto_p->status == DAT_DTO_ERR_FLUSHED ? "FAILURE" : "LengthError"))); DT_Test_Error (); - if ( (ep_expected != NULL) && (dto_p->ep_handle != ep_expected) ) + if ( (ep_expected != NULL) && (dto_p->ep != ep_expected) ) { DT_Tdep_PT_Printf (phead, "\tEndPoint mismatch (got %p wanted %p)\n", - dto_p->ep_handle, + dto_p->ep, ep_expected); } if (dto_p->transfered_length != len_expected) @@ -620,11 +620,11 @@ DT_dto_check ( DT_Tdep_Print_Head *phead boolean_t DT_rmr_check ( DT_Tdep_Print_Head *phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message) { - if ( rmr_p->rmr_handle != rmr_expected + if ( rmr_p->rmr != rmr_expected || rmr_p->user_cookie.as_ptr != cookie_expected || rmr_p->status != DAT_RMR_BIND_SUCCESS ) { @@ -634,10 +634,10 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead (rmr_p->status == DAT_RMR_BIND_SUCCESS ? "OK" : "FAILURE")); DT_Test_Error (); - if (rmr_p->rmr_handle != rmr_expected) + if (rmr_p->rmr != rmr_expected) { DT_Tdep_PT_Printf (phead, "\tRMR handle mismatch (got 0x%p wanted 0x%p)\n", - rmr_p->rmr_handle, + rmr_p->rmr, rmr_expected); } if (rmr_p->user_cookie.as_ptr != cookie_expected) @@ -659,21 +659,21 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead boolean_t DT_cr_check ( DT_Tdep_Print_Head *phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp * psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr * *crp, char *message) { u32 ret; - if (cr_handlep) + if (crp) { - *cr_handlep = (DAT_CR_HANDLE) 0; + *crp = (struct dat_cr *) 0; } if (cr_stat_p->conn_qual != port_expected || - (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected)) + (psp_expected && + cr_stat_p->sp != psp_expected)) { DT_Tdep_PT_Printf (phead, "Test Error: %s CR data problem\n", message); @@ -684,20 +684,20 @@ DT_cr_check ( DT_Tdep_Print_Head *phead " (got 0x" F64x " wanted 0x" F64x ")\n", cr_stat_p->conn_qual, port_expected); } - if (psp_handle_expected && - cr_stat_p->sp_handle.psp_handle != psp_handle_expected) + if (psp_expected && + cr_stat_p->sp != psp_expected) { DT_Tdep_PT_Printf (phead, "\tPSP mismatch (got 0x%p wanted 0x%p)\n", - cr_stat_p->sp_handle.psp_handle, - psp_handle_expected); + cr_stat_p->sp, + psp_expected); } - if (!cr_stat_p->cr_handle) + if (!cr_stat_p->cr) { - DT_Tdep_PT_Printf (phead, "\tGot NULL cr_handle\n"); + DT_Tdep_PT_Printf (phead, "\tGot NULL cr\n"); } else { - ret = dat_cr_reject (cr_stat_p->cr_handle); + ret = dat_cr_reject (cr_stat_p->cr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "\tdat_cr_reject error: %s\n", @@ -707,9 +707,9 @@ DT_cr_check ( DT_Tdep_Print_Head *phead return ( FALSE ); } - if (cr_handlep) + if (crp) { - *cr_handlep = cr_stat_p->cr_handle; + *crp = cr_stat_p->cr; } return ( TRUE ); } Index: test/dapltest/test/dapl_client.c =================================================================== --- test/dapltest/test/dapl_client.c (revision 2532) +++ test/dapltest/test/dapl_client.c (working copy) @@ -41,13 +41,13 @@ DT_cs_Client (Params_t * params_ptr, u32 total_threads) { Per_Test_Data_t *pt_ptr = NULL; - DAT_IA_HANDLE ia_handle = DAT_HANDLE_NULL; - DAT_PZ_HANDLE pz_handle = DAT_HANDLE_NULL; - DAT_EVD_HANDLE recv_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE reqt_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE conn_evd_hdl = DAT_HANDLE_NULL; - DAT_EVD_HANDLE async_evd_hdl = DAT_HANDLE_NULL; - DAT_EP_HANDLE ep_handle = DAT_HANDLE_NULL; + struct dat_ia *ia = NULL; + struct dat_pz *pz = NULL; + struct dat_evd *recv_evd_hdl = NULL; + struct dat_evd *reqt_evd_hdl = NULL; + struct dat_evd *conn_evd_hdl = NULL; + struct dat_evd *async_evd_hdl = NULL; + struct dat_ep *ep = NULL; Server_Info_t *sinfo = NULL; Transaction_Cmd_t *Transaction_Cmd = NULL; Quit_Cmd_t *Quit_Cmd = NULL; @@ -98,7 +98,7 @@ DT_cs_Client (Params_t * params_ptr, ret = dat_ia_open (dapl_name, DFLT_QLEN, &async_evd_hdl, - &ia_handle); + &ia); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -106,28 +106,28 @@ DT_cs_Client (Params_t * params_ptr, module, dapl_name, DT_RetToString (ret)); - ia_handle = DAT_HANDLE_NULL; + ia = NULL; status = 1; goto client_exit; } DT_Tdep_PT_Debug (1,(phead, "%s: IA %s opened\n", module, dapl_name)); /* Create a PZ */ - ret = dat_pz_create (ia_handle, &pz_handle); + ret = dat_pz_create (ia, &pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create error: %s\n", module, DT_RetToString (ret)); - pz_handle = DAT_HANDLE_NULL; + pz = NULL; status = 1; goto client_exit; } /* Create 3 events - recv, request, connect */ - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG, @@ -138,11 +138,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); - recv_evd_hdl = DAT_HANDLE_NULL; + recv_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, @@ -153,11 +153,11 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); - reqt_evd_hdl = DAT_HANDLE_NULL; + reqt_evd_hdl = NULL; status = 1; goto client_exit; } - ret = DT_Tdep_evd_create (ia_handle, + ret = DT_Tdep_evd_create (ia, DFLT_QLEN, NULL, DAT_EVD_CONNECTION_FLAG, @@ -168,26 +168,26 @@ DT_cs_Client (Params_t * params_ptr, "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); - conn_evd_hdl = DAT_HANDLE_NULL; + conn_evd_hdl = NULL; status = 1; goto client_exit; } /* Create an EP */ - ret = dat_ep_create (ia_handle, /* IA */ - pz_handle, /* PZ */ + ret = dat_ep_create (ia, /* IA */ + pz, /* PZ */ recv_evd_hdl, /* recv */ reqt_evd_hdl, /* request */ conn_evd_hdl, /* connect */ (struct dat_ep_attr *) NULL, - &ep_handle); + &ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create error: %s\n", module, DT_RetToString (ret)); - ep_handle = DAT_HANDLE_NULL; + ep = NULL; status = 1; goto client_exit; } @@ -197,7 +197,7 @@ DT_cs_Client (Params_t * params_ptr, * Gather whatever info we want about defaults, * and check that we can handle the requested parameters. */ - if (!DT_query (pt_ptr, ia_handle, ep_handle) || + if (!DT_query (pt_ptr, ia, ep) || !DT_check_params (pt_ptr, module)) { status = 1; @@ -206,10 +206,10 @@ DT_cs_Client (Params_t * params_ptr, bpool = DT_BpoolAlloc (pt_ptr, phead, - ia_handle, - pz_handle, - ep_handle, - DAT_HANDLE_NULL, /* no RMR */ + ia, + pz, + ep, + NULL, /* no RMR */ DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ pt_ptr->provider_attr.optimal_buffer_alignment, @@ -238,7 +238,7 @@ DT_cs_Client (Params_t * params_ptr, DT_Tdep_PT_Debug (1,(phead, "%s: Posting 1 recv buffer\n", module)); retry_repost: if (!DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, DT_Bpool_GetBuffSize (bpool, 0))) @@ -253,7 +253,7 @@ retry_repost: DT_Tdep_PT_Debug (1,(phead, "%s: Connect Endpoint\n", module)); try_connect =1; retry: - ret = dat_ep_connect (ep_handle, + ret = dat_ep_connect (ep, server_netaddr, SERVER_PORT_NUMBER, DAT_TIMEOUT_MAX, @@ -271,7 +271,7 @@ retry: } DT_Tdep_PT_Debug (1,(phead, "%s: Await connection ...\n", module)); - if (!DT_conn_event_wait (phead, ep_handle, conn_evd_hdl, &event_num)) + if (!DT_conn_event_wait (phead, ep, conn_evd_hdl, &event_num)) { if ( event_num == DAT_CONNECTION_EVENT_PEER_REJECTED ) { @@ -287,7 +287,7 @@ retry: * See if any buffers were flushed as a result of * the REJECT; clean them up and repost if so */ - dat_ep_reset (ep_handle); + dat_ep_reset (ep); do { @@ -315,7 +315,7 @@ retry: if (DT_dapltest_debug) { DT_Tdep_PT_Debug (1,(phead, "%s: Connected!\n", module)); - get_ep_connection_state (phead, ep_handle); + get_ep_connection_state (phead, ep); } /* Send Client_Info (using 2nd buffer in the pool) */ @@ -326,7 +326,7 @@ retry: sizeof (Client_Info_t)); DT_Client_Info_Endian ((Client_Info_t *) buffp); if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 1, DT_Bpool_GetBuffSize (bpool, 1))) @@ -343,7 +343,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 1), dto_cookie, "Client_Info_Send")) @@ -396,7 +396,7 @@ retry: /* Send the Command buffer */ if (!DT_post_send_buffer ( phead, - ep_handle, + ep, bpool, 2, DT_Bpool_GetBuffSize (bpool, 2))) @@ -414,7 +414,7 @@ retry: if (!DT_dto_event_wait (phead, reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 2), dto_cookie, "Client_Cmd_Send")) @@ -430,7 +430,7 @@ retry: if (!DT_dto_event_wait (phead, recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, - ep_handle, + ep, DT_Bpool_GetBuffSize (bpool, 0), dto_cookie, "Server_Info_Recv")) @@ -476,7 +476,7 @@ retry: DT_Transaction_Cmd_PT_Print (phead, Transaction_Cmd); } status = DT_Transaction_Test_Client (pt_ptr, - ia_handle, + ia, server_netaddr); params_ptr->Client_Stats_T.bytes_rdma_read = pt_ptr->Client_Stats.bytes_rdma_read; @@ -508,7 +508,7 @@ retry: status = DT_Performance_Test_Client (params_ptr, pt_ptr, - ia_handle, + (struct dat_ia *)ia, server_netaddr); break; } @@ -521,13 +521,13 @@ client_exit: DT_Tdep_PT_Debug (1,(phead, "%s: Cleaning Up ...\n", module)); /* Disconnect the EP */ - if (ep_handle && try_connect) + if (ep && try_connect) { /* * graceful attempt might fail because we got here due to * some error above, so we may as well try harder. */ - ret = dat_ep_disconnect (ep_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ep_disconnect (ep, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -547,7 +547,7 @@ client_exit: DT_Bpool_Destroy (pt_ptr, phead, bpool); /* Free the EP */ - if (ep_handle) + if (ep) { struct dat_event event; /* @@ -560,7 +560,7 @@ client_exit: &event); } while (ret == DAT_SUCCESS); - ret = dat_ep_free (ep_handle); + ret = dat_ep_free (ep); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -614,9 +614,9 @@ client_exit: } /* Free the PZ */ - if (pz_handle) + if (pz) { - ret = dat_pz_free (pz_handle); + ret = dat_pz_free (pz); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -629,10 +629,10 @@ client_exit: } /* Close the IA */ - if (ia_handle) + if (ia) { /* dat_ia_close cleans up async evd handle, too */ - ret = dat_ia_close (ia_handle, DAT_CLOSE_GRACEFUL_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -640,7 +640,7 @@ client_exit: module, DT_RetToString (ret)); status = 1; - ret = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + ret = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, Index: test/dapltest/test/dapl_fft_endpoint.c =================================================================== --- test/dapltest/test/dapl_fft_endpoint.c (revision 2532) +++ test/dapltest/test/dapl_fft_endpoint.c (working copy) @@ -36,59 +36,59 @@ int DT_endpoint_generic (Params_t *param boolean_t destroy_pz_early) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE evd_handle; - DAT_EVD_HANDLE conn_evd_handle; - DAT_EVD_HANDLE send_evd_handle; - DAT_EVD_HANDLE recv_evd_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *evd; + struct dat_evd *conn_evd; + struct dat_evd *send_evd; + struct dat_evd *recv_evd; u32 rc, wanted; int res; DT_Tdep_Print_Head *phead; res = 1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; - evd_handle = NULL; - conn_evd_handle = NULL; - send_evd_handle = NULL; - recv_evd_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; + evd = NULL; + conn_evd = NULL; + send_evd = NULL; + recv_evd = NULL; dev_name = cmd->device_name; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; phead = params_ptr->phead; - rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd_handle, &ia_handle); + rc = dat_ia_open (dev_name, DEFAULT_QUEUE_LEN, &evd, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); if (destroy_pz_early) { - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } } - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG, - &send_evd_handle); + &send_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, - &recv_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, DAT_EVD_DTO_FLAG, + &recv_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, NULL, - DAT_EVD_CONNECTION_FLAG, &conn_evd_handle); + rc = DT_Tdep_evd_create (ia, DEFAULT_QUEUE_LEN, NULL, + DAT_EVD_CONNECTION_FLAG, &conn_evd); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_ep_create (ia_handle, pz_handle, recv_evd_handle, send_evd_handle, - conn_evd_handle, NULL, &ep_handle); + rc = dat_ep_create (ia, pz, recv_evd, send_evd, + conn_evd, NULL, &ep); if (destroy_pz_early) { wanted = DAT_INVALID_HANDLE; @@ -100,39 +100,39 @@ int DT_endpoint_generic (Params_t *param DT_assert_dat (phead, DAT_GET_TYPE (rc) == wanted); cleanup: - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (send_evd_handle) + if (send_evd) { - rc = DT_Tdep_evd_free (send_evd_handle); + rc = DT_Tdep_evd_free (send_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (recv_evd_handle) + if (recv_evd) { - rc = DT_Tdep_evd_free (recv_evd_handle); + rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (conn_evd_handle) + if (conn_evd) { - rc = DT_Tdep_evd_free (conn_evd_handle); + rc = DT_Tdep_evd_free (conn_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (!destroy_pz_early && pz_handle) + if (!destroy_pz_early && pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; @@ -163,10 +163,10 @@ int DT_endpoint_case1 (Params_t *params_ int DT_endpoint_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char *dev_name; - DAT_IA_HANDLE ia_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE send_evd, conn_evd, recv_evd, cr_evd; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_ep *ep; + struct dat_evd *send_evd, *conn_evd, *recv_evd, *cr_evd; + struct dat_pz *pz; struct dat_event event; Bpool *bpool; int res; @@ -177,39 +177,39 @@ int DT_endpoint_case2 (Params_t *params_ Description: try to destroy ep with descriptor still in working queue\n"); res = 1; bpool = NULL; - pz_handle = NULL; - ia_handle = NULL; - ep_handle = NULL; + pz = NULL; + ia = NULL; + ep = NULL; send_evd = NULL; conn_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name = cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, 4096, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, 4096, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); DT_assert (phead, DT_post_recv_buffer (phead, - ep_handle, + ep, bpool, 0, 4096) == TRUE); - if (ep_handle) + if (ep) { - rc = dat_ep_free (ep_handle); + rc = dat_ep_free (ep); DT_assert_dat (phead, rc == DAT_SUCCESS); } @@ -229,14 +229,14 @@ cleanup: rc = DT_Bpool_Destroy (NULL, phead, bpool); DT_assert_clean (phead, rc != FALSE); } - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } return res; Index: test/dapltest/test/dapl_transaction_util.c =================================================================== --- test/dapltest/test/dapl_transaction_util.c (revision 2532) +++ test/dapltest/test/dapl_transaction_util.c (working copy) @@ -59,7 +59,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the recv */ - ret = dat_ep_post_recv ( ep_context[i].ep_handle, + ret = dat_ep_post_recv ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -85,7 +85,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H boolean_t DT_handle_send_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll) @@ -122,7 +122,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the send */ - ret = dat_ep_post_send ( ep_context[i].ep_handle, + ret = dat_ep_post_send ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -173,7 +173,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -188,7 +188,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Send")) @@ -201,7 +201,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -215,7 +215,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -235,8 +235,8 @@ DT_handle_send_op (DT_Tdep_Print_Head *p boolean_t DT_handle_recv_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -283,7 +283,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Receive: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -298,7 +298,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, "Recv")) @@ -315,7 +315,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -353,7 +353,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: "F64u "\n", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); @@ -374,11 +374,11 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p /* * If we have multiple EPs we can't guarantee the order of - * completions, so disable ep_handle check + * completions, so disable ep check */ if (!DT_dto_check (phead, &dto_stat, - num_eps == 1?ep_context[i].ep_handle: NULL, + num_eps == 1?ep_context[i].ep: NULL, op->num_segs * op->seg_size, dto_cookie, "Send-reaped-on-recv")) @@ -395,7 +395,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): Secondary completion seen for endpoint 0x%p (%d)\n", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -411,7 +411,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Receive: No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -428,7 +428,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: Send (ror): No completion seen for endpoint 0x%p (#%d)\n", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); @@ -463,7 +463,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -516,7 +516,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (opcode == RDMA_WRITE) { - ret = dat_ep_post_rdma_write (ep_context[i].ep_handle, + ret = dat_ep_post_rdma_write (ep_context[i].ep, op->num_segs, iov, cookie, @@ -527,7 +527,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p else /* opcode == RDMA_READ */ { - ret = dat_ep_post_rdma_read ( ep_context[i].ep_handle, + ret = dat_ep_post_rdma_read ( ep_context[i].ep, op->num_segs, iov, cookie, @@ -574,7 +574,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p DT_Tdep_PT_Printf (phead, "Test Error: %s: Invalid endpoint completion reaped.\n" "\tEndpoint: 0x%p, Cookie: 0x" F64x ", Length: " F64u "\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - dto_stat.ep_handle, dto_stat.user_cookie.as_64, + dto_stat.ep, dto_stat.user_cookie.as_64, dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); @@ -588,7 +588,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (!DT_dto_check (phead, &dto_stat, - ep_context[epnum].ep_handle, + ep_context[epnum].ep, op->num_segs * op->seg_size, dto_cookie, (opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD"))) @@ -601,7 +601,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: Secondary completion seen for endpoint 0x%p (%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[epnum].ep_handle, epnum); + ep_context[epnum].ep, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); @@ -619,7 +619,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p { DT_Tdep_PT_Printf (phead, "Test Error: %s: No completion seen for endpoint 0x%p (#%d)\n", opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD", - ep_context[i].ep_handle, i); + ep_context[i].ep, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); return ( FALSE ); Index: test/dapltest/test/dapl_fft_pz.c =================================================================== --- test/dapltest/test/dapl_fft_pz.c (revision 2532) +++ test/dapltest/test/dapl_fft_pz.c (working copy) @@ -35,9 +35,9 @@ int DT_pz_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_evd *evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -47,25 +47,25 @@ int DT_pz_case0 ( Params_t *params_ptr, Description: Test if we can normally create pz and destroy it.\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + ia = NULL; + pz = NULL; + evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); cleanup: - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc = dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); } return res; @@ -75,10 +75,10 @@ cleanup: int DT_pz_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE conn_evd, send_evd, recv_evd, cr_evd; + struct dat_ia * ia; + struct dat_pz *pz; + struct dat_ep *ep; + struct dat_evd *conn_evd, *send_evd, *recv_evd, *cr_evd; u32 rc; int res; DT_Tdep_Print_Head *phead; @@ -88,42 +88,42 @@ int DT_pz_case1 (Params_t *params_ptr, F Description: try to destroy pz with vi still associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; - ep_handle = NULL; + ia = NULL; + pz = NULL; + ep = NULL; conn_evd = NULL; send_evd = NULL; recv_evd = NULL; cr_evd = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); rc = DT_ep_create (params_ptr, - ia_handle, - pz_handle, + ia, + pz, &cr_evd, &conn_evd, &send_evd, &recv_evd, - &ep_handle); + &ep); DT_assert_dat (phead, rc == DAT_SUCCESS); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } cleanup: /* corrrect order */ - if (ep_handle) + if (ep) { - rc=dat_ep_free (ep_handle); + rc=dat_ep_free (ep); DT_assert_clean (phead, rc == DAT_SUCCESS); } if (conn_evd) @@ -141,15 +141,15 @@ cleanup: rc = DT_Tdep_evd_free (recv_evd); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } @@ -160,8 +160,8 @@ cleanup: int DT_pz_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz *pz; Bpool *bpool; u32 rc; int res; @@ -174,26 +174,26 @@ int DT_pz_case2 (Params_t *params_ptr, F associated with it\n"); res=1; - ia_handle = NULL; - pz_handle = NULL; + ia = NULL; + pz = NULL; bpool = NULL; dev_name= cmd->device_name; - rc = DT_ia_open (dev_name, &ia_handle); + rc = DT_ia_open (dev_name, &ia); DT_assert_dat (phead, rc == DAT_SUCCESS); - rc = dat_pz_create (ia_handle, &pz_handle); + rc = dat_pz_create (ia, &pz); DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate and register bpool */ - bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, + bpool = DT_BpoolAlloc (NULL, phead, ia, pz, NULL, NULL, BUFFSIZE, 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); - if (pz_handle) + if (pz) { - rc = dat_pz_free (pz_handle); + rc = dat_pz_free (pz); DT_assert_dat (phead, DAT_GET_TYPE (rc) == DAT_INVALID_STATE); } @@ -205,15 +205,15 @@ cleanup: DT_Tdep_PT_Printf (phead, "Warning: Destroy bpool fails, reboot for cleanup\n"); return 0; } - if (pz_handle) + if (pz) { - rc=dat_pz_free (pz_handle); + rc=dat_pz_free (pz); DT_assert_clean (phead, rc == DAT_SUCCESS); } - if (ia_handle) + if (ia) { - rc=dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc=dat_ia_close (ia, DAT_CLOSE_ABRUPT_FLAG); DT_assert_clean (phead, rc == DAT_SUCCESS); } Index: test/dapltest/test/dapl_fft_hwconn.c =================================================================== --- test/dapltest/test/dapl_fft_hwconn.c (revision 2532) +++ test/dapltest/test/dapl_fft_hwconn.c (working copy) @@ -31,8 +31,8 @@ int DT_hwconn_case0 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { char* dev_name; - DAT_IA_HANDLE nic_handle; - DAT_EVD_HANDLE evd_handle; + struct dat_ia * nic_handle; + struct dat_evd *evd; u32 rc; int res = 1; DT_Tdep_Print_Head *phead; @@ -43,9 +43,9 @@ int DT_hwconn_case0 ( Params_t *params_p dev_name= cmd->device_name; nic_handle = NULL; - evd_handle = DAT_HANDLE_NULL; + evd = NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); @@ -58,9 +58,9 @@ cleanup: /*--------------------------------------------------------*/ int DT_hwconn_case1 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; char dev_name[100]; int i; DT_Tdep_Print_Head *phead; @@ -94,8 +94,8 @@ int DT_hwconn_case1 ( Params_t *params_p sprintf (dev_name, "%s", "34df"); /* number_letter */ } - evd_handle = DAT_HANDLE_NULL; - rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); + evd = NULL; + rc=dat_ia_open ((const char *)dev_name, 10, &evd, &nic_handle); if (DAT_GET_TYPE (rc) != DAT_PROVIDER_NOT_FOUND) { const char *major_msg, *minor_msg; @@ -120,7 +120,7 @@ int DT_hwconn_case1 ( Params_t *params_p /*--------------------------------------------------------*/ int DT_hwconn_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { - DAT_IA_HANDLE nic_handle; + struct dat_ia * nic_handle; u32 rc; int res=1; DT_Tdep_Print_Head *phead; @@ -156,9 +156,9 @@ int DT_hwconn_case3 (Params_t *params_pt DT_fft_init_client (params_ptr, cmd, &conn); /* try to close nic when vi have not destroyed */ - if (conn.ia_handle) + if (conn.ia) { - rc= dat_ia_close (conn.ia_handle, DAT_CLOSE_ABRUPT_FLAG); + rc= dat_ia_close (conn.ia, DAT_CLOSE_ABRUPT_FLAG); if (rc !=DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Warning: dat_ia_close fails %s, reboot for cleanup\n", Index: test/dapltest/test/dapl_fft_dataxfer.c =================================================================== --- test/dapltest/test/dapl_fft_dataxfer.c (revision 2532) +++ test/dapltest/test/dapl_fft_dataxfer.c (working copy) @@ -34,7 +34,7 @@ int DT_dataxfer_generic ( DT_Tdep_Print_ u32 rc=0; int res=1; DT_fft_init_server (phead, cmd, &conn); - DT_assert (phead, NULL != conn.ia_handle); + DT_assert (phead, NULL != conn.ia); DT_fft_listen (phead, &conn); @@ -81,7 +81,7 @@ cleanup: int DT_dataxfer_case0 ( DT_Tdep_Print_Head *phead, FFT_Cmd_t *cmd) { DT_Tdep_PT_Printf (phead, "\ - Description: Call dat_ep_post_send with null ep_handle.\n"); + Description: Call dat_ep_post_send with null ep.\n"); return DT_dataxfer_generic (phead, cmd, 0); } Index: test/dapltest/include/dapl_tdep.h =================================================================== --- test/dapltest/include/dapl_tdep.h (revision 2532) +++ test/dapltest/include/dapl_tdep.h (working copy) @@ -31,7 +31,7 @@ #include "dapl_proto.h" #ifdef __KDAPL__ -typedef DAT_HANDLE DAT_CNO_HANDLE; +typedef void * DAT_CNO_HANDLE; #endif /* function prototypes */ @@ -45,21 +45,21 @@ int DT_Tdep_Execute_Test ( Params_t *params_ptr ) ; u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr); + struct dat_evd * *evd_ptr); u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle); +DT_Tdep_evd_free (struct dat_evd * evd); u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd * evd, unsigned long timeout, struct dat_event *event); u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd * evd, struct dat_event *event); #endif Index: test/dapltest/include/dapl_performance_test.h =================================================================== --- test/dapltest/include/dapl_performance_test.h (revision 2532) +++ test/dapltest/include/dapl_performance_test.h (working copy) @@ -54,7 +54,7 @@ typedef struct { typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL port; int pipeline_len; @@ -70,17 +70,17 @@ typedef struct boolean_t is_remote_little_endian; DAT_CONN_QUAL base_port; struct dat_ia_attr ia_attr; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia * ia; + struct dat_pz * pz; DAT_CNO_HANDLE cno_handle; int reqt_evd_length; - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ int recv_evd_length; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ + struct dat_evd * recv_evd_hdl; /* receive */ int conn_evd_length; - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ + struct dat_evd * conn_evd_hdl; /* connect */ int creq_evd_length; - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_evd * creq_evd_hdl; /* "" request */ Performance_Ep_Context_t ep_context; } Performance_Test_t; Index: test/dapltest/include/dapl_proto.h =================================================================== --- test/dapltest/include/dapl_proto.h (revision 2572) +++ test/dapltest/include/dapl_proto.h (working copy) @@ -81,10 +81,10 @@ extern int g_status; /* dapl_bpool.c */ Bpool * DT_BpoolAlloc (Per_Test_Data_t * pt_ptr, DT_Tdep_Print_Head* phead, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE rmr_evd_handle, + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_ep *ep, + struct dat_evd *rmr_evd, int seg_size, int num_segs, int alignment, @@ -106,7 +106,7 @@ void DT_Bpool_print (DT_Tdep_ /* dapl_cnxn.c */ int get_ep_connection_state (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle); + struct dat_ep *ep); /* dapl_client.c */ int DT_cs_Client (Params_t * params_ptr, @@ -236,7 +236,7 @@ void DT_Performance_Cmd_Endia /* dapl_performance_client.c */ int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote); boolean_t DT_Performance_Test_Client_Connect ( @@ -261,7 +261,7 @@ boolean_t DT_Performance_Test /* dapl_performance_util.c */ boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, @@ -272,20 +272,20 @@ int DT_Performance_Test_Dest boolean_t is_server); boolean_t DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd * reqt_evd_hdl, Performance_Stats_t *stats); unsigned int DT_performance_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Mode_Type mode, Performance_Stats_t *stats); unsigned int DT_performance_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); unsigned int DT_performance_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, Performance_Stats_t *stats); /* dapl_performance_stats.c */ @@ -346,69 +346,69 @@ void DT_Free_Per_Test_Data (Per_Test_D /* dapl_test_util.c */ boolean_t DT_query (Per_Test_Data_t *pt_ptr, - DAT_IA_HANDLE ia_handle, - DAT_EP_HANDLE ep_handle); + struct dat_ia * ia, + struct dat_ep * ep); boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_post_send_buffer (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, Bpool * bp, int index, int size); boolean_t DT_conn_event_wait (DT_Tdep_Print_Head* phead, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, + struct dat_ep *ep, + struct dat_evd *evd, enum dat_event_number *event_number); boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, - DAT_EP_HANDLE *ep_handle ); + struct dat_evd *evd, + struct dat_ep * *ep ); boolean_t DT_cr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_cr_arrival_event_data *cr_stat_p); boolean_t DT_dto_event_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, boolean_t poll, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_dto_event_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_dto_completion_event_data *dtop); boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, struct dat_rmr_bind_completion_event_data *rmr_ptr); boolean_t DT_dto_check ( DT_Tdep_Print_Head* phead, struct dat_dto_completion_event_data *dto_p, - DAT_EP_HANDLE ep_expected, + struct dat_ep * ep_expected, int len_expected, DAT_DTO_COOKIE cookie_expected, char *message); boolean_t DT_rmr_check ( DT_Tdep_Print_Head* phead, struct dat_rmr_bind_completion_event_data *rmr_p, - DAT_RMR_HANDLE rmr_expected, + struct dat_rmr * rmr_expected, void * cookie_expected, char *message); boolean_t DT_cr_check (DT_Tdep_Print_Head* phead, struct dat_cr_arrival_event_data *cr_stat_p, - DAT_PSP_HANDLE psp_handle_expected, + struct dat_sp *psp_expected, DAT_CONN_QUAL port_expected, - DAT_CR_HANDLE *cr_handlep, + struct dat_cr **crp, char *message); /* dapl_thread.c */ @@ -460,13 +460,13 @@ void DT_Transaction_Cmd_Endia boolean_t to_wire); /* dapl_transaction_test.c */ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE ia_handle, + struct dat_ia *ia, struct sockaddr *remote); void DT_Transaction_Test_Server (void *params); boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, - DAT_IA_HANDLE * ia_handle, + struct dat_ia *ia, boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, @@ -491,15 +491,15 @@ boolean_t DT_handle_post_recv boolean_t DT_handle_send_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll); boolean_t DT_handle_recv_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE recv_evd_hdl, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *recv_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, int op_indx, boolean_t poll, @@ -507,7 +507,7 @@ boolean_t DT_handle_recv_op ( boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, - DAT_EVD_HANDLE reqt_evd_hdl, + struct dat_evd *reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, @@ -600,15 +600,15 @@ void DT_assert_fail (DT_Tdep_Print_H char *file, char *baseFile, int line); -int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle); +int DT_ia_open (char *dev_name, struct dat_ia **ia); int DT_ep_create (Params_t *params_ptr, - DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE *cr_evd, - DAT_EVD_HANDLE *conn_evd, - DAT_EVD_HANDLE *send_evd, - DAT_EVD_HANDLE *recv_evd, - DAT_EP_HANDLE *ep_handle); + struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd **cr_evd, + struct dat_evd **conn_evd, + struct dat_evd **send_evd, + struct dat_evd **recv_evd, + struct dat_ep **ep); void DT_fft_init_conn_struct (FFT_Connection_t *conn); void DT_fft_init_client (Params_t *params_ptr, FFT_Cmd_t *cmd, Index: test/dapltest/include/dapl_test_data.h =================================================================== --- test/dapltest/include/dapl_test_data.h (revision 2532) +++ test/dapltest/include/dapl_test_data.h (working copy) @@ -46,16 +46,16 @@ typedef struct int NextPortNumber; int num_clients; spinlock_t num_clients_lock; - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; - DAT_EVD_HANDLE reqt_evd_hdl; - DAT_EVD_HANDLE conn_evd_hdl; - DAT_EVD_HANDLE creq_evd_hdl; - DAT_EVD_HANDLE async_evd_hdl; - DAT_EVD_HANDLE rmr_evd_hdl; - DAT_EP_HANDLE ep_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_evd *recv_evd_hdl; + struct dat_evd *reqt_evd_hdl; + struct dat_evd *conn_evd_hdl; + struct dat_evd *creq_evd_hdl; + struct dat_evd *async_evd_hdl; + struct dat_evd *rmr_evd_hdl; + struct dat_ep *ep; + struct dat_sp *psp; Bpool *bpool; } Per_Server_Data_t; Index: test/dapltest/include/dapl_bpool.h =================================================================== --- test/dapltest/include/dapl_bpool.h (revision 2532) +++ test/dapltest/include/dapl_bpool.h (working copy) @@ -38,21 +38,21 @@ typedef struct Bpool_tag Bpool; struct Bpool_tag { unsigned char *alloc_ptr; uint32_t alloc_size; - DAT_PZ_HANDLE pz_handle; + struct dat_pz * pz; int seg_size; int num_segs; /* num segments */ unsigned char *buffer_start; /* Start of buffer area */ u64 buffer_size; /* Size of data buffer (rounded) */ u64 reg_addr; /* start of registered area */ u64 reg_size; /* size of registered area */ - DAT_EP_HANDLE ep_handle; /* EP area is registered to */ - DAT_LMR_HANDLE lmr_handle; /* local access */ + struct dat_ep * ep; /* EP area is registered to */ + struct dat_lmr * lmr; /* local access */ DAT_LMR_CONTEXT lmr_context; struct dat_lmr_triplet *tripl_start; /* local IOV */ boolean_t enable_rdma_write; /* remote access */ boolean_t enable_rdma_read; - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr * rmr_handle; DAT_RMR_CONTEXT rmr_context; - DAT_EVD_HANDLE rmr_evd_handle; + struct dat_evd * rmr_evd; }; #endif Index: test/dapltest/include/dapl_fft_util.h =================================================================== --- test/dapltest/include/dapl_fft_util.h (revision 2532) +++ test/dapltest/include/dapl_fft_util.h (working copy) @@ -55,14 +55,14 @@ typedef struct { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; - DAT_PSP_HANDLE psp_handle; - DAT_EP_HANDLE ep_handle; - DAT_EVD_HANDLE cr_evd, conn_evd, send_evd, recv_evd; + struct dat_ia *ia; + struct dat_pz *pz; + struct dat_sp *psp; + struct dat_ep *ep; + struct dat_evd *cr_evd, *conn_evd, *send_evd, *recv_evd; struct dat_event event; int count; - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; Bpool *bpool; struct dat_cr_arrival_event_data cr_stat; enum dat_event_number event_num; Index: test/dapltest/include/dapl_transaction_test.h =================================================================== --- test/dapltest/include/dapl_transaction_test.h (revision 2532) +++ test/dapltest/include/dapl_transaction_test.h (working copy) @@ -49,13 +49,13 @@ typedef struct typedef struct { - DAT_EP_HANDLE ep_handle; + struct dat_ep * ep; struct dat_ep_attr ep_attr; DAT_CONN_QUAL ia_port; Bpool *bp; Transaction_Test_Op_t op[ MAX_OPS ]; - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; + struct dat_sp * rsp; + struct dat_sp * psp; } Ep_Context_t; @@ -75,7 +75,7 @@ typedef struct boolean_t is_server; boolean_t remote_is_little_endian; Per_Test_Data_t *pt_ptr; - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; @@ -84,11 +84,11 @@ typedef struct Thread *thread; /* This group set up by each thread in DT_Transaction_Main() */ - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_hdl; /* receive */ - DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ - DAT_EVD_HANDLE conn_evd_hdl; /* connect */ - DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ + struct dat_pz * pz; + struct dat_evd * recv_evd_hdl; /* receive */ + struct dat_evd * reqt_evd_hdl; /* request+rmr */ + struct dat_evd * conn_evd_hdl; /* connect */ + struct dat_evd * creq_evd_hdl; /* "" request */ Ep_Context_t *ep_context; /* Statistics set by DT_Transaction_Run() */ Index: test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2532) +++ test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -39,7 +39,7 @@ typedef struct Tdep_Evd_Tag struct Tdep_Evd_Tag *evd_next; struct Tdep_Event_Tag *event_next; struct Tdep_Event_Tag *event_last; - DAT_EVD_HANDLE evd_handle; + struct dat_evd * evd; DT_WAIT_OBJECT wait_object; } Tdep_Evd; @@ -94,11 +94,11 @@ KDT_Evd_Destroy(void) } u32 -DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, +DT_Tdep_evd_create (struct dat_ia * ia, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE *evd_handle_ptr) + struct dat_evd **evd_ptr_ptr) { u32 dat_status; @@ -120,12 +120,12 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha upcall.upcall_func = DT_Tdep_Event_Callback; upcall.instance_data = evd_ptr; - dat_status = dat_evd_kcreate (ia_handle, + dat_status = dat_evd_kcreate (ia, evd_min_qlen, DAT_UPCALL_SINGLE_INSTANCE, &upcall, evd_flags, - evd_handle_ptr); + evd_ptr_ptr); if (dat_status != DAT_SUCCESS) { goto error; @@ -134,7 +134,7 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha /* initialize evd_ptr */ evd_ptr->event_next = NULL; evd_ptr->event_last = NULL; - evd_ptr->evd_handle = *evd_handle_ptr; + evd_ptr->evd = *evd_ptr_ptr; DT_Mdep_wait_object_init (&evd_ptr->wait_object); /* add evd_ptr to front of evd list */ @@ -154,7 +154,7 @@ error: return (dat_status); } u32 -DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_dequeue (struct dat_evd *evd, struct dat_event *dat_event) { u32 dat_status; @@ -163,11 +163,11 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ unsigned long flags; dat_status = DAT_SUCCESS; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -181,7 +181,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } /* Get event */ spin_lock_irqsave (&DT_Evd_Lock, flags); - event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd_handle */ + event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd */ if (event) { evd_ptr->event_next = event->event_next; /* point to next event */ @@ -212,7 +212,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ } u32 -DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, +DT_Tdep_evd_wait (struct dat_evd *evd, unsigned long timeout, struct dat_event *dat_event) { @@ -223,11 +223,11 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han dat_status = DAT_CLASS_ERROR | DAT_INTERNAL_ERROR; - /* find the evd_ptr associated with evd_handle */ + /* find the evd_ptr associated with evd */ evd_ptr = DT_Evd_List; while (evd_ptr) { - if (evd_ptr->evd_handle == evd_handle) + if (evd_ptr->evd == evd) { break; } @@ -290,7 +290,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han } u32 -DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle) +DT_Tdep_evd_free (struct dat_evd *evd) { Tdep_Evd *next; Tdep_Evd *last; @@ -298,13 +298,13 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han last = NULL; spin_lock_irq (&DT_Evd_Lock); next = DT_Evd_List; - if (next->evd_handle == evd_handle) + if (next->evd == evd) { DT_Evd_List = next->evd_next; } else { - while (next && next->evd_handle != evd_handle) + while (next && next->evd != evd) { last = next; next = next->evd_next; @@ -317,7 +317,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han spin_unlock_irq (&DT_Evd_Lock); DT_Mdep_Free (next); - return dat_evd_free(evd_handle); + return dat_evd_free(evd); } static void DT_Tdep_Event_Callback ( Index: dat-provider/dapl_cr_reject.c =================================================================== --- dat-provider/dapl_cr_reject.c (revision 2562) +++ dat-provider/dapl_cr_reject.c (working copy) @@ -40,36 +40,36 @@ * Reject a connection request from the active remote side requesting * an Endpoint. */ -u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) +u32 dapl_cr_reject(struct dat_cr *cr) { struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; struct dapl_sp *sp_ptr; u32 dat_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* * Clean up provider created EP if there is one: only if * DAT_PSP_PROVIDER_FLAG was set on the PSP */ - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - entry_ep_handle = cr_ptr->param.local_ep_handle; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; + entry_ep = cr_ptr->param.local_ep; entry_ep_state = 0; if (ep_ptr) { entry_ep_state = ep_ptr->param.ep_state; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - cr_ptr->param.local_ep_handle = NULL; + cr_ptr->param.local_ep = NULL; } dat_status = dapl_ib_reject_connection(cr_ptr->ib_cm_handle); @@ -77,8 +77,8 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand if (ep_ptr != NULL) { /* Revert our state to the beginning */ ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; - cr_ptr->param.local_ep_handle = (DAT_EP_HANDLE) ep_ptr; + cr_ptr->param.local_ep = entry_ep; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; } } else { /* @@ -88,13 +88,13 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand */ sp_ptr = cr_ptr->sp_ptr; if (ep_ptr && sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) - (void)dapl_ep_free(ep_ptr); + (void)dapl_ep_free((struct dat_ep *)ep_ptr); /* Remove the CR from the queue, then free it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); } Index: dat-provider/dapl_srq.c =================================================================== --- dat-provider/dapl_srq.c (revision 2562) +++ dat-provider/dapl_srq.c (working copy) @@ -51,13 +51,9 @@ */ static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) { - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); - - /* reset magic to prevent reuse */ - srq_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); + dapl_ia_unlink_srq(srq_ptr->common.owner_ia, srq_ptr); dapl_cb_free(&srq_ptr->recv_buffer); - /* no need to destroy srq_ptr->header.lock */ + /* no need to destroy srq_ptr->common.lock */ kfree(srq_ptr); } @@ -91,19 +87,12 @@ static struct dapl_srq *dapl_srq_alloc(s /* zero the structure */ memset(srq_ptr, 0, sizeof *srq_ptr); - /* - * initialize the header - */ - srq_ptr->header.provider = ia_ptr->header.provider; - srq_ptr->header.magic = DAPL_MAGIC_SRQ; - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; - srq_ptr->header.owner_ia = ia_ptr; - srq_ptr->header.user_context.as_64 = 0; - srq_ptr->header.user_context.as_ptr = NULL; + srq_ptr->srq.common.provider = ia_ptr->ia.common.provider; + srq_ptr->common.owner_ia = ia_ptr; atomic_set(&srq_ptr->srq_ref_count, 0); - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); - spin_lock_init(&srq_ptr->header.lock); + dapl_llist_init_entry(&srq_ptr->common.ia_list_entry); + spin_lock_init(&srq_ptr->common.lock); /* * Initialize the body. @@ -130,15 +119,15 @@ bail: * dapl_srq_create * * Create an instance of a Shared Receive Queue that is provided to the - * consumer at srq_handle. + * consumer at srq. * * Input: - * ia_handle - * pz_handle + * ia + * pz * srq_attr * * Output: - * srq_handle + * srq * * Returns: * DAT_SUCCESS @@ -148,8 +137,8 @@ bail: * ?DAT_INVALID_ATTRIBUTE?? * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) +u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_srq_attr *srq_attr, struct dat_srq **srq) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; @@ -157,14 +146,14 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_create (%p, %p, %p, %p)\n", - ia_handle, pz_handle, srq_attr, srq_handle); + ia, pz, srq_attr, srq); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -175,12 +164,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - - if (srq_handle == NULL) { + if (srq == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -197,9 +181,9 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han goto bail; } - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; + srq_ptr->param.ia = (struct dat_ia *) ia_ptr; srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; - srq_ptr->param.pz_handle = pz_handle; + srq_ptr->param.pz = pz; /* * XXX Allocate provider resource here!!! @@ -211,7 +195,7 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_han /* Link it onto the IA */ dapl_ia_link_srq(ia_ptr, srq_ptr); - *srq_handle = srq_ptr; + *srq = (struct dat_srq *)srq_ptr; bail: return status; @@ -223,7 +207,7 @@ bail: * Destroy an instance of an SRQ * * Input: - * srq_handle + * srq * * Output: * none @@ -233,22 +217,22 @@ bail: * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) +u32 dapl_srq_free(struct dat_srq *srq) { struct dapl_srq *srq_ptr; struct dapl_ia *ia_ptr; struct dat_srq_param *param; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq); - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; param = &srq_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { + if (!srq_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -263,17 +247,17 @@ u32 dapl_srq_free(DAT_SRQ_HANDLE srq_han goto bail; } - ia_ptr = srq_ptr->header.owner_ia; + ia_ptr = srq_ptr->common.owner_ia; /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_lock_irqsave(&srq_ptr->common.lock, srq_ptr->common.flags); /* Remove the SRQ from the IA */ dapl_ia_unlink_srq(ia_ptr, srq_ptr); - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); + spin_unlock_irqrestore(&srq_ptr->common.lock, srq_ptr->common.flags); /* * Finish tearing everything down. @@ -301,7 +285,7 @@ bail: * over a connection of any ep handle into local_iov * * Input: - * srq_handle + * srq * num_segments * local_iov * user_cookie @@ -318,7 +302,7 @@ bail: * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie) { @@ -328,14 +312,14 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_post_recv (%p, %d, %p, %P)\n", - srq_handle, num_segments, local_iov, user_cookie.as_64); + srq, num_segments, local_iov, user_cookie.as_64); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * Synchronization ok since this buffer is only used for receive @@ -369,16 +353,15 @@ bail: return status; } -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) +u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_srq_query (%p, %x, %p)\n", - srq_handle, srq_param); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", srq, + srq_param); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } @@ -387,7 +370,7 @@ u32 dapl_srq_query(DAT_SRQ_HANDLE srq_ha goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* * XXX Need to calculate available_dto_count and outstanding_dto_count @@ -407,7 +390,7 @@ bail: * Modify the size fo the event queue of a Shared Recieve Queue * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -421,22 +404,22 @@ bail: * DAT_INVALID_STATE */ -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) +u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto) { struct dapl_ia *ia_ptr; struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", - srq_handle, srq_max_recv_dto); + srq, srq_max_recv_dto); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; - ia_ptr = srq_ptr->header.owner_ia; + srq_ptr = (struct dapl_srq *)srq; + ia_ptr = srq_ptr->common.owner_ia; /* * Check for nonsense requests per the spec @@ -463,7 +446,7 @@ bail: * event if it is reached. * * Input: - * srq_handle + * srq * srq_max_recv_dto * * Output: @@ -476,20 +459,20 @@ bail: * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) +u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark) { struct dapl_srq *srq_ptr; u32 status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", - srq_handle, low_watermark); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq, + low_watermark); - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - srq_ptr = (struct dapl_srq *)srq_handle; + srq_ptr = (struct dapl_srq *)srq; /* XXX Put implementation here XXX */ Index: dat-provider/dapl_openib_util.c =================================================================== --- dat-provider/dapl_openib_util.c (revision 2563) +++ dat-provider/dapl_openib_util.c (working copy) @@ -171,7 +171,7 @@ u32 dapl_ib_close_hca(struct dapl_hca *h * Alloc a CQ * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * cqlen minimum QLen * @@ -227,7 +227,7 @@ u32 dapl_ib_cq_resize(struct dapl_ia *ia * Set up CQ completion notifications * * Input: - * ia_handle IA handle + * ia IA handle * evd_ptr pointer to EVD struct * * Output: @@ -253,7 +253,7 @@ u32 dapl_set_cq_notify(struct dapl_ia *i * Register a virtual memory region * * Input: - * ia_handle IA handle + * ia IA handle * lmr pointer to dapl_lmr struct * virt_addr virtual address of beginning of mem region * length length of memory region @@ -300,7 +300,7 @@ u32 dapl_ib_mr_register_physical(struct iova = buf_list[0].addr; acl = dapl_ib_convert_mem_privileges(privileges); acl |= IB_ACCESS_MW_BIND; - mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz_handle)->pd, + mr = ib_reg_phys_mr(((struct dapl_pz *)lmr->param.pz)->pd, buf_list, length, acl, &iova); kfree(buf_list); if (IS_ERR(mr)) { @@ -393,7 +393,7 @@ u32 dapl_ib_mw_alloc(struct dapl_rmr *rm int ib_status; struct ib_mw *mw; - mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz_handle)->pd); + mw = ib_alloc_mw(((struct dapl_pz *)rmr->param.pz)->pd); if (IS_ERR(mw)) { ib_status = PTR_ERR(mw); dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -540,7 +540,7 @@ u32 dapl_ib_mw_unbind(struct dapl_rmr *r * Set up an asynchronous callbacks of various kinds * * Input: - * ia_handle IA handle + * ia IA handle * handler_type type of handler to set up * callback_handle handle param for completion callbacks * callback callback routine pointer Index: dat-provider/dapl_ia_memtype_hint.c =================================================================== --- dat-provider/dapl_ia_memtype_hint.c (revision 2562) +++ dat-provider/dapl_ia_memtype_hint.c (working copy) @@ -37,7 +37,7 @@ * Provide a hint about optimal memory alignment and size * * Input: - * ia_handle, + * ia, * mem_type, * length, * mem_optimization, @@ -53,7 +53,7 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, @@ -63,11 +63,11 @@ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE i u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", - ia_handle); + ia); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; Index: dat-provider/dapl_openib_qp.c =================================================================== --- dat-provider/dapl_openib_qp.c (revision 2563) +++ dat-provider/dapl_openib_qp.c (working copy) @@ -74,17 +74,17 @@ u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ attr = &ep_ptr->param.ep_attr; - dapl_os_assert(ep_ptr->param.pz_handle != NULL); - ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz_handle)->pd; + dapl_os_assert(ep_ptr->param.pz != NULL); + ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz)->pd; ib_hca_handle = ia_ptr->hca_ptr->ib_hca_handle; cq_recv = NULL; cq_send = NULL; cq_recv = dapl_get_dto_cq(ia_ptr, - (struct dapl_evd *)ep_ptr->param.recv_evd_handle); + (struct dapl_evd *)ep_ptr->param.recv_evd); cq_send = dapl_get_dto_cq(ia_ptr, (struct dapl_evd *)ep_ptr->param. - request_evd_handle); + request_evd); qp_attr.send_cq = cq_send; qp_attr.recv_cq = cq_recv; @@ -160,7 +160,7 @@ struct ib_cq *dapl_get_dto_cq(struct dap int cq_entries; int status; - if (evd_ptr != DAT_HANDLE_NULL) + if (evd_ptr != NULL) cq = evd_ptr->cq; else if (ia_ptr->hca_ptr->null_cq != NULL) cq = ia_ptr->hca_ptr->null_cq; Index: dat-provider/dapl_openib_util.h =================================================================== --- dat-provider/dapl_openib_util.h (revision 2563) +++ dat-provider/dapl_openib_util.h (working copy) @@ -59,7 +59,7 @@ u32 dapl_ib_qp_free(struct dapl_ep *ep_p u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, struct dat_ep_attr *ep_attr); -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data); @@ -70,12 +70,12 @@ u32 dapl_ib_setup_conn_listener(struct d u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr); -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, + struct dat_ep *ep, int private_data_size, const void *private_data); -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle); +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm); u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, DAPL_ASYNC_HANDLER_TYPE handler_type, @@ -126,14 +126,14 @@ u32 dapl_ib_completion_notify(struct dap enum dat_dto_completion_status dapl_ib_get_dto_status(struct ib_wc *cqe_ptr); -void dapl_ib_reinit_ep(struct dapl_ep * ep_ptr); +void dapl_ib_reinit_ep(struct dapl_ep *ep_ptr); -void dapl_ib_disconnect_clean(struct dapl_ep * ep_ptr, boolean_t passive); +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t passive); u32 dapl_ib_get_async_event(struct ib_event *cause_ptr, enum dat_event_number *async_event); -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address); static inline u32 dapl_ib_status_convert(int32_t ib_status) Index: dat-provider/dapl_ia_close.c =================================================================== --- dat-provider/dapl_ia_close.c (revision 2562) +++ dat-provider/dapl_ia_close.c (working copy) @@ -40,7 +40,7 @@ * Close a provider, clean up resources, etc. * * Input: - * ia_handle + * ia * * Output: * none @@ -50,30 +50,29 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags ia_flags) +u32 dapl_ia_close(struct dat_ia *ia, enum dat_close_flags ia_flags) { struct dapl_ia *ia_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); + "dapl_ia_close (%p, %d)\n", ia, ia_flags); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) { + if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) dat_status = dapl_ia_abrupt_close(ia_ptr); - } else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) { + else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) dat_status = dapl_ia_graceful_close(ia_ptr); - } else { + else dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - } - bail: +bail: return dat_status; } Index: dat-provider/dapl_cr_callback.c =================================================================== --- dat-provider/dapl_cr_callback.c (revision 2563) +++ dat-provider/dapl_cr_callback.c (working copy) @@ -50,9 +50,9 @@ static u32 dapl_connection_request(struc struct dapl_cr *cr_ptr; struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; - cr_ptr = dapl_cr_alloc(sp_ptr->header.owner_ia); + cr_ptr = dapl_cr_alloc(sp_ptr->common.owner_ia); if (cr_ptr == NULL) { /* Invoking function will call dapl_ib_cm_reject() */ return DAT_INSUFFICIENT_RESOURCES; @@ -79,7 +79,7 @@ static u32 dapl_connection_request(struc } /* EP will be NULL unless RSP service point */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { /* @@ -89,7 +89,7 @@ static u32 dapl_connection_request(struc * EP we are out of resources and need to tell the * requestor that we cant help them. */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; ep_ptr = dapl_ep_alloc(ia_ptr, NULL); if (ep_ptr == NULL) { dapl_cr_free(cr_ptr); @@ -100,7 +100,7 @@ static u32 dapl_connection_request(struc dapl_ia_link_ep(ia_ptr, ep_ptr); } - cr_ptr->param.local_ep_handle = ep_ptr; + cr_ptr->param.local_ep = (struct dat_ep *)ep_ptr; if (ep_ptr != NULL) { /* Assign valid EP fields: RSP and PSP_PROVIDER_FLAG only */ @@ -109,12 +109,11 @@ static u32 dapl_connection_request(struc DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING; } else { /* RSP */ - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == DAT_SP_TYPE_RSP); ep_ptr->param.ep_state = DAT_EP_STATE_PASSIVE_CONNECTION_PENDING; } - ep_ptr->cm_handle = ib_cm_handle; + ep_ptr->cm = ib_cm_handle; } /* link the CR onto the SP so we can pick it up later */ @@ -122,16 +121,16 @@ static u32 dapl_connection_request(struc /* Post the event. */ /* assign sp_ptr to union to avoid typecast errors from some compilers */ - sp_handle.psp_handle = (DAT_PSP_HANDLE) sp_ptr; - dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp_handle, - (struct sockaddr *) &sp_ptr->header.owner_ia->hca_ptr->hca_address, - sp_ptr->conn_qual, (DAT_CR_HANDLE) cr_ptr); + sp = (struct dat_sp *)sp_ptr; + dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, sp, + (struct sockaddr *)&sp_ptr->common.owner_ia->hca_ptr->hca_address, + sp_ptr->conn_qual, (struct dat_cr *)cr_ptr); if (dat_status != DAT_SUCCESS) { /* Take the CR off the list, we can't use it */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); (void)dapl_ib_reject_connection(ib_cm_handle); @@ -157,13 +156,7 @@ struct dapl_ep *dapl_get_sp_ep(struct da /* * acquire the lock, we may be racing with other threads here */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); - /* Verify under lock that the SP is still valid */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - return NULL; - } + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * There are potentially multiple connections in progress. Need to * go through the list and find the one we are interested @@ -172,16 +165,12 @@ struct dapl_ep *dapl_get_sp_ep(struct da */ cr_ptr = dapl_sp_search_cr(sp_ptr, ib_cm_handle); if (cr_ptr == NULL) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); return NULL; } - ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; - - /* Quick check to ensure our EP is still valid */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) - ep_ptr = NULL; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep; /* The CR record is discarded in all except for the CONNECTED case, * as it will have no further relevance. @@ -211,31 +200,31 @@ struct dapl_ep *dapl_get_sp_ep(struct da "dapl_get_sp_ep disconnect dump sp: %p \n", sp_ptr); /* Decrement the ref count on the EVD */ - if (sp_ptr->evd_handle) { + if (sp_ptr->evd) { atomic_dec(&((struct dapl_evd *)sp_ptr-> - evd_handle)->evd_ref_count); - sp_ptr->evd_handle = NULL; + evd)->evd_ref_count); + sp_ptr->evd = NULL; } sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - (void)dapl_ib_remove_conn_listener(sp_ptr->header. + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + (void)dapl_ib_remove_conn_listener(sp_ptr->common. owner_ia, sp_ptr); dapl_ia_unlink_sp((struct dapl_ia *) - sp_ptr->header.owner_ia, + sp_ptr->common.owner_ia, sp_ptr); dapl_sp_dealloc(sp_ptr); dapl_cr_free(cr_ptr); goto skip_unlock; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); } else - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); skip_unlock: return ep_ptr; @@ -255,16 +244,6 @@ void dapl_cr_callback(struct dapl_cm_id ib_cm_handle, event, sp_ptr); /* - * The SP pointer could have been cleaned up in a racing - * CM callback, check to see if we should just exit here - */ - if (sp_ptr->header.magic == DAPL_MAGIC_INVALID) { - return; - } - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); - - /* * CONNECT_REQUEST events create an event on the PSP * EVD, which will trigger connection processing. The * sequence is: @@ -277,7 +256,7 @@ void dapl_cr_callback(struct dapl_cm_id */ if (event == DAT_CONNECTION_REQUEST_EVENT) { ep_ptr = NULL; - evd_ptr = sp_ptr->evd_handle; + evd_ptr = (struct dapl_evd *)sp_ptr->evd; } else { /* see if there is an EP connected with this CM handle */ ep_ptr = dapl_get_sp_ep(ib_cm_handle, sp_ptr, event); @@ -286,7 +265,7 @@ void dapl_cr_callback(struct dapl_cm_id if (ep_ptr == NULL) return; - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; /* if something has happened to our EVD, bail. */ if (evd_ptr == NULL) return; @@ -296,30 +275,29 @@ void dapl_cr_callback(struct dapl_cm_id switch (event) { case DAT_CONNECTION_REQUEST_EVENT: - { /* * Requests arriving on a disabled SP are immediately rejected */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); if (sp_ptr->listening == FALSE) { - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, "---> dapl_cr_callback: conn event on down SP\n"); (void)dapl_ib_reject_connection(ib_cm_handle); return; } - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_RSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_RSP) { /* * RSP connections only allow a single connection. Close * it down NOW so we reject any further connections. */ sp_ptr->listening = FALSE; } - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* * Only occurs on the passive side of a connection @@ -332,24 +310,20 @@ void dapl_cr_callback(struct dapl_cm_id evd_ptr = NULL; break; - } case DAT_CONNECTION_EVENT_ESTABLISHED: - { /* This is just a notification the connection is now * established, there isn't any private data to deal with. * * Update the EP state and cache a copy of the cm handle, * then let the user know we are ready to go. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->header.magic != DAPL_MAGIC_EP || - ep_ptr->param.ep_state != - DAT_EP_STATE_COMPLETION_PENDING) { + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING) { /* If someone pulled the plug on the EP or connection, * just exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; /* Set evd_ptr = NULL so we don't generate an event below */ evd_ptr = NULL; @@ -358,72 +332,63 @@ void dapl_cr_callback(struct dapl_cm_id } ep_ptr->param.ep_state = DAT_EP_STATE_CONNECTED; - ep_ptr->cm_handle = ib_cm_handle; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + ep_ptr->cm = ib_cm_handle; + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_DISCONNECTED: - { /* * EP is now fully disconnected; initiate any post processing * to reset the underlying QP and get the EP ready for * another connection */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { /* The disconnect has already occurred, we are now * cleaned up and ready to exit */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); return; } ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: - { /* * After posting an accept the requesting node has * stopped talking. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } case DAT_CONNECTION_EVENT_BROKEN: - { - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep_ptr, FALSE); - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); break; - } default: - { evd_ptr = NULL; dapl_os_assert(0); /* shouldn't happen */ break; } - } if (evd_ptr != NULL) dat_status = dapl_evd_post_connection_event(evd_ptr, event, - (DAT_HANDLE) ep_ptr, + (struct dat_ep *)ep_ptr, 0, NULL); if (dat_status != DAT_SUCCESS) { Index: dat-provider/dapl_cr_accept.c =================================================================== --- dat-provider/dapl_cr_accept.c (revision 2563) +++ dat-provider/dapl_cr_accept.c (working copy) @@ -39,42 +39,39 @@ * Establish a connection between active side (remote endpoint) * and passive side (local endpoint). */ -u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - int private_data_size, const void *private_data) +u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, + const void *private_data) { struct dapl_ep *ep_ptr; u32 dat_status; struct dapl_cr *cr_ptr; enum dat_ep_state entry_ep_state; - DAT_EP_HANDLE entry_ep_handle; + struct dat_ep *entry_ep; - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_cr_accept (%p, %p, %d, %p)\n", - cr_handle, ep_handle, private_data_size, private_data); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_accept (%p, %p, %d, %p)\n", + cr, ep, private_data_size, private_data); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (!cr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* - * Return an error if we have an ep_handle and the CR already has an + * Return an error if we have an ep and the CR already has an * EP, indicating this is an RSP connection or PSP_PROVIDER_FLAG was * specified. */ - if (ep_handle != NULL && - (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP) || - cr_ptr->param.local_ep_handle != NULL)) { + if (ep != NULL && cr_ptr->param.local_ep != NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* Make sure we have an EP handle in one place or another */ - if (ep_handle == NULL && cr_ptr->param.local_ep_handle == NULL) { + if (ep == NULL && cr_ptr->param.local_ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -86,84 +83,71 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand } /* - * ep_handle is NULL if the user specified DAT_PSP_PROVIDER_FLAG + * ep is NULL if the user specified DAT_PSP_PROVIDER_FLAG * OR this is an RSP connection; retrieve it from the cr. */ - if (ep_handle == NULL) { - ep_handle = cr_ptr->param.local_ep_handle; - if ((((struct dapl_ep *)ep_handle)->param.ep_state != + if (ep == NULL) { + ep = cr_ptr->param.local_ep; + if ((((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) - && (((struct dapl_ep *)ep_handle)->param.ep_state != + && (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING)) { return DAT_INVALID_STATE; } } else { /* ensure this EP isn't connected or in use */ - if (((struct dapl_ep *)ep_handle)->param.ep_state != + if (((struct dapl_ep *)ep)->param.ep_state != DAT_EP_STATE_UNCONNECTED) { return DAT_INVALID_STATE; } } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required * We do all verification and state change under lock, at which * point the EP state should protect us from most races. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } /* The qp must be attached by this point! */ - if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) - { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } entry_ep_state = ep_ptr->param.ep_state; - entry_ep_handle = cr_ptr->param.local_ep_handle; + entry_ep = cr_ptr->param.local_ep; ep_ptr->param.ep_state = DAT_EP_STATE_COMPLETION_PENDING; - ep_ptr->cm_handle = cr_ptr->ib_cm_handle; + ep_ptr->cm = cr_ptr->ib_cm_handle; ep_ptr->cr_ptr = cr_ptr; ep_ptr->param.remote_ia_address_ptr = cr_ptr->param.remote_ia_address_ptr; - cr_ptr->param.local_ep_handle = ep_handle; + cr_ptr->param.local_ep = ep; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); - dat_status = dapl_ib_accept_connection(cr_handle, - ep_handle, + dat_status = dapl_ib_accept_connection(cr, + ep, private_data_size, private_data); /* @@ -176,16 +160,14 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand * system error, app termination, perhaps other reasons. */ dat_status = - dapl_evd_post_connection_event(ep_ptr->param. - request_evd_handle, + dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. + request_evd, DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR, - (DAT_HANDLE) ep_ptr, + (void *) ep_ptr, 0, NULL); - - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } else { ep_ptr->param.ep_state = entry_ep_state; - cr_ptr->param.local_ep_handle = entry_ep_handle; + cr_ptr->param.local_ep = entry_ep; ep_ptr->cr_ptr = NULL; ep_ptr->param.remote_ia_address_ptr = NULL; } @@ -199,14 +181,8 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); } - } else { - /* Make this CR invalid. We need to hang on to it until - * the connection terminates, but it's destroyed from - * the app point of view. - */ - cr_ptr->header.magic = DAPL_MAGIC_CR_DESTROYED; } - bail: +bail: return dat_status; } Index: dat-provider/dapl_pz.c =================================================================== --- dat-provider/dapl_pz.c (revision 2562) +++ dat-provider/dapl_pz.c (working copy) @@ -44,18 +44,11 @@ static struct dapl_pz *dapl_pz_alloc(str /* zero the structure */ memset(pz, 0, sizeof *pz); - /* - * initialize the header - */ - pz->header.provider = ia->header.provider; - pz->header.magic = DAPL_MAGIC_PZ; - pz->header.handle_type = DAT_HANDLE_TYPE_PZ; - pz->header.owner_ia = ia; - pz->header.user_context.as_64 = 0; - pz->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&pz->header.ia_list_entry); + pz->pz.common.provider = ia->ia.common.provider; + pz->common.owner_ia = ia; + dapl_llist_init_entry(&pz->common.ia_list_entry); dapl_ia_link_pz(ia, pz); - spin_lock_init(&pz->header.lock); + spin_lock_init(&pz->common.lock); /* * initialize the body @@ -67,83 +60,81 @@ static struct dapl_pz *dapl_pz_alloc(str static void dapl_pz_dealloc(struct dapl_pz *pz) { - /* reset magic to prevent reuse */ - pz->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_pz(pz->header.owner_ia, pz); + dapl_ia_unlink_pz(pz->common.owner_ia, pz); kfree(pz); } -u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) +u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; int ib_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", - ia_handle, pz_handle); + ia, pz); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error1; } - if (NULL == pz_handle) { + if (NULL == pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto error1; } - ia = (struct dapl_ia *)ia_handle; + dapl_ia = (struct dapl_ia *)ia; - pz = dapl_pz_alloc(ia); - if (NULL == pz) { + dapl_pz = dapl_pz_alloc(dapl_ia); + if (!dapl_pz) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - pz->pd = ib_alloc_pd(ia->hca_ptr->ib_hca_handle); - if (IS_ERR(pz->pd)) { - ib_status = PTR_ERR(pz->pd); + dapl_pz->pd = ib_alloc_pd(dapl_ia->hca_ptr->ib_hca_handle); + if (IS_ERR(dapl_pz->pd)) { + ib_status = PTR_ERR(dapl_pz->pd); dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_alloc_pd failed: %X\n", ib_status); status = dapl_ib_status_convert(ib_status); goto error2; } - *pz_handle = pz; + *pz = (struct dat_pz *)dapl_pz; return DAT_SUCCESS; error2: - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); error1: - *pz_handle = NULL; + *pz = NULL; return status; } -u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) +u32 dapl_pz_free(struct dat_pz *pz) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; int ib_status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - if (0 != atomic_read(&pz->pz_ref_count)) { + if (0 != atomic_read(&dapl_pz->pz_ref_count)) { status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_PZ_IN_USE); goto error; } - ib_status = ib_dealloc_pd(pz->pd); + ib_status = ib_dealloc_pd(dapl_pz->pd); if (ib_status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_dealloc_pd failed: %X\n", ib_status); @@ -151,22 +142,22 @@ u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle goto error; } - dapl_pz_dealloc(pz); + dapl_pz_dealloc(dapl_pz); return DAT_SUCCESS; error: return status; } -u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) +u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_param) { - struct dapl_pz *pz; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query(%p, %x, %p)\n", - pz_handle, pz_param); + pz, pz_param); - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto error; } @@ -175,8 +166,8 @@ u32 dapl_pz_query(DAT_PZ_HANDLE pz_handl goto error; } - pz = (struct dapl_pz *)pz_handle; - pz_param->ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; + dapl_pz = (struct dapl_pz *)pz; + pz_param->ia = (struct dat_ia *)dapl_pz->common.owner_ia; return DAT_SUCCESS; error: Index: dat-provider/dapl_cr_handoff.c =================================================================== --- dat-provider/dapl_cr_handoff.c (revision 2562) +++ dat-provider/dapl_cr_handoff.c (working copy) @@ -37,7 +37,7 @@ * Hand the connection request to another Sevice pont specified by the * Connectin Qualifier. */ -u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) +u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL cr_handoff) { /* handoff */ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); } Index: dat-provider/dapl_lmr.c =================================================================== --- dat-provider/dapl_lmr.c (revision 2562) +++ dat-provider/dapl_lmr.c (working copy) @@ -36,7 +36,7 @@ static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_desc, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags mem_priv) { struct dapl_lmr *lmr; @@ -49,27 +49,20 @@ static struct dapl_lmr *dapl_lmr_alloc(s /* zero the structure */ memset(lmr, 0, sizeof *lmr); - /* - * initialize the header - */ - lmr->header.provider = ia->header.provider; - lmr->header.magic = DAPL_MAGIC_LMR; - lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; - lmr->header.owner_ia = ia; - lmr->header.user_context.as_64 = 0; - lmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&lmr->header.ia_list_entry); + lmr->lmr.common.provider = ia->ia.common.provider; + lmr->common.owner_ia = ia; + dapl_llist_init_entry(&lmr->common.ia_list_entry); dapl_ia_link_lmr(ia, lmr); - spin_lock_init(&lmr->header.lock); + spin_lock_init(&lmr->common.lock); /* * initialize the body */ - lmr->param.ia_handle = (DAT_IA_HANDLE) ia; + lmr->param.ia = (struct dat_ia *)ia; lmr->param.mem_type = mem_type; lmr->param.region_desc = region_desc; lmr->param.length = length; - lmr->param.pz_handle = pz_handle; + lmr->param.pz = pz; lmr->param.mem_priv = mem_priv; atomic_set(&lmr->lmr_ref_count, 0); @@ -78,9 +71,7 @@ static struct dapl_lmr *dapl_lmr_alloc(s static void dapl_lmr_dealloc(struct dapl_lmr *lmr) { - /* reset magic to prevent reuse */ - lmr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); + dapl_ia_unlink_lmr(lmr->common.owner_ia, lmr); kfree(lmr); } @@ -88,49 +79,50 @@ static void dapl_lmr_dealloc(struct dapl static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, u64 length, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; reg_desc.for_va = virt_addr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, reg_desc, length, + (struct dat_pz *)pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register(ia, lmr, virt_addr, length, privileges); + status = dapl_ib_mr_register(ia, new_lmr, virt_addr, length, + privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64)(unsigned long) virt_addr; if (registered_length) @@ -139,9 +131,9 @@ static inline u32 dapl_lmr_create_virtua return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; @@ -151,26 +143,26 @@ static inline u32 dapl_lmr_create_physic DAT_REGION_DESCRIPTION phys_addr, u64 page_count, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; u64 *array = phys_addr.for_array; u32 status; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, page_count, - (DAT_PZ_HANDLE) pz, privileges); + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, + page_count, (struct dat_pz *) pz, privileges); - if (NULL == lmr) { + if (NULL == new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, + status = dapl_ib_mr_register_physical(ia, new_lmr, phys_addr.for_array, page_count, privileges); if (DAT_SUCCESS != status) @@ -178,23 +170,23 @@ static inline u32 dapl_lmr_create_physic /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = array[0]; if (registered_length) @@ -203,9 +195,9 @@ static inline u32 dapl_lmr_create_physic return DAT_SUCCESS; error3: - (void)dapl_ib_mr_deregister(lmr); + (void)dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } @@ -214,13 +206,13 @@ static inline u32 dapl_lmr_create_lmr(st struct dapl_lmr *original_lmr, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - struct dapl_lmr *lmr; + struct dapl_lmr *new_lmr; DAT_REGION_DESCRIPTION reg_desc; u32 status; @@ -230,39 +222,39 @@ static inline u32 dapl_lmr_create_lmr(st if (status != DAT_SUCCESS) goto error1; - reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; + reg_desc.for_lmr = (struct dat_lmr *) original_lmr; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, - (DAT_PZ_HANDLE) pz, privileges); - if (NULL == lmr) { + new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, + (struct dat_pz *) pz, privileges); + if (!new_lmr) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto error1; } - status = dapl_ib_mr_register_shared(ia, lmr, privileges); + status = dapl_ib_mr_register_shared(ia, new_lmr, privileges); if (DAT_SUCCESS != status) goto error2; /* if the LMR context is already in the hash table */ status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + new_lmr->param.lmr_context, NULL); if (status == DAT_SUCCESS) goto error3; status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + new_lmr->param.lmr_context, lmr); if (status != DAT_SUCCESS) goto error3; atomic_inc(&pz->pz_ref_count); - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr) + *lmr = (struct dat_lmr *)new_lmr; if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + *lmr_context = (DAT_LMR_CONTEXT)new_lmr->param.lmr_context; if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + *rmr_context = (DAT_LMR_CONTEXT)new_lmr->param.rmr_context; if (registered_address) *registered_address = (u64) (unsigned long) original_lmr->param.region_desc.for_va; @@ -272,73 +264,72 @@ static inline u32 dapl_lmr_create_lmr(st return DAT_SUCCESS; error3: - dapl_ib_mr_deregister(lmr); + dapl_ib_mr_deregister(new_lmr); error2: - dapl_lmr_dealloc(lmr); + dapl_lmr_dealloc(new_lmr); error1: return status; } -u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags privileges, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - u64 *registered_length, u64 *registered_address) + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, + u64 *registered_address) { - struct dapl_ia *ia; - struct dapl_pz *pz; + struct dapl_ia *dapl_ia; + struct dapl_pz *dapl_pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", - ia_handle, mem_type); + ia, mem_type); - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - ia = (struct dapl_ia *)ia_handle; - pz = (struct dapl_pz *)pz_handle; + dapl_ia = (struct dapl_ia *)ia; + dapl_pz = (struct dapl_pz *)pz; switch (mem_type) { case DAT_MEM_TYPE_VIRTUAL: - status = dapl_lmr_create_virtual(ia, region_description.for_va, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_virtual(dapl_ia, + region_description.for_va, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_PHYSICAL: - status = dapl_lmr_create_physical(ia, region_description, - length, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_physical(dapl_ia, region_description, + length, dapl_pz, privileges, + lmr, lmr_context, rmr_context, registered_length, registered_address); break; case DAT_MEM_TYPE_LMR: { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; - if (DAPL_BAD_HANDLE(region_description.for_lmr_handle, - DAPL_MAGIC_LMR)) { + if (!region_description.for_lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto bail; } - lmr = (struct dapl_lmr *)region_description.for_lmr_handle; + dapl_lmr = (struct dapl_lmr *)region_description.for_lmr; - status = dapl_lmr_create_lmr(ia, lmr, pz, privileges, - lmr_handle, lmr_context, + status = dapl_lmr_create_lmr(dapl_ia, dapl_lmr, dapl_pz, + privileges, lmr, lmr_context, rmr_context, registered_length, registered_address); break; @@ -357,45 +348,45 @@ bail: return status; } -u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) +u32 dapl_lmr_free(struct dat_lmr *lmr) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } - lmr = (struct dapl_lmr *)lmr_handle; + dapl_lmr = (struct dapl_lmr *)lmr; - switch (lmr->param.mem_type) { + switch (dapl_lmr->param.mem_type) { case DAT_MEM_TYPE_PHYSICAL: case DAT_MEM_TYPE_VIRTUAL: case DAT_MEM_TYPE_LMR: { struct dapl_pz *pz; - if (0 != atomic_read(&lmr->lmr_ref_count)) + if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) return DAT_INVALID_STATE; status = dapl_hash_remove( - lmr->header.owner_ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); + dapl_lmr->common.owner_ia->hca_ptr->lmr_hash_table, + dapl_lmr->param.lmr_context, NULL); if (status != DAT_SUCCESS) goto error; - status = dapl_ib_mr_deregister(lmr); + status = dapl_ib_mr_deregister(dapl_lmr); if (status == DAT_SUCCESS) { - pz = (struct dapl_pz *)lmr->param.pz_handle; + pz = (struct dapl_pz *)dapl_lmr->param.pz; atomic_dec(&pz->pz_ref_count); - dapl_lmr_dealloc(lmr); - } else /* failure; put lmr back in hash table */ - dapl_hash_insert(lmr->header.owner_ia-> + dapl_lmr_dealloc(dapl_lmr); + } else /* failure; put dapl_lmr back in hash table */ + dapl_hash_insert(dapl_lmr->common.owner_ia-> hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); + dapl_lmr->param.lmr_context, dapl_lmr); break; } @@ -412,15 +403,15 @@ error: return status; } -u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) +u32 dapl_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *lmr_param) { - struct dapl_lmr *lmr; + struct dapl_lmr *dapl_lmr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", - lmr_handle, lmr_param); + lmr, lmr_param); - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + if (!lmr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); goto error; } @@ -429,8 +420,8 @@ u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_ha goto error; } - lmr = (struct dapl_lmr *)lmr_handle; - memcpy(lmr_param, &lmr->param, sizeof *lmr_param); + dapl_lmr = (struct dapl_lmr *)lmr; + memcpy(lmr_param, &dapl_lmr->param, sizeof *lmr_param); status = DAT_SUCCESS; error: return status; @@ -441,7 +432,7 @@ error: * will sync the cache with memory. */ -u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -449,11 +440,11 @@ u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDL u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_read(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } @@ -462,7 +453,7 @@ error: return status; } -u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments) { @@ -470,11 +461,11 @@ u32 dapl_lmr_sync_rdma_write(DAT_IA_HAND u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_write(%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); + ia, local_segments, num_segments); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto error; } Index: dat-provider/Makefile =================================================================== --- dat-provider/Makefile (revision 2563) +++ dat-provider/Makefile (working copy) @@ -27,8 +27,6 @@ PROVIDER_MODULES := \ dapl_cr_util \ dapl_ep \ dapl_evd \ - dapl_get_consumer_context \ - dapl_get_handle_type \ dapl_hash \ dapl_hca_util \ dapl_ia_close \ @@ -45,7 +43,6 @@ PROVIDER_MODULES := \ dapl_ring_buffer_util \ dapl_rmr \ dapl_rsp \ - dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ dapl_util Index: dat-provider/dapl_rmr.c =================================================================== --- dat-provider/dapl_rmr.c (revision 2562) +++ dat-provider/dapl_rmr.c (working copy) @@ -61,24 +61,17 @@ static struct dapl_rmr *dapl_rmr_alloc(s /* zero the structure */ memset(rmr, 0, sizeof *rmr); - /* - * initialize the header - */ - rmr->header.provider = pz->header.provider; - rmr->header.magic = DAPL_MAGIC_RMR; - rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; - rmr->header.owner_ia = pz->header.owner_ia; - rmr->header.user_context.as_64 = 0; - rmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&rmr->header.ia_list_entry); - dapl_ia_link_rmr(rmr->header.owner_ia, rmr); - spin_lock_init(&rmr->header.lock); + rmr->rmr.common.provider = pz->pz.common.provider; + rmr->common.owner_ia = pz->common.owner_ia; + dapl_llist_init_entry(&rmr->common.ia_list_entry); + dapl_ia_link_rmr(rmr->common.owner_ia, rmr); + spin_lock_init(&rmr->common.lock); /* * initialize the body */ - rmr->param.ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; - rmr->param.pz_handle = (DAT_PZ_HANDLE)pz; + rmr->param.ia = (struct dat_ia *)pz->common.owner_ia; + rmr->param.pz = (struct dat_pz *)pz; rmr->param.lmr_triplet.lmr_context = 0; rmr->param.lmr_triplet.pad = 0; rmr->param.lmr_triplet.virtual_address = 0; @@ -93,11 +86,8 @@ static struct dapl_rmr *dapl_rmr_alloc(s static void dapl_rmr_dealloc(struct dapl_rmr *rmr) { - /* reset magic to prevent reuse */ - rmr->header.magic = DAPL_MAGIC_INVALID; - - dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); - /* no need to destroy rmr->header.lock */ + dapl_ia_unlink_rmr(rmr->common.owner_ia, rmr); + /* no need to destroy rmr->common.lock */ kfree(rmr); } @@ -115,7 +105,7 @@ static inline u32 dapl_rmr_bind_fuse(str u32 status; boolean_t is_signaled; - status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, + status = dapl_hash_search(rmr->common.owner_ia->hca_ptr->lmr_hash_table, lmr_triplet->lmr_context, (DAPL_HASH_DATA *) &lmr); if (DAT_SUCCESS != status) { @@ -150,8 +140,8 @@ static inline u32 dapl_rmr_bind_fuse(str } /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ - if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || - (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { + if ((ep_ptr->param.pz != lmr->param.pz) || + (ep_ptr->param.pz != rmr->param.pz)) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } @@ -240,7 +230,7 @@ static inline u32 dapl_rmr_bind_unfuse(s } /* If the RMR and EP are not in the same PZ, there is an error */ - if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { + if (ep_ptr->param.pz != rmr->param.pz) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail1; } @@ -306,9 +296,9 @@ bail1: * Input: * Output: */ -u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, const struct dat_lmr_triplet *lmr_triplet, - enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, + enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) @@ -316,14 +306,14 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han struct dapl_rmr *rmr; struct dapl_ep *ep_ptr; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) + if (!rmr_handle) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) + if (!ep) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); rmr = (struct dapl_rmr *)rmr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* if the rmr should be bound */ if (0 != lmr_triplet->segment_length) @@ -341,7 +331,7 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * Create a remote memory region for the specified protection zone * * Input: - * pz_handle + * pz * * Output: * rmr_handle @@ -351,20 +341,20 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) +u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr) { - struct dapl_pz *pz; - struct dapl_rmr *rmr; + struct dapl_pz *dapl_pz; + struct dapl_rmr *dapl_rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + if (!pz) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); goto bail; } - pz = (struct dapl_pz *)pz_handle; + dapl_pz = (struct dapl_pz *)pz; - rmr = dapl_rmr_alloc(pz); + dapl_rmr = dapl_rmr_alloc(dapl_pz); if (rmr == NULL) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, @@ -372,18 +362,18 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han goto bail; } - status = dapl_ib_mw_alloc(rmr); + status = dapl_ib_mw_alloc(dapl_rmr); if (status != DAT_SUCCESS) { - dapl_rmr_dealloc(rmr); + dapl_rmr_dealloc(dapl_rmr); status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY_REGION); goto bail; } - atomic_inc(&pz->pz_ref_count); + atomic_inc(&dapl_pz->pz_ref_count); - *rmr_handle = rmr; + *rmr = (struct dat_rmr *)dapl_rmr; bail: return status; @@ -404,12 +394,12 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) +u32 dapl_rmr_free(struct dat_rmr *rmr_handle) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } @@ -438,12 +428,12 @@ bail: return status; } -u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) +u32 dapl_rmr_query(struct dat_rmr *rmr_handle, struct dat_rmr_param *rmr_param) { struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + if (!rmr_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); goto bail; } Index: dat-provider/dapl_cr_util.c =================================================================== --- dat-provider/dapl_cr_util.c (revision 2562) +++ dat-provider/dapl_cr_util.c (working copy) @@ -49,17 +49,10 @@ struct dapl_cr *dapl_cr_alloc(struct dap /* zero the structure */ memset(cr_ptr, 0, sizeof *cr_ptr); - /* - * initialize the header - */ - cr_ptr->header.provider = ia_ptr->header.provider; - cr_ptr->header.magic = DAPL_MAGIC_CR; - cr_ptr->header.handle_type = DAT_HANDLE_TYPE_CR; - cr_ptr->header.owner_ia = ia_ptr; - cr_ptr->header.user_context.as_64 = 0; - cr_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&cr_ptr->header.ia_list_entry); - spin_lock_init(&cr_ptr->header.lock); + cr_ptr->cr.common.provider = ia_ptr->ia.common.provider; + cr_ptr->common.owner_ia = ia_ptr; + dapl_llist_init_entry(&cr_ptr->common.ia_list_entry); + spin_lock_init(&cr_ptr->common.lock); return cr_ptr; } @@ -69,9 +62,5 @@ struct dapl_cr *dapl_cr_alloc(struct dap */ void dapl_cr_free(struct dapl_cr *cr_ptr) { - dapl_os_assert(cr_ptr->header.magic == DAPL_MAGIC_CR || - cr_ptr->header.magic == DAPL_MAGIC_CR_DESTROYED); - /* reset magic to prevent reuse */ - cr_ptr->header.magic = DAPL_MAGIC_INVALID; kfree(cr_ptr); } Index: dat-provider/dapl_psp.c =================================================================== --- dat-provider/dapl_psp.c (revision 2562) +++ dat-provider/dapl_psp.c (working copy) @@ -45,13 +45,13 @@ * the user. * * Input: - * ia_handle - * evd_handle + * ia + * evd * psp_flags * * Output: * conn_qual - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -61,9 +61,9 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ struct dapl_ia *ia_ptr; @@ -72,19 +72,19 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia u32 status = DAT_SUCCESS; int i; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } @@ -93,7 +93,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -117,10 +117,10 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; - sp_ptr->evd_handle = evd_handle; + sp_ptr->ia = ia; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -153,7 +153,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia if (status != DAT_SUCCESS) { atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -168,7 +168,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia * Return handle to the user */ *conn_qual = sp_ptr->conn_qual; - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -183,13 +183,13 @@ bail: * in a notification event. * * Input: - * ia_handle + * ia * conn_qual - * evd_handle + * evd * psp_flags * * Output: - * psp_handle + * psp * * Returns: * DAT_SUCCESS @@ -198,9 +198,9 @@ bail: * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) +u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -208,24 +208,24 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (psp_handle == NULL) { + if (psp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -261,11 +261,11 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Fill out the args for a PSP */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Take a reference on the EVD handle @@ -298,7 +298,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han * wrong! Decrements the EVD refcount & release it. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -313,7 +313,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + *psp = (struct dat_sp *) sp_ptr; bail: return status; @@ -325,7 +325,7 @@ bail: * Destroy a specific instance of a Service Point. * * Input: - * psp_handle + * psp * * Output: * none @@ -334,26 +334,25 @@ bail: * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) +u32 dapl_psp_free(struct dat_sp *psp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; DAPL_SP_STATE save_state; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if it has been established * and there are no current connections in progress. @@ -361,18 +360,18 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * container until the last connection is disconnected, after * which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); sp_ptr->listening = FALSE; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -389,8 +388,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han sp_ptr->cr_list_count == 0) { save_state = sp_ptr->state; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -407,7 +406,8 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free: PSP PENDING\n"); } @@ -416,13 +416,12 @@ bail: return status; } -u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) +u32 dapl_psp_query(struct dat_sp *psp, struct dat_psp_param *psp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || - ((struct dapl_sp *)psp_handle)->listening != TRUE) { + if (!psp || ((struct dapl_sp *)psp)->listening != TRUE) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; } @@ -432,11 +431,11 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha goto bail; } - sp_ptr = (struct dapl_sp *)psp_handle; + sp_ptr = (struct dapl_sp *)psp; - psp_param->ia_handle = sp_ptr->ia_handle; + psp_param->ia = sp_ptr->ia; psp_param->conn_qual = sp_ptr->conn_qual; - psp_param->evd_handle = sp_ptr->evd_handle; + psp_param->evd = sp_ptr->evd; psp_param->psp_flags = sp_ptr->psp_flags; status = DAT_SUCCESS; Index: dat-provider/dapl_rsp.c =================================================================== --- dat-provider/dapl_rsp.c (revision 2562) +++ dat-provider/dapl_rsp.c (working copy) @@ -44,13 +44,13 @@ * event * * Input: - * ia_handle + * ia * conn_qual - * ep_handle - * evd_handle + * ep + * evd * * Output: - * rsp_handle + * rsp * * Returns: * DAT_SUCCESS @@ -59,9 +59,9 @@ * DAT_INVALID_STATE * DAT_CONN_QUAL_IN_USE */ -u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle) +u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; @@ -70,39 +70,39 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han boolean_t sp_found; u32 status = DAT_SUCCESS; - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free conn_qual: %x EP: %p\n", - conn_qual, ep_handle); + conn_qual, ep); - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); goto bail; } - if (rsp_handle == NULL) { + if (rsp == NULL) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; } - evd_ptr = (struct dapl_evd *)evd_handle; + evd_ptr = (struct dapl_evd *)evd; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -126,11 +126,11 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Fill out the RSP args */ - sp_ptr->ia_handle = ia_handle; + sp_ptr->ia = ia; sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; + sp_ptr->evd = evd; sp_ptr->psp_flags = 0; - sp_ptr->ep_handle = ep_handle; + sp_ptr->ep = ep; /* * Take a reference on the EVD handle @@ -162,7 +162,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han * the state to FREE, so we know the call failed. */ atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; sp_ptr->state = DAPL_SP_STATE_FREE; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); @@ -178,7 +178,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Return handle to the user */ - *rsp_handle = (DAT_RSP_HANDLE)sp_ptr; + *rsp = (struct dat_sp *)sp_ptr; bail: return status; @@ -190,7 +190,7 @@ bail: * Destroy a specific instance of a Reserved Service Point. * * Input: - * rsp_handle + * rsp * * Output: * none @@ -199,50 +199,49 @@ bail: * DAT_SUCCESS * DAT_INVALID_HANDLE */ -u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) +u32 dapl_rsp_free(struct dat_sp *rsp) { struct dapl_ia *ia_ptr; struct dapl_sp *sp_ptr; struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Verify handle */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp_handle); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_RSP)) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp); + if (!sp_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; + ia_ptr = sp_ptr->common.owner_ia; /* * Remove the connection listener if there are no connections. If * we defer removing the sp it becomes something of a zombie * container until disconnection, after which it will be cleaned up. */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* * Make sure we don't leave a dangling EP. If the state is still * RESERVED then the RSP still owns it. */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep; if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - sp_ptr->ep_handle = NULL; + sp_ptr->ep = NULL; /* * Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL + * free the evd will be NULL */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> evd_ref_count); - sp_ptr->evd_handle = NULL; + sp_ptr->evd = NULL; } /* @@ -258,8 +257,8 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han /* This RSP has never been used. Clean it up */ sp_ptr->listening = FALSE; sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); if (status != DAT_SUCCESS) { @@ -275,20 +274,20 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han * ServiceID again, which will reactivate it. */ sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); } bail: return status; } -u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) +u32 dapl_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *rsp_param) { struct dapl_sp *sp_ptr; u32 status; - if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { + if (!rsp) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); goto bail; } @@ -298,15 +297,15 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha goto bail; } - sp_ptr = (struct dapl_sp *)rsp_handle; + sp_ptr = (struct dapl_sp *)rsp; /* * Fill in the RSP params */ - rsp_param->ia_handle = sp_ptr->ia_handle; + rsp_param->ia = sp_ptr->ia; rsp_param->conn_qual = sp_ptr->conn_qual; - rsp_param->evd_handle = sp_ptr->evd_handle; - rsp_param->ep_handle = sp_ptr->ep_handle; + rsp_param->evd = sp_ptr->evd; + rsp_param->ep = sp_ptr->ep; status = DAT_SUCCESS; Index: dat-provider/dapl_ep.c =================================================================== --- dat-provider/dapl_ep.c (revision 2571) +++ dat-provider/dapl_ep.c (working copy) @@ -43,11 +43,6 @@ static void dapl_ep_dealloc(struct dapl_ep *ep_ptr) { - dapl_os_assert(ep_ptr->header.magic == DAPL_MAGIC_EP); - - /* reset magic to prevent reuse */ - ep_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_cb_free(&ep_ptr->req_buffer); dapl_cb_free(&ep_ptr->recv_buffer); @@ -73,18 +68,10 @@ struct dapl_ep *dapl_ep_alloc(struct dap /* zero the structure */ memset(ep_ptr, 0, sizeof *ep_ptr + sizeof (struct sockaddr)); - /* - * initialize the header - */ - ep_ptr->header.provider = ia_ptr->header.provider; - ep_ptr->header.magic = DAPL_MAGIC_EP; - ep_ptr->header.handle_type = DAT_HANDLE_TYPE_EP; - ep_ptr->header.owner_ia = ia_ptr; - ep_ptr->header.user_context.as_64 = 0; - ep_ptr->header.user_context.as_ptr = NULL; - - dapl_llist_init_entry(&ep_ptr->header.ia_list_entry); - spin_lock_init(&ep_ptr->header.lock); + ep_ptr->ep.common.provider = ia_ptr->ia.common.provider; + ep_ptr->common.owner_ia = ia_ptr; + dapl_llist_init_entry(&ep_ptr->common.ia_list_entry); + spin_lock_init(&ep_ptr->common.lock); /* * Initialize the body @@ -105,7 +92,7 @@ struct dapl_ep *dapl_ep_alloc(struct dap ep_ptr->qp = NULL; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - ep_ptr->cm_handle = NULL; + ep_ptr->cm = NULL; atomic_set(&ep_ptr->req_count, 0); atomic_set(&ep_ptr->recv_count, 0); @@ -176,7 +163,7 @@ static u32 dapl_ep_check_recv_completion return DAT_SUCCESS; } -static u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, +static u32 dapl_ep_post_send_req(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -188,13 +175,13 @@ static u32 dapl_ep_post_send_req(DAT_EP_ struct dapl_cookie *cookie; u32 dat_status; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Synchronization ok since this buffer is only used for send @@ -292,11 +279,11 @@ enum dat_return_subtype dapl_ep_state_su return dat_status; } -u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) +u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + const struct dat_ep_attr *ep_attr, struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -305,18 +292,18 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create (%p, %p, %p, %p, %p, %p, %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -328,44 +315,34 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != DAT_HANDLE_NULL && - DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -381,12 +358,12 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -424,25 +401,25 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (recv_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_recv_dtos = 0; - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) + if (request_evd == NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_request_dtos = 0; /* @@ -451,9 +428,9 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -461,7 +438,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -471,36 +448,36 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free - * evd_handles: + * pz: dat_pz_free + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = (struct dat_ep *)ep_ptr; bail: return dat_status; } -u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +u32 dapl_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *ep_attr, - DAT_EP_HANDLE *ep_handle) + struct dat_ep **ep) { struct dapl_ia *ia_ptr; struct dapl_ep *ep_ptr; @@ -509,18 +486,18 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create_with_srq (%p, %p, %p, %p, %p, %p, %p %p)\n", - ia_handle, - pz_handle, - recv_evd_handle, - request_evd_handle, - connect_evd_handle, srq_handle, ep_attr, ep_handle); + ia, + pz, + recv_evd, + request_evd, + connect_evd, srq, ep_attr, ep); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; /* * Verify parameters */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; @@ -532,35 +509,27 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * modified by dat_ep_modify() is not strictly * required when the EP is created */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } /* If connect handle is !NULL verify handle is good */ - if (connect_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG))) { + if (connect_evd != NULL && + !(((struct dapl_evd *)connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; } /* If recv_evd is !NULL, verify handle is good and flags are valid */ - if (recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + if (recv_evd != NULL && + !(((struct dapl_evd *)recv_evd)->evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; } /* If req_evd is !NULL, verify handle is good and flags are valid */ - if (request_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((struct dapl_evd *)request_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG))) { + if (request_evd != NULL && + !(((struct dapl_evd *)request_evd)-> + evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_REQUEST); @@ -571,13 +540,13 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * Verify the SRQ handle. It is an error to invoke this call with * a NULL handle */ - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + if (!srq) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); goto bail; } - if (ep_handle == NULL) { + if (ep == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); goto bail; } @@ -593,12 +562,12 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * the respective queue. */ if (ep_attr != NULL && - ((recv_evd_handle == DAT_HANDLE_NULL && ep_attr->max_recv_dtos != 0) - || (recv_evd_handle != DAT_HANDLE_NULL + ((recv_evd == NULL && ep_attr->max_recv_dtos != 0) + || (recv_evd != NULL && ep_attr->max_recv_dtos == 0) - || (request_evd_handle == DAT_HANDLE_NULL + || (request_evd == NULL && ep_attr->max_request_dtos != 0) - || (request_evd_handle != DAT_HANDLE_NULL + || (request_evd != NULL && ep_attr->max_request_dtos == 0) || ep_attr->max_recv_iov == 0 || ep_attr->max_request_iov == 0 || (DAT_SUCCESS != @@ -647,26 +616,26 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Fill in the EP */ - ep_ptr->param.ia_handle = ia_handle; + ep_ptr->param.ia = ia; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); - ep_ptr->param.pz_handle = pz_handle; - ep_ptr->param.recv_evd_handle = recv_evd_handle; - ep_ptr->param.request_evd_handle = request_evd_handle; - ep_ptr->param.connect_evd_handle = connect_evd_handle; + ep_ptr->param.pz = pz; + ep_ptr->param.recv_evd = recv_evd; + ep_ptr->param.request_evd = request_evd; + ep_ptr->param.connect_evd = connect_evd; /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (recv_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_recv_dtos = 0; } - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (request_evd == NULL && ep_attr == NULL) { ep_ptr->param.ep_attr.max_request_dtos = 0; } @@ -676,9 +645,9 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * As much as possible we try to keep QP creation out of the * connect path to avoid resource errors in strange places. */ - if (pz_handle != DAT_HANDLE_NULL) { + if (pz != NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz)->pz_ref_count); /* * Get a QP from the IB provider @@ -686,7 +655,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } @@ -696,24 +665,24 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* * Update ref counts. See the spec where the endpoint marks * a data object as 'in use' - * pz_handle: dat_pz_free, - * evd_handles: + * pz: dat_pz_free, + * evds: * * N.B. This should really be done by a util routine. */ - atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> + if (recv_evd != NULL) + atomic_inc(&((struct dapl_evd *)recv_evd)-> evd_ref_count); - if (request_evd_handle != DAT_HANDLE_NULL) - atomic_inc(&((struct dapl_evd *)request_evd_handle)-> + if (request_evd != NULL) + atomic_inc(&((struct dapl_evd *)request_evd)-> evd_ref_count); /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); - *ep_handle = ep_ptr; + *ep = ep_ptr; #endif /* notdef */ @@ -721,7 +690,7 @@ bail: return dat_status; } -u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_free(struct dat_ep *ep) { struct dapl_ep *ep_ptr; struct dapl_ia *ia_ptr; @@ -729,15 +698,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle enum ib_qp_state save_qp_state; u32 dat_status = DAT_SUCCESS; - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; param = &ep_ptr->param; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -755,7 +724,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle goto bail; } - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* If we are connected, issue a disconnect. If we are in the * disconnect_pending state, disconnect with the ABRUPT flag @@ -765,14 +734,15 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle /* * Invoke ep_disconnect to clean up outstanding connections */ - (void)dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + (void)dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); dapl_os_assert(ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_UNCONNECTED); /* * Do verification of parameters and the state change atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); @@ -782,24 +752,24 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle * to set handles to NULL. Set handles to NULL so this routine * is idempotent. */ - if (param->pz_handle != NULL) { - atomic_dec(&((struct dapl_pz *)param->pz_handle)->pz_ref_count); - param->pz_handle = NULL; + if (param->pz != NULL) { + atomic_dec(&((struct dapl_pz *)param->pz)->pz_ref_count); + param->pz = NULL; } - if (param->recv_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->recv_evd_handle)-> + if (param->recv_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->recv_evd)-> evd_ref_count); - param->recv_evd_handle = NULL; + param->recv_evd = NULL; } - if (param->request_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->request_evd_handle)-> + if (param->request_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->request_evd)-> evd_ref_count); - param->request_evd_handle = NULL; + param->request_evd = NULL; } - if (param->connect_evd_handle != NULL) { - atomic_dec(&((struct dapl_evd *)param->connect_evd_handle)-> + if (param->connect_evd != NULL) { + atomic_dec(&((struct dapl_evd *)param->connect_evd)-> evd_ref_count); - param->connect_evd_handle = NULL; + param->connect_evd = NULL; } /* @@ -815,7 +785,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ save_qp_state = ep_ptr->qp_state; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Free the QP. If the EP has never been used, the QP is invalid */ if (save_qp_state != DAPL_QP_STATE_UNATTACHED) { @@ -839,7 +809,7 @@ bail: } -u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, @@ -850,7 +820,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_connect(%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", - ep_handle, + ep, remote_ia_address->sa_data[2], remote_ia_address->sa_data[3], remote_ia_address->sa_data[4], @@ -858,19 +828,19 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han remote_conn_qual, timeout, private_data_size, private_data, qos, connect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state. The connection handle must be good * at this point. */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - if (DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD)) { + if (!ep_ptr->param.connect_evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); goto bail; @@ -900,9 +870,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * * Assign the QP and release the lock. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ)) - { + if (ep_ptr->param.pz == NULL) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); @@ -910,7 +878,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } alloc_ep = *ep_ptr; - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, &alloc_ep, ep_ptr); if (dat_status != DAT_SUCCESS) { dat_status = @@ -919,15 +887,15 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han goto bail; } - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * PZ shouldn't have changed since we're only racing with * dapl_cr_accept() */ if (ep_ptr->qp_state != DAPL_QP_STATE_UNATTACHED) { /* Bail, cleaning up. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = dapl_ib_qp_free(&alloc_ep); if (dat_status != DAT_SUCCESS) { dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -943,42 +911,30 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->qp = alloc_ep.qp; ep_ptr->qp_state = alloc_ep.qp_state; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); } /* * We do state checks and transitions under lock. * The only code we're racing against is dapl_cr_accept. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* * Verify the attributes of the EP handle before we connect it. Test * all of the handles to make sure they are currently valid. * Specifically: - * pz_handle required - * recv_evd_handle optional, but must be valid - * request_evd_handle optional, but must be valid - * connect_evd_handle required - */ - if (ep_ptr->param.pz_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.pz_handle, DAPL_MAGIC_PZ) - /* test connect handle */ - || ep_ptr->param.connect_evd_handle == NULL - || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> - evd_flags & DAT_EVD_CONNECTION_FLAG) - /* test optional completion handles */ - || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && - (DAPL_BAD_HANDLE - (ep_ptr->param.recv_evd_handle, DAPL_MAGIC_EVD))) - || (ep_ptr->param.request_evd_handle != DAT_HANDLE_NULL - && - (DAPL_BAD_HANDLE - (ep_ptr->param.request_evd_handle, DAPL_MAGIC_EVD)))) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + * pz required + * recv_evd optional, but must be valid + * request_evd optional, but must be valid + * connect_evd required + */ + if (ep_ptr->param.pz == NULL || ep_ptr->param.connect_evd == NULL + || !(((struct dapl_evd *)ep_ptr->param.connect_evd)-> + evd_flags & DAT_EVD_CONNECTION_FLAG)) { + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EP_NOTREADY); goto bail; @@ -988,12 +944,12 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * we need to attach one now. */ if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { - dat_status = dapl_ib_qp_alloc(ep_ptr->header.owner_ia, + dat_status = dapl_ib_qp_alloc(ep_ptr->common.owner_ia, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_TEP); @@ -1002,8 +958,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han } if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1014,8 +970,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han /* * At this point we only support one QOS level */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_MODEL_NOT_SUPPORTED, 0); goto bail; } @@ -1029,7 +985,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * At this point we're committed, and done with the endpoint * except for the connect, so we can drop the lock. */ - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); /* Copy the connection qualifiers */ memcpy(ep_ptr->param.remote_ia_address_ptr, remote_ia_address, @@ -1037,7 +993,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han ep_ptr->param.remote_port_qual = remote_conn_qual; memcpy(ep_ptr->private_data,private_data,private_data_size); - dat_status = dapl_ib_connect(ep_ptr, remote_ia_address, + dat_status = dapl_ib_connect((struct dat_ep *)ep_ptr, remote_ia_address, remote_conn_qual, timeout, private_data_size, ep_ptr->private_data); @@ -1054,10 +1010,10 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han if (dat_status == DAT_ERROR(DAT_INTERNAL_ERROR, 1)) { dapl_evd_post_connection_event((struct dapl_evd *) ep_ptr->param. - connect_evd_handle, + connect_evd, DAT_CONNECTION_EVENT_UNREACHABLE, - (DAT_HANDLE) ep_ptr, 0, - NULL); + (struct dat_ep *)ep_ptr, + 0, NULL); dat_status = DAT_SUCCESS; } } @@ -1069,7 +1025,7 @@ bail: return dat_status; } -u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, +u32 dapl_ep_dup_connect(struct dat_ep *ep, struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { @@ -1078,13 +1034,13 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep DAT_CONN_QUAL remote_conn_qual; u32 dat_status; - ep_dup_ptr = (struct dapl_ep *)ep_dup_handle; + ep_dup_ptr = (struct dapl_ep *)ep_dup; /* * Verify the dup handle, which must be connected. All other * parameters will be verified by dapl_ep_connect */ - if (DAPL_BAD_HANDLE(ep_dup_handle, DAPL_MAGIC_EP)) { + if (!ep_dup) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1100,10 +1056,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep * there is a problem. Do this under a lock and pull out * the connection parameters for atomicity. */ - spin_lock_irqsave(&ep_dup_ptr->header.lock, ep_dup_ptr->header.flags); + spin_lock_irqsave(&ep_dup_ptr->common.lock, ep_dup_ptr->common.flags); if (ep_dup_ptr->param.ep_state != DAT_EP_STATE_CONNECTED) { - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_dup_ptr)); @@ -1111,10 +1067,10 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep } remote_ia_address_ptr = ep_dup_ptr->param.remote_ia_address_ptr; remote_conn_qual = ep_dup_ptr->param.remote_port_qual; - spin_unlock_irqrestore(&ep_dup_ptr->header.lock, - ep_dup_ptr->header.flags); + spin_unlock_irqrestore(&ep_dup_ptr->common.lock, + ep_dup_ptr->common.flags); - dat_status = dapl_ep_connect(ep_handle, + dat_status = dapl_ep_connect(ep, remote_ia_address_ptr, remote_conn_qual, timeout, @@ -1125,7 +1081,7 @@ bail: return dat_status; } -u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags disconnect_flags) { struct dapl_ep *ep_ptr; @@ -1135,14 +1091,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_disconnect (%p, %x)\n", - ep_handle, disconnect_flags); + ep, disconnect_flags); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1152,12 +1108,12 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * Do the verification of parameters and the state change * atomically. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); /* Disconnecting a disconnected EP is a no-op. */ if (ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_SUCCESS; goto bail; } @@ -1171,8 +1127,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ ep_ptr->param.ep_state != DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_COMPLETION_PENDING && ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECT_PENDING) { - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; @@ -1184,8 +1140,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * If in state DISCONNECT_PENDING then this must be an * ABRUPT disconnect */ - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } @@ -1198,18 +1154,19 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ */ ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); /* disconnect and make sure we get no callbacks */ (void)dapl_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); /* clean up connection state */ dapl_sp_remove_ep(ep_ptr); - evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd; dapl_evd_post_connection_event(evd_ptr, DAT_CONNECTION_EVENT_DISCONNECTED, - (DAT_HANDLE) ep_ptr, 0, NULL); + (struct dat_ep *)ep_ptr, 0, + NULL); dat_status = DAT_SUCCESS; goto bail; } @@ -1227,14 +1184,14 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECT_PENDING; } - spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, ep_ptr->common.flags); dat_status = dapl_ib_disconnect(ep_ptr, disconnect_flags); /* * Reacquire the lock and make sure we didn't get a callback * that cleaned up. */ - spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); + spin_lock_irqsave(&ep_ptr->common.lock, ep_ptr->common.flags); if (disconnect_flags == DAT_CLOSE_ABRUPT_FLAG && ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { /* @@ -1245,8 +1202,8 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ * on the way out. */ cr_ptr = ep_ptr->cr_ptr; - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); if (cr_ptr != NULL) { dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, @@ -1257,13 +1214,13 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ DAT_CONNECTION_EVENT_DISCONNECTED, NULL, cr_ptr->sp_ptr); } else { - dapl_evd_connection_callback(ep_ptr->cm_handle, + dapl_evd_connection_callback(ep_ptr->cm, DAT_CONNECTION_EVENT_DISCONNECTED, NULL, ep_ptr); } } else - spin_unlock_irqrestore(&ep_ptr->header.lock, - ep_ptr->header.flags); + spin_unlock_irqrestore(&ep_ptr->common.lock, + ep_ptr->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, @@ -1272,7 +1229,7 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ return dat_status; } -u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, enum dat_ep_state *ep_state, +u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle) { struct dapl_ep *ep_ptr; @@ -1280,14 +1237,14 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_get_status (%p, %p, %p, %p)\n", - ep_handle, ep_state, in_dto_idle, out_dto_idle); + ep, ep_state, in_dto_idle, out_dto_idle); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -1315,15 +1272,15 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_ } static inline u32 -dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, +dapl_ep_modify_validate_parameters(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param, struct dapl_ia **ia_ptr, struct dapl_ep **ep_ptr, struct dat_ep_attr *ep_attr_ptr) { - struct dapl_ia *ia; - struct dapl_ep *ep; + struct dapl_ia *dapl_ia; + struct dapl_ep *dapl_ep; struct dat_ep_attr ep_attr; struct dat_ep_attr ep_attr_limit; struct dat_ep_attr ep_attr_request; @@ -1332,14 +1289,14 @@ dapl_ep_modify_validate_parameters(DAT_E *ia_ptr = NULL; *ep_ptr = NULL; - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep = (struct dapl_ep *)ep_handle; - ia = ep->header.owner_ia; + dapl_ep = (struct dapl_ep *)ep; + dapl_ia = dapl_ep->common.owner_ia; /* * Verify parameters valid in current EP state @@ -1359,10 +1316,10 @@ dapl_ep_modify_validate_parameters(DAT_E * TENTATIVE_CONNECTION_PENDING (psp PROVIDER allocated EP) */ if ((mask & DAT_EP_FIELD_PZ_HANDLE) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } @@ -1378,41 +1335,21 @@ dapl_ep_modify_validate_parameters(DAT_E DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS | DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV | DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV)) && - (ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && - ep->param.ep_state != DAT_EP_STATE_RESERVED && - ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep)); + (dapl_ep->param.ep_state != DAT_EP_STATE_UNCONNECTED && + dapl_ep->param.ep_state != DAT_EP_STATE_RESERVED && + dapl_ep->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING)) { + dat_status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(dapl_ep)); goto bail; } /* * Validate handles being modified */ - if (mask & DAT_EP_FIELD_PZ_HANDLE) { - if (ep_param->pz_handle != NULL && - DAPL_BAD_HANDLE(ep_param->pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { - if (ep_param->recv_evd_handle != NULL && - (DAPL_BAD_HANDLE(ep_param->recv_evd_handle, DAPL_MAGIC_EVD) - || !((struct dapl_evd *)ep_param->recv_evd_handle)-> - evd_flags & DAT_EVD_DTO_FLAG)) { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - } - - if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) { - if (ep_param->request_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->request_evd_handle, - DAPL_MAGIC_EVD)) { + if (ep_param->recv_evd != NULL && + !((struct dapl_evd *)ep_param->recv_evd)-> + evd_flags & DAT_EVD_DTO_FLAG) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; @@ -1420,10 +1357,8 @@ dapl_ep_modify_validate_parameters(DAT_E } if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) { - if (ep_param->connect_evd_handle != NULL && - DAPL_BAD_HANDLE(ep_param->connect_evd_handle, - DAPL_MAGIC_EVD) - && !(((struct dapl_evd *)ep_param->connect_evd_handle)-> + if (ep_param->connect_evd != NULL && + !(((struct dapl_evd *)ep_param->connect_evd)-> evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1434,8 +1369,8 @@ dapl_ep_modify_validate_parameters(DAT_E /* * Validate the attributes against the HCA limits */ - ep_attr = ep->param.ep_attr; - ep_attr_limit = ia->hca_ptr->ep_attr; + ep_attr = dapl_ep->param.ep_attr; + ep_attr_limit = dapl_ia->hca_ptr->ep_attr; ep_attr_request = ep_param->ep_attr; if (mask & DAT_EP_FIELD_EP_ATTR_MAX_MESSAGE_SIZE) { @@ -1482,7 +1417,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS) { if (ep_attr_request.max_recv_dtos > ep_attr_limit.max_recv_dtos - || (ep_param->recv_evd_handle == DAT_HANDLE_NULL + || (ep_param->recv_evd == NULL && (ep_attr_request.max_recv_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1495,7 +1430,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS) { if (ep_attr_request.max_request_dtos > ep_attr_limit.max_request_dtos - || (ep_param->request_evd_handle == DAT_HANDLE_NULL + || (ep_param->request_evd == NULL && (ep_attr_request.max_request_dtos > 0))) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1552,15 +1487,15 @@ dapl_ep_modify_validate_parameters(DAT_E } } - *ia_ptr = ia; - *ep_ptr = ep; + *ia_ptr = dapl_ia; + *ep_ptr = dapl_ep; *ep_attr_ptr = ep_attr; bail: return dat_status; } -u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, +u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param) { struct dapl_ia *ia; @@ -1586,7 +1521,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand boolean_t rqst_iov_used = FALSE; boolean_t recv_iov_used = FALSE; - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep1, &ep_attr1); @@ -1600,16 +1535,16 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep = *ep1; alloc_ep.param.ep_attr = ep_attr1; if (mask & DAT_EP_FIELD_PZ_HANDLE) - alloc_ep.param.pz_handle = ep_param->pz_handle; + alloc_ep.param.pz = ep_param->pz; if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) - alloc_ep.param.recv_evd_handle = ep_param->recv_evd_handle; + alloc_ep.param.recv_evd = ep_param->recv_evd; if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) - alloc_ep.param.request_evd_handle = ep_param->request_evd_handle; + alloc_ep.param.request_evd = ep_param->request_evd; if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) - alloc_ep.param.connect_evd_handle = ep_param->connect_evd_handle; + alloc_ep.param.connect_evd = ep_param->connect_evd; /* * Allocate everything that might be needed. @@ -1673,18 +1608,18 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * There's a lot of work done here, but there should be no * allocation or blocking. */ - spin_lock_irqsave(&ep1->header.lock, ep1->header.flags); + spin_lock_irqsave(&ep1->common.lock, ep1->common.flags); /* * Revalidate parameters; make sure that races haven't * changed anything important. */ - dat_status = dapl_ep_modify_validate_parameters(ep_handle, + dat_status = dapl_ep_modify_validate_parameters(ep, mask, ep_param, &ia, &ep2, &ep_attr2); if (DAT_SUCCESS != dat_status) { - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); goto bail; } @@ -1713,10 +1648,10 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * is stuff changed by dapl_cr_accept, and neither PZ nor EVD is in that * list. */ - new_ep.param.pz_handle = alloc_ep.param.pz_handle; - new_ep.param.recv_evd_handle = alloc_ep.param.recv_evd_handle; - new_ep.param.request_evd_handle = alloc_ep.param.request_evd_handle; - new_ep.param.connect_evd_handle = alloc_ep.param.connect_evd_handle; + new_ep.param.pz = alloc_ep.param.pz; + new_ep.param.recv_evd = alloc_ep.param.recv_evd; + new_ep.param.request_evd = alloc_ep.param.request_evd; + new_ep.param.connect_evd = alloc_ep.param.connect_evd; /* Deal with each of the allocation fields. */ if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_DTOS @@ -1778,50 +1713,50 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand if (DAPL_QP_STATE_UNATTACHED != new_ep.qp_state) { dat_status = dapl_ib_qp_modify(ia, ep2, &ep_attr2); if (dat_status != DAT_SUCCESS) { - spin_unlock_irqrestore(&ep2->header.lock, - ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, + ep2->common.flags); goto bail; } } *ep2 = new_ep; - spin_unlock_irqrestore(&ep2->header.lock, ep2->header.flags); + spin_unlock_irqrestore(&ep2->common.lock, ep2->common.flags); /* * Modify reference counts, incrementing new ones * and then decrementing old ones (so if they're the same * the refcount never drops to zero). */ - tmp_pz = (struct dapl_pz *)new_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)new_ep.param.pz; if (tmp_pz) atomic_inc(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.request_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd; if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); /* decreament the old reference counts */ - tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz_handle; + tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz; if (tmp_pz) atomic_dec(&tmp_pz->pz_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd_handle; + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd; if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); @@ -1869,7 +1804,7 @@ bail: return dat_status; } -u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1880,12 +1815,12 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_read (%p, %d, %p, %p, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1900,7 +1835,7 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE return dat_status; } -u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -1910,12 +1845,12 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_write (%p, %d, %p, %P, %p, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, remote_iov, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -1930,7 +1865,7 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL return dat_status; } -u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1941,15 +1876,15 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_recv (%p, %d, %p, %P, %x)\n", - ep_handle, num_segments, local_iov, user_cookie.as_64, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + if (!ep) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } - ep_ptr = (struct dapl_ep *) ep_handle; + ep_ptr = (struct dapl_ep *) ep; /* * Synchronization ok since this buffer is only used for receive @@ -1979,7 +1914,7 @@ bail: return status; } -u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) @@ -1989,11 +1924,11 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_send (%p, %d, %p, %P, %x)\n", - ep_handle, + ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - dat_status = dapl_ep_post_send_req(ep_handle, + dat_status = dapl_ep_post_send_req(ep, num_segments, local_iov, user_cookie, @@ -2007,21 +1942,21 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_h return dat_status; } -u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) +u32 dapl_ep_query(struct dat_ep *ep, struct dat_ep_param *ep_param) { struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_query (%p, %p)\n", - ep_handle, ep_param); + ep, ep_param); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } @@ -2037,12 +1972,10 @@ u32 dapl_ep_query(DAT_EP_HANDLE ep_handl * N.B. Just slam all values into the user structure, there * is nothing to be gained by checking for each bit. */ - if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) { + if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED) /* obtain the remote IP address */ - status = - dapl_ib_cm_remote_addr((DAT_HANDLE) ep_handle, - &ep_ptr->remote_ia_address); - } + status = dapl_ib_cm_remote_addr(ep, &ep_ptr->remote_ia_address); + ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *)&ep_ptr->remote_ia_address; *ep_param = ep_ptr->param; @@ -2051,21 +1984,21 @@ bail: return status; } -u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, int *nbufs_allocate, +u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocate, int *bufs_alloc_span) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", - ep_handle, nbufs_allocate, bufs_alloc_span); + ep, nbufs_allocate, bufs_alloc_span); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2078,17 +2011,17 @@ bail: } -u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_reset(struct dat_ep *ep) { struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; @@ -2110,7 +2043,7 @@ bail: return dat_status; } -u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -2118,14 +2051,14 @@ u32 dapl_ep_set_watermark(DAT_EP_HANDLE u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", - ep_handle, soft_high_watermark, hard_high_watermark); + ep, soft_high_watermark, hard_high_watermark); - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; /* * Verify parameter & state */ - if (DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP)) { + if (!ep_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; Index: dat-provider/dapl_ia_open.c =================================================================== --- dat-provider/dapl_ia_open.c (revision 2563) +++ dat-provider/dapl_ia_open.c (working copy) @@ -45,18 +45,18 @@ * and calls this function. */ u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr) + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr) { u32 dat_status = DAT_SUCCESS; struct dat_provider *provider; struct dapl_hca *hca_ptr = NULL; - struct dapl_ia *ia_ptr = NULL; - struct dapl_evd *evd_ptr; + struct dapl_ia *ia = NULL; + struct dapl_evd *evd; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_open (%s, %d, %p, %p)\n", - name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); + name, async_evd_qlen, async_evd_ptr, ia_ptr); dat_status = dapl_provider_list_search(name, &provider); if (DAT_SUCCESS != dat_status) { @@ -64,25 +64,25 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - /* ia_handle_ptr and async_evd_handle_ptr cannot be NULL */ - if (ia_handle_ptr == NULL) { + /* ia_ptr and async_evd_ptr cannot be NULL */ + if (ia_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } - if (async_evd_handle_ptr == NULL) { + if (async_evd_ptr == NULL) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); goto bail; } /* initialize the caller's param */ - *ia_handle_ptr = DAT_HANDLE_NULL; + *ia_ptr = NULL; hca_ptr = (struct dapl_hca *)provider->extension; atomic_inc(&hca_ptr->handle_ref_count); /* Allocate and initialize ia structure */ - ia_ptr = dapl_ia_alloc(provider, hca_ptr); - if (!ia_ptr) { + ia = dapl_ia_alloc(provider, hca_ptr); + if (!ia) { atomic_dec(&hca_ptr->handle_ref_count); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -93,17 +93,16 @@ u32 dapl_ia_open(const char *name, int a * use the one passed in (if non-NULL) or create one */ - evd_ptr = (struct dapl_evd *)*async_evd_handle_ptr; - if (evd_ptr) { - if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || - !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { + evd = (struct dapl_evd *)*async_evd_ptr; + if (evd) { + if (!(evd->evd_flags & DAT_EVD_ASYNC_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - if (evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle != + if (evd->common.owner_ia->hca_ptr->ib_hca_handle != hca_ptr->ib_hca_handle) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, @@ -111,8 +110,8 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - ia_ptr->cleanup_async_error_evd = FALSE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = FALSE; + ia->async_error_evd = evd; } else { /* Verify we have >0 length, and let the provider check the size */ if (async_evd_qlen <= 0) { @@ -120,32 +119,33 @@ u32 dapl_ia_open(const char *name, int a DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } - dat_status = dapl_evd_internal_create(ia_ptr, async_evd_qlen, + dat_status = dapl_evd_internal_create(ia, + async_evd_qlen, DAT_UPCALL_DISABLE, NULL, DAT_EVD_ASYNC_FLAG, - &evd_ptr); + &evd); if (dat_status != DAT_SUCCESS) goto bail; - atomic_inc(&evd_ptr->evd_ref_count); + atomic_inc(&evd->evd_ref_count); /* Register the handlers associated with the async EVD. */ - dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); + dat_status = dapl_ia_setup_callbacks(ia, evd); /* Assign the EVD so it gets cleaned up */ - ia_ptr->cleanup_async_error_evd = TRUE; - ia_ptr->async_error_evd = evd_ptr; + ia->cleanup_async_error_evd = TRUE; + ia->async_error_evd = evd; if (dat_status != DAT_SUCCESS) goto bail; } dat_status = DAT_SUCCESS; - *ia_handle_ptr = ia_ptr; - *async_evd_handle_ptr = evd_ptr; + *ia_ptr = (struct dat_ia *)ia; + *async_evd_ptr = (struct dat_evd *)evd; - bail: - if (dat_status != DAT_SUCCESS) { - if (ia_ptr) - dapl_ia_close(ia_ptr, DAT_CLOSE_ABRUPT_FLAG); - } +bail: + if (dat_status != DAT_SUCCESS) + if (ia) + dapl_ia_close((struct dat_ia *)ia, + DAT_CLOSE_ABRUPT_FLAG); dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_ia_open () returns 0x%x\n", dat_status); Index: dat-provider/dapl_mr_util.c =================================================================== --- dat-provider/dapl_mr_util.c (revision 2562) +++ dat-provider/dapl_mr_util.c (working copy) @@ -59,7 +59,7 @@ u64 dapl_mr_get_address(DAT_REGION_DESCR case DAT_MEM_TYPE_VIRTUAL: return (u64) (unsigned long) desc.for_va; case DAT_MEM_TYPE_LMR: - lmr = (struct dapl_lmr *)desc.for_lmr_handle; + lmr = (struct dapl_lmr *)desc.for_lmr; /* Since this function is recoursive we cannot inline it */ return dapl_mr_get_address(lmr->param.region_desc, Index: dat-provider/dapl_evd.c =================================================================== --- dat-provider/dapl_evd.c (revision 2563) +++ dat-provider/dapl_evd.c (working copy) @@ -51,7 +51,7 @@ static void dapl_evd_upcall_trigger(stru } for (;;) { - status = dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); + status = dapl_evd_dequeue((struct dat_evd *)evd, &event); if (DAT_SUCCESS != status) return; @@ -157,17 +157,10 @@ struct dapl_evd *dapl_evd_alloc(struct d /* zero the structure */ memset(evd, 0, sizeof *evd); - /* - * initialize the header - */ - evd->header.provider = ia->header.provider; - evd->header.magic = DAPL_MAGIC_EVD; - evd->header.handle_type = DAT_HANDLE_TYPE_EVD; - evd->header.owner_ia = ia; - evd->header.user_context.as_64 = 0; - evd->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&evd->header.ia_list_entry); - spin_lock_init(&evd->header.lock); + evd->evd.common.provider = ia->ia.common.provider; + evd->common.owner_ia = ia; + dapl_llist_init_entry(&evd->common.ia_list_entry); + spin_lock_init(&evd->common.lock); /* * Initialize the body @@ -249,7 +242,6 @@ static u32 dapl_evd_dealloc(struct dapl_ u32 status = DAT_SUCCESS; struct dapl_ia *ia; - dapl_os_assert(evd->header.magic == DAPL_MAGIC_EVD); dapl_os_assert(atomic_read(&evd->evd_ref_count) == 0); /* @@ -257,7 +249,7 @@ static u32 dapl_evd_dealloc(struct dapl_ * up from it. */ if (evd->cq != NULL) { - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; status = dapl_ib_cq_free(evd); if (status != DAT_SUCCESS) { @@ -265,12 +257,6 @@ static u32 dapl_evd_dealloc(struct dapl_ } } - /* - * We should now be safe to invalidate the EVD; reset the - * magic to prevent reuse. - */ - evd->header.magic = DAPL_MAGIC_INVALID; - /* If the ring buffer allocation failed, then the dapl_rbuf_destroy */ /* function will detect that the ring buffer's internal data (ex. base */ /* pointer) are invalid and will handle the situation appropriately */ @@ -314,14 +300,14 @@ static struct dat_event *dapl_evd_get_ev struct dat_event *event; if (evd->evd_producer_locking_needed) { - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); } event = (struct dat_event *) dapl_rbuf_remove(&evd->free_event_queue); /* Release the lock if it was taken and the call failed. */ if (!event && evd->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); return event; } @@ -348,8 +334,8 @@ static void dapl_evd_post_event(struct d dapl_os_assert(evd->evd_state == DAPL_EVD_STATE_OPEN); if (evd->evd_producer_locking_needed) - spin_unlock_irqrestore(&evd->header.lock, - evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, + evd->common.flags); dapl_evd_upcall_trigger(evd); } @@ -363,12 +349,12 @@ static void dapl_evd_format_overflow_eve { struct dapl_ia *ia; - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; event->event_number = DAT_ASYNC_ERROR_EVD_OVERFLOW; event->event_data.asynch_error_event_data.dat_handle = - (DAT_HANDLE) ia; + (struct dat_ia *)ia; } /* @@ -409,9 +395,9 @@ static struct dat_event *dapl_evd_get_an event = dapl_evd_get_event(evd); if (!event) dapl_evd_post_overflow_event( - evd->header.owner_ia->async_error_evd, evd); + evd->common.owner_ia->async_error_evd, evd); else { - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; event->event_number = evno; } @@ -419,10 +405,10 @@ static struct dat_event *dapl_evd_get_an } u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp, struct sockaddr *ia_address, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle) + struct dat_cr *cr) { struct dat_event *event; event = dapl_evd_get_and_init_event(evd, DAT_CONNECTION_REQUEST_EVENT); @@ -436,11 +422,11 @@ u32 dapl_evd_post_cr_arrival_event(struc return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event->event_data.cr_arrival_event_data.sp_handle = sp_handle; + event->event_data.cr_arrival_event_data.sp = sp; event->event_data.cr_arrival_event_data.local_ia_address_ptr = ia_address; event->event_data.cr_arrival_event_data.conn_qual = conn_qual; - event->event_data.cr_arrival_event_data.cr_handle = cr_handle; + event->event_data.cr_arrival_event_data.cr = cr; dapl_evd_post_event(evd, event); @@ -449,7 +435,7 @@ u32 dapl_evd_post_cr_arrival_event(struc u32 dapl_evd_post_connection_event(struct dapl_evd *evd, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, int private_data_size, void *private_data) { @@ -466,7 +452,7 @@ u32 dapl_evd_post_connection_event(struc DAT_RESOURCE_MEMORY); } - event->event_data.connect_event_data.ep_handle = ep_handle; + event->event_data.connect_event_data.ep = ep; event->event_data.connect_event_data.private_data_size = private_data_size; event->event_data.connect_event_data.private_data = private_data; @@ -492,7 +478,7 @@ static u32 dapl_evd_post_async_error_eve return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - event->event_data.asynch_error_event_data.dat_handle = (DAT_HANDLE) ia; + event->event_data.asynch_error_event_data.dat_handle = ia; dapl_evd_post_event(evd, event); @@ -539,12 +525,8 @@ static void dapl_evd_wc_to_event(struct ep = cookie->ep; dapl_os_assert(NULL != ep); - if (ep->header.magic != DAPL_MAGIC_EP) { - /* ep may have been freed, just return */ - return; - } - event->evd_handle = (DAT_EVD_HANDLE) evd; + event->evd = (struct dat_evd *)evd; switch (cookie->type) { case DAPL_COOKIE_TYPE_DTO: @@ -563,7 +545,7 @@ static void dapl_evd_wc_to_event(struct } event->event_number = DAT_DTO_COMPLETION_EVENT; - event_data->ep_handle = cookie->ep; + event_data->ep = (struct dat_ep *)cookie->ep; event_data->user_cookie = cookie->val.dto.cookie; event_data->status = dto_status; @@ -606,7 +588,7 @@ static void dapl_evd_wc_to_event(struct event->event_number = DAT_RMR_BIND_COMPLETION_EVENT; event_data = &event->event_data.rmr_completion_event_data; - event_data->rmr_handle = cookie->val.rmr.rmr; + event_data->rmr = (struct dat_rmr *)cookie->val.rmr.rmr; event_data->user_cookie = cookie->val.rmr.cookie; if (dto_status == DAT_DTO_SUCCESS) { @@ -654,7 +636,7 @@ void dapl_evd_qp_async_error_callback(st u32 status; ep = (struct dapl_ep *)context; - async_evd = (struct dapl_evd *)ep->header.owner_ia->async_error_evd; + async_evd = (struct dapl_evd *)ep->common.owner_ia->async_error_evd; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback: " @@ -679,7 +661,7 @@ void dapl_evd_qp_async_error_callback(st * we are not interested in. */ (void)dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_qp_async_error_callback () returns\n"); @@ -698,11 +680,11 @@ void dapl_evd_cq_async_error_callback(st if (!evd) dapl_os_panic("NULL == context\n"); - async_evd = evd->header.owner_ia->async_error_evd; + async_evd = evd->common.owner_ia->async_error_evd; status = dapl_evd_post_async_error_event(async_evd, DAT_ASYNC_ERROR_EVD_OVERFLOW, - async_evd->header.owner_ia); + async_evd->common.owner_ia); if (status != DAT_SUCCESS) { dapl_os_panic("async EVD overflow\n"); @@ -738,7 +720,7 @@ void dapl_evd_un_async_error_callback(st */ dapl_evd_post_async_error_event(async_evd, async_event, - async_evd->header.owner_ia); + async_evd->common.owner_ia); } dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_un_async_error_callback () returns\n"); @@ -764,14 +746,14 @@ void dapl_evd_connection_callback(struct "ep: %p\n", cm_handle, event, ep); - evd = (struct dapl_evd *)ep->param.connect_evd_handle; + evd = (struct dapl_evd *)ep->param.connect_evd; private_data_size = 0; /* * All operations effect the EP, so lock it once and unlock * when necessary */ - spin_lock_irqsave(&ep->header.lock, ep->header.flags); + spin_lock_irqsave(&ep->common.lock, ep->common.flags); switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: @@ -784,13 +766,13 @@ void dapl_evd_connection_callback(struct /* If someone pulled the plug on the connection, just * exit */ - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); status = DAT_SUCCESS; break; } ep->param.ep_state = DAT_EP_STATE_CONNECTED; - ep->cm_handle = cm_handle; + ep->cm = cm_handle; if (private_data == NULL) { private_data_size = 0; } else { @@ -798,7 +780,7 @@ void dapl_evd_connection_callback(struct memcpy(ep->private_data, private_data, private_data_size); } - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -811,8 +793,8 @@ void dapl_evd_connection_callback(struct */ ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); break; } @@ -820,7 +802,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -828,7 +810,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -836,7 +818,7 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } @@ -844,14 +826,14 @@ void dapl_evd_connection_callback(struct { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; dapl_ib_disconnect_clean(ep, FALSE); - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); break; } case DAT_CONNECTION_REQUEST_EVENT: default: { - spin_unlock_irqrestore(&ep->header.lock, ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); evd = NULL; dapl_os_assert(0); /* shouldn't happen */ @@ -865,7 +847,7 @@ void dapl_evd_connection_callback(struct */ if (evd != NULL) { status = dapl_evd_post_connection_event(evd, event, - (DAT_HANDLE) ep, + (struct dat_ep *)ep, private_data_size, /* 0 except on CONNECT */ ep->private_data); @@ -877,10 +859,10 @@ void dapl_evd_connection_callback(struct * expect a callback on an ABRUPT disconnect. */ dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); - spin_lock_irqsave(&ep->header.lock, ep->header.flags); + spin_lock_irqsave(&ep->common.lock, ep->common.flags); ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock_irqrestore(&ep->header.lock, - ep->header.flags); + spin_unlock_irqrestore(&ep->common.lock, + ep->common.flags); } } @@ -901,7 +883,6 @@ void dapl_evd_dto_callback(struct ib_cq evd = (struct dapl_evd *)user_context; dapl_os_assert(evd->cq == cq); - dapl_os_assert(evd->header.magic == DAPL_MAGIC_EVD); /* Read once. */ state = *(volatile DAPL_EVD_STATE *)&evd->evd_state; @@ -927,9 +908,9 @@ void dapl_evd_dto_callback(struct ib_cq status = dapl_ib_completion_notify(evd); if (DAT_SUCCESS != status) (void)dapl_evd_post_async_error_event( - evd->header.owner_ia->async_error_evd, + evd->common.owner_ia->async_error_evd, DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR, - evd->header.owner_ia); + evd->common.owner_ia); dapl_evd_upcall_trigger(evd); } @@ -1008,10 +989,10 @@ bail: return status; } -u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int min_qlen, +u32 dapl_evd_kcreate(struct dat_ia *ia_handle, int min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object *upcall, - enum dat_evd_flags flags, DAT_EVD_HANDLE *evd_handle) + enum dat_evd_flags flags, struct dat_evd **evd_handle) { struct dapl_ia *ia; struct dapl_evd *evd; @@ -1025,7 +1006,7 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha evd = NULL; *evd_handle = NULL; - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + if (!ia_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } @@ -1053,17 +1034,17 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha if (status != DAT_SUCCESS) goto bail; - *evd_handle = (DAT_EVD_HANDLE) evd; + *evd_handle = (struct dat_evd *)evd; bail: if (status != DAT_SUCCESS) if (evd) - dapl_evd_free(evd); + dapl_evd_free((struct dat_evd *)evd); return status; } -u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) +u32 dapl_evd_free(struct dat_evd *evd_handle) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; @@ -1072,7 +1053,7 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1083,11 +1064,11 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han goto bail; } - dapl_ia_unlink_evd(evd->header.owner_ia, evd); + dapl_ia_unlink_evd(evd->common.owner_ia, evd); status = dapl_evd_dealloc(evd); if (status != DAT_SUCCESS) { - dapl_ia_link_evd(evd->header.owner_ia, evd); + dapl_ia_link_evd(evd->common.owner_ia, evd); } bail: @@ -1097,7 +1078,7 @@ bail: return status; } -u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) +u32 dapl_evd_kquery(struct dat_evd *evd_handle, struct dat_evd_param *evd_param) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; @@ -1108,11 +1089,11 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h } /* Note: the spec. allows for events to be directed to a NULL EVD */ - /* with handle of type DAT_HANDLE_NULL. See 6.3.1 */ - if (DAT_HANDLE_NULL == evd_handle) { + /* with handle of type NULL. See 6.3.1 */ + if (NULL == evd_handle) { memset(evd_param, 0, sizeof *evd_param); } else { - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1128,7 +1109,7 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h * about reading the state variable atomically when we add * in waitable/unwaitable. */ - evd_param->ia_handle = evd->header.owner_ia; + evd_param->ia = (struct dat_ia *)evd->common.owner_ia; evd_param->evd_qlen = evd->qlen; evd_param->evd_flags = evd->evd_flags; evd_param->upcall_policy = evd->upcall_policy; @@ -1139,7 +1120,7 @@ bail: return status; } -u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +u32 dapl_evd_modify_upcall(struct dat_evd *evd_handle, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object *upcall) { @@ -1150,7 +1131,7 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1162,14 +1143,14 @@ bail: return status; } -u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) +u32 dapl_evd_post_se(struct dat_evd *evd_handle, const struct dat_event *event) { struct dapl_evd *evd; u32 status = DAT_SUCCESS; evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1196,7 +1177,7 @@ u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_ return status; } -u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) +u32 dapl_evd_dequeue(struct dat_evd *evd_handle, struct dat_event *event) { struct dapl_evd *evd; struct dat_event *local_event; @@ -1207,7 +1188,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ evd = (struct dapl_evd *)evd_handle; - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } @@ -1222,7 +1203,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ * restricts us from having multiple threads in CQ poll, and the * DAT API allows multiple threads in dat_evd_dequeue() */ - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); /* * Make sure there are no other waiters and the evd is active. @@ -1230,7 +1211,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ */ if (evd->evd_state != DAPL_EVD_STATE_OPEN || evd->catastrophic_overflow) { - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); status = DAT_ERROR(DAT_INVALID_STATE, 0); goto bail; } @@ -1249,7 +1230,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ status = DAT_ERROR(DAT_QUEUE_EMPTY, 0); } - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_dequeue () returns 0x%x\n", status); @@ -1257,7 +1238,7 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_ return status; } -u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) +u32 dapl_evd_resize(struct dat_evd *evd_handle, int evd_qlen) { struct dapl_ia *ia; struct dapl_evd *evd; @@ -1267,13 +1248,13 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", evd_handle, evd_qlen); - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + if (!evd_handle) { status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } evd = (struct dapl_evd *)evd_handle; - ia = evd->header.owner_ia; + ia = evd->common.owner_ia; if (evd_qlen == evd->qlen) { status = DAT_SUCCESS; @@ -1285,7 +1266,7 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; } - spin_lock_irqsave(&evd->header.lock, evd->header.flags); + spin_lock_irqsave(&evd->common.lock, evd->common.flags); pend_cnt = dapl_rbuf_count(&evd->pending_event_queue); if (pend_cnt > evd_qlen) { @@ -1302,6 +1283,6 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; bail: - spin_unlock_irqrestore(&evd->header.lock, evd->header.flags); + spin_unlock_irqrestore(&evd->common.lock, evd->common.flags); return status; } Index: dat-provider/dapl_ia_query.c =================================================================== --- dat-provider/dapl_ia_query.c (revision 2562) +++ dat-provider/dapl_ia_query.c (working copy) @@ -38,12 +38,12 @@ * Provide the consumer with Interface Adapter and Provider parameters. * * Input: - * ia_handle + * ia * ia_mask * provider_mask * * Output: - * async_evd_handle + * async_evd * ia_parameters * provider_parameters * @@ -51,7 +51,7 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE *async_evd_handle, +u32 dapl_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -66,21 +66,21 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_query (%p, %p, %p, %p)\n", - ia_handle, - async_evd_handle, + ia, + async_evd, ia_attr, provider_attr); - ia_ptr = (struct dapl_ia *)ia_handle; + ia_ptr = (struct dapl_ia *)ia; - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + if (!ia_ptr) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } - if (NULL != async_evd_handle) { - *async_evd_handle = ia_ptr->async_error_evd; + if (NULL != async_evd) { + *async_evd = (struct dat_evd *)ia_ptr->async_error_evd; } if (NULL != ia_attr) @@ -88,7 +88,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl if (NULL != provider_attr) { strncpy(provider_attr->provider_name, - ia_ptr->header.provider->device_name, + ia_ptr->ia.common.provider->device_name, DAT_NAME_MAX_LENGTH); provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; Index: dat-provider/dapl_sp_util.c =================================================================== --- dat-provider/dapl_sp_util.c (revision 2562) +++ dat-provider/dapl_sp_util.c (working copy) @@ -65,22 +65,14 @@ struct dapl_sp *dapl_sp_alloc(struct dap /* zero the structure */ memset(sp_ptr, 0, sizeof *sp_ptr); - /* - * initialize the header - */ - sp_ptr->header.provider = ia_ptr->header.provider; - if (is_psp) { - sp_ptr->header.magic = DAPL_MAGIC_PSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_PSP; - } else { - sp_ptr->header.magic = DAPL_MAGIC_RSP; - sp_ptr->header.handle_type = DAT_HANDLE_TYPE_RSP; - } - sp_ptr->header.owner_ia = ia_ptr; - sp_ptr->header.user_context.as_64 = 0; - sp_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&sp_ptr->header.ia_list_entry); - spin_lock_init(&sp_ptr->header.lock); + sp_ptr->sp.common.provider = ia_ptr->ia.common.provider; + if (is_psp) + sp_ptr->sp.type = DAT_SP_TYPE_PSP; + else + sp_ptr->sp.type = DAT_SP_TYPE_RSP; + sp_ptr->common.owner_ia = ia_ptr; + dapl_llist_init_entry(&sp_ptr->common.ia_list_entry); + spin_lock_init(&sp_ptr->common.lock); /* * Initialize the Body (set to NULL above) @@ -107,12 +99,8 @@ struct dapl_sp *dapl_sp_alloc(struct dap */ void dapl_sp_dealloc(struct dapl_sp *sp_ptr) { - dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || - sp_ptr->header.magic == DAPL_MAGIC_RSP); dapl_os_assert(dapl_llist_is_empty(&sp_ptr->cr_list_head)); - sp_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - kfree(sp_ptr); } @@ -134,11 +122,11 @@ void dapl_sp_dealloc(struct dapl_sp *sp_ */ void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) { - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); dapl_llist_add_tail(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry, cr_ptr); + &cr_ptr->common.ia_list_entry, cr_ptr); sp_ptr->cr_list_count++; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, sp_ptr->common.flags); } /* @@ -174,7 +162,7 @@ struct dapl_cr *dapl_sp_search_cr(struct if (cr_ptr->ib_cm_handle == ib_cm_handle) return cr_ptr; - cr_ptr = cr_ptr->header.ia_list_entry.flink->data; + cr_ptr = cr_ptr->common.ia_list_entry.flink->data; } while ((void *)cr_ptr != (void *)sp_ptr->cr_list_head->data); return NULL; @@ -208,7 +196,7 @@ void dapl_sp_remove_cr(struct dapl_sp *s } dapl_llist_remove_entry(&sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); sp_ptr->cr_list_count--; } @@ -238,13 +226,13 @@ void dapl_sp_remove_ep(struct dapl_ep *e if (cr_ptr != NULL) { sp_ptr = cr_ptr->sp_ptr; - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); /* Remove the CR from the queue */ dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); /* free memory outside of the lock */ dapl_cr_free(cr_ptr); Index: dat-provider/dapl_evd.h =================================================================== --- dat-provider/dapl_evd.h (revision 2563) +++ dat-provider/dapl_evd.h (working copy) @@ -60,14 +60,14 @@ extern void dapl_evd_un_async_error_call */ extern u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd, - DAT_SP_HANDLE sp_handle, + struct dat_sp *sp_handle, struct sockaddr *ia_address, DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle); + struct dat_cr *cr_handle); extern u32 dapl_evd_post_connection_event(struct dapl_evd *evd, enum dat_event_number event_number, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep_handle, int private_data_size, void *private_data); Index: dat-provider/dapl.h =================================================================== --- dat-provider/dapl.h (revision 2563) +++ dat-provider/dapl.h (working copy) @@ -46,23 +46,6 @@ * * *********************************************************************/ -typedef enum dapl_magic { - /* magic number values for verification & debug */ - DAPL_MAGIC_IA = 0xCafeF00d, - DAPL_MAGIC_EVD = 0xFeedFace, - DAPL_MAGIC_EP = 0xDeadBabe, - DAPL_MAGIC_LMR = 0xBeefCafe, - DAPL_MAGIC_RMR = 0xABadCafe, - DAPL_MAGIC_PZ = 0xDeafBeef, - DAPL_MAGIC_PSP = 0xBeadeD0c, - DAPL_MAGIC_RSP = 0xFab4Feed, - DAPL_MAGIC_SRQ = 0xC001Babe, - DAPL_MAGIC_CR = 0xBe12Cee1, - DAPL_MAGIC_CR_DESTROYED = 0xB12bDead, - DAPL_MAGIC_CNO = 0xDeadF00d, - DAPL_MAGIC_INVALID = 0xFFFFFFFF -} DAPL_MAGIC; - typedef enum dapl_evd_state { DAPL_EVD_STATE_OPEN, DAPL_EVD_STATE_DEAD = 0xDEAD @@ -92,23 +75,6 @@ typedef enum dapl_evd_completion { /********************************************************************* * * - * Macros * - * * - *********************************************************************/ - -/* - * Simple macro to verify a handle is bad. Conditions: - * - pointer is NULL - * - pointer is not word aligned - * - pointer's magic number is wrong - */ -#define DAPL_BAD_HANDLE(h, magicNum) ( \ - ((h) == NULL) || \ - ((unsigned long)(h) & 3) || \ - (((struct dapl_header *)(h))->magic != (magicNum))) - -/********************************************************************* - * * * Typedefs * * * *********************************************************************/ @@ -170,21 +136,17 @@ struct dapl_hca { struct dat_ep_attr ep_attr; }; -/* DAPL Objects always have the following header */ -struct dapl_header { - struct dat_provider *provider; /* required by DAT - must be first */ - DAPL_MAGIC magic; /* magic number for verification */ - enum dat_handle_type handle_type; +/* DAPL Objects always have the following items */ +struct dapl_common { struct dapl_ia *owner_ia; struct dapl_llist_entry ia_list_entry; - union dat_context user_context; /* user context - opaque to DAPL */ spinlock_t lock; unsigned long flags; /* saved lock flag values */ }; -/* struct dapl_ia maps to DAT_IA_HANDLE */ struct dapl_ia { - struct dapl_header header; /* WARNING: must be first */ + struct dat_ia ia; + struct dapl_common common; struct dapl_hca *hca_ptr; struct dapl_evd *async_error_evd; boolean_t cleanup_async_error_evd; @@ -200,9 +162,9 @@ struct dapl_ia { struct dapl_llist_entry *srq_list_head; /* SRQ queue */ }; -/* struct dapl_evd maps to DAT_EVD_HANDLE */ struct dapl_evd { - struct dapl_header header; /* WARNING: must be first */ + struct dat_evd evd; + struct dapl_common common; DAPL_EVD_STATE evd_state; enum dat_evd_flags evd_flags; @@ -230,9 +192,9 @@ struct dapl_evd { struct dat_upcall_object upcall; }; -/* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { - struct dapl_header header; + struct dat_ep ep; + struct dapl_common common; /* What the DAT Consumer asked for */ struct dat_ep_param param; @@ -241,7 +203,7 @@ struct dapl_ep { enum ib_qp_state qp_state; /* communications manager handle */ - struct dapl_cm_id *cm_handle; + struct dapl_cm_id *cm; /* store the remote IA address here, reference from the param * struct which only has a pointer, no storage */ @@ -267,33 +229,33 @@ struct dapl_ep { int send_iov_num; }; -/* struct dapl_srq maps to DAT_SRQ_HANDLE */ struct dapl_srq { - struct dapl_header header; /* WARNING: must be first */ + struct dat_srq srq; + struct dapl_common common; struct dat_srq_param param; atomic_t srq_ref_count; struct dapl_cookie_buffer recv_buffer; atomic_t recv_count; }; -/* struct dapl_pz maps to DAT_PZ_HANDLE */ struct dapl_pz { - struct dapl_header header; /* WARNING: must be first */ + struct dat_pz pz; + struct dapl_common common; struct ib_pd *pd; atomic_t pz_ref_count; }; -/* struct dapl_lmr maps to DAT_LMR_HANDLE */ struct dapl_lmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_lmr lmr; + struct dapl_common common; struct dat_lmr_param param; struct ib_mr *mr; atomic_t lmr_ref_count; }; -/* struct dapl_rmr maps to DAT_RMR_HANDLE */ struct dapl_rmr { - struct dapl_header header; /* WARNING: must be first */ + struct dat_rmr rmr; + struct dapl_common common; struct dat_rmr_param param; struct dapl_ep *ep; struct dapl_pz *pz; @@ -310,17 +272,17 @@ typedef enum dapl_sp_state { DAPL_SP_STATE_RSP_PENDING } DAPL_SP_STATE; -/* struct dapl_sp maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ struct dapl_sp { - struct dapl_header header; /* WARNING: must be first */ + struct dat_sp sp; + struct dapl_common common; DAPL_SP_STATE state; /* type and queue of the SP */ /* PSP/RSP PARAM fields */ - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; /* maintenance fields */ boolean_t listening; /* PSP is registered & active */ @@ -329,9 +291,9 @@ struct dapl_sp { int cr_list_count; /* count of CRs on queue */ }; -/* struct dapl_cr maps to DAT_CR_HANDLE */ struct dapl_cr { - struct dapl_header header; + struct dat_cr cr; + struct dapl_common common; /* for convenience the data is kept as a struct dat_cr_param. * however, the "local_endpoint" field is always NULL * so this wastes a pointer. This is probably ok to @@ -396,91 +358,88 @@ struct dapl_cookie { */ extern u32 dapl_ia_open(const char *name, int async_evd_qlen, - DAT_EVD_HANDLE *async_evd_handle_ptr, - DAT_IA_HANDLE *ia_handle_ptr); + struct dat_evd **async_evd_ptr, + struct dat_ia **ia_ptr); -extern u32 dapl_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dapl_ia_close(struct dat_ia *, enum dat_close_flags); -extern u32 dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, +extern u32 dapl_ia_query(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); /* helper functions */ -extern u32 dapl_set_consumer_context(DAT_HANDLE handle, +extern u32 dapl_set_consumer_context(void *handle, union dat_context context); -extern u32 dapl_get_consumer_context(DAT_HANDLE handle, +extern u32 dapl_get_consumer_context(void *handle, union dat_context *context); -extern u32 dapl_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type); - /* CR Functions */ -extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); +extern u32 dapl_cr_query(struct dat_cr *, struct dat_cr_param *); -extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +extern u32 dapl_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data); -extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); +extern u32 dapl_cr_reject(struct dat_cr *cr); -extern u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL handoff); +extern u32 dapl_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff); /* EVD Functions */ -extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, +extern u32 dapl_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, u64 *suggested_alignment); -extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, +extern u32 dapl_evd_kcreate(struct dat_ia *ia, int evd_min_qlen, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall, enum dat_evd_flags evd_flags, - DAT_EVD_HANDLE * evd_handle); + struct dat_evd ** evd); -extern u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_kquery(struct dat_evd *evd, struct dat_evd_param *evd_args); -extern u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall); -extern u32 dapl_evd_enable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_enable(struct dat_evd *evd); -extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_disable(struct dat_evd *evd); -extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen); +extern u32 dapl_evd_resize(struct dat_evd *evd, int evd_qlen); -extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, +extern u32 dapl_evd_post_se(struct dat_evd *evd, const struct dat_event *event); -extern u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event); +extern u32 dapl_evd_dequeue(struct dat_evd *evd, struct dat_event *event); -extern u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_free(struct dat_evd *evd); -extern u32 dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_set_unwaitable(struct dat_evd *evd); -extern u32 dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_clear_unwaitable(struct dat_evd *evd); /* EP functions */ -extern u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE in_dto_completion_evd_handle, - DAT_EVD_HANDLE out_dto_completion_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, +extern u32 dapl_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_dto_completion_evd, + struct dat_evd *out_dto_completion_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *ep_parameters, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); +extern u32 dapl_ep_query(struct dat_ep *, struct dat_ep_param *); -extern u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask ep_args_mask, const struct dat_ep_param *ep_args); -extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, @@ -489,161 +448,161 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); -extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, +extern u32 dapl_ep_dup_connect(struct dat_ep *ep, + struct dat_ep *ep_dup, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); -extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_disconnect(struct dat_ep *ep, enum dat_close_flags close_flags); -extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_send(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, +extern u32 dapl_ep_post_recv(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_read(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_post_rdma_write(struct dat_ep *ep, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_get_status(struct dat_ep *ep, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle); -extern u32 dapl_ep_free(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_free(struct dat_ep *ep); -extern u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle); +extern u32 dapl_ep_reset(struct dat_ep *ep); -extern u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_ep_create_with_srq(struct dat_ia *ia, + struct dat_pz *pz, + struct dat_evd *recv_evd, + struct dat_evd *request_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq_handle, const struct dat_ep_attr *ep_attributes, - DAT_EP_HANDLE *ep_handle); + struct dat_ep **ep); -extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_recv_query(struct dat_ep *ep, int *nbufs_allocated, int *bufs_alloc_span); -extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, +extern u32 dapl_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark); /* LMR functions */ -extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, +extern u32 dapl_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, - u64 length, DAT_PZ_HANDLE pz_handle, + u64 length, struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address); -extern u32 dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); +extern u32 dapl_lmr_query(struct dat_lmr *, struct dat_lmr_param *); -extern u32 dapl_lmr_free(DAT_LMR_HANDLE); +extern u32 dapl_lmr_free(struct dat_lmr *); -extern u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); -extern u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, +extern u32 dapl_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *local_segments, u64 num_segments); /* RMR Functions */ -extern u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle); +extern u32 dapl_rmr_create(struct dat_pz *pz, struct dat_rmr **rmr); -extern u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_query(struct dat_rmr *rmr, struct dat_rmr_param *rmr_args); -extern u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, +extern u32 dapl_rmr_bind(struct dat_rmr *rmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, - DAT_EP_HANDLE ep_handle, + struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *context); -extern u32 dapl_rmr_free(DAT_RMR_HANDLE); +extern u32 dapl_rmr_free(struct dat_rmr *rmr); /* PSP Functions */ -extern u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, +extern u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, +extern u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, + struct dat_evd *evd, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle); + struct dat_sp **psp); -extern u32 dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); +extern u32 dapl_psp_query(struct dat_sp *, struct dat_psp_param *); -extern u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle); +extern u32 dapl_psp_free(struct dat_sp *psp); /* RSP Functions */ -extern u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle); +extern u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp); -extern u32 dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); +extern u32 dapl_rsp_query(struct dat_sp *, struct dat_rsp_param *); -extern u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle); +extern u32 dapl_rsp_free(struct dat_sp *rsp); /* PZ Functions */ -extern u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle); +extern u32 dapl_pz_create(struct dat_ia *ia, struct dat_pz **pz); -extern u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_args); +extern u32 dapl_pz_query(struct dat_pz *pz, struct dat_pz_param *pz_args); -extern u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle); +extern u32 dapl_pz_free(struct dat_pz *pz); /* SRQ functions */ -extern u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, +extern u32 dapl_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *srq_attr, - DAT_SRQ_HANDLE *srq_handle); + struct dat_srq **srq); -extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); +extern u32 dapl_srq_free(struct dat_srq *srq); -extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, +extern u32 dapl_srq_post_recv(struct dat_srq *srq, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie); -extern u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_query(struct dat_srq *srq, struct dat_srq_param *srq_param); -extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, +extern u32 dapl_srq_resize(struct dat_srq *srq, int srq_max_recv_dto); -extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark); +extern u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark); /* * DAPL internal utility function prototypes Index: dat-provider/dapl_ia_util.c =================================================================== --- dat-provider/dapl_ia_util.c (revision 2563) +++ dat-provider/dapl_ia_util.c (working copy) @@ -65,17 +65,10 @@ struct dapl_ia *dapl_ia_alloc(struct dat /* zero the structure */ memset(ia_ptr, 0, sizeof *ia_ptr); - /* - * initialize the header - */ - ia_ptr->header.provider = provider; - ia_ptr->header.magic = DAPL_MAGIC_IA; - ia_ptr->header.handle_type = DAT_HANDLE_TYPE_IA; - ia_ptr->header.owner_ia = ia_ptr; - ia_ptr->header.user_context.as_64 = 0; - ia_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&ia_ptr->header.ia_list_entry); - spin_lock_init(&ia_ptr->header.lock); + ia_ptr->ia.common.provider = provider; + ia_ptr->common.owner_ia = ia_ptr; + dapl_llist_init_entry(&ia_ptr->common.ia_list_entry); + spin_lock_init(&ia_ptr->common.lock); /* * initialize the body @@ -127,9 +120,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rmr_list_head)); while (rmr_ptr != NULL) { next_rmr_ptr = dapl_llist_next_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header. + &rmr_ptr->common. ia_list_entry); - dat_status = dapl_rmr_free(rmr_ptr); + dat_status = dapl_rmr_free((struct dat_rmr *)rmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rmr_free(%p) returns %x\n", @@ -141,9 +134,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->rsp_list_head)); while (sp_ptr != NULL) { next_sp_ptr = dapl_llist_next_entry(&ia_ptr->rsp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); - dat_status = dapl_rsp_free(sp_ptr); + dat_status = dapl_rsp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rsp_free(%p) returns %x\n", @@ -155,7 +148,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->ep_list_head)); while (ep_ptr != NULL) { next_ep_ptr = dapl_llist_next_entry(&ia_ptr->ep_list_head, - &ep_ptr->header. + &ep_ptr->common. ia_list_entry); /* * Issue a disconnect if the EP needs it @@ -163,21 +156,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || ep_ptr->param.ep_state == - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - || ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING - || ep_ptr->param.ep_state == - DAT_EP_STATE_DISCONNECT_PENDING) { - dat_status = - dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + DAT_EP_STATE_ACTIVE_CONNECTION_PENDING || + ep_ptr->param.ep_state == DAT_EP_STATE_COMPLETION_PENDING || ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECT_PENDING) { + dat_status = dapl_ep_disconnect((struct dat_ep *)ep_ptr, + DAT_CLOSE_ABRUPT_FLAG); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_disconnect(%p) returns %x\n", ep_ptr, dat_status); } /* force the EP into error state to force flush all posted DTOs. */ - (void) dapl_modify_qp_state_to_error(ep_ptr->qp); + (void)dapl_modify_qp_state_to_error(ep_ptr->qp); - dat_status = dapl_ep_free(ep_ptr); + dat_status = dapl_ep_free((struct dat_ep *)ep_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_free(%p) returns %x\n", @@ -189,9 +180,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->lmr_list_head)); while (lmr_ptr != NULL) { next_lmr_ptr = dapl_llist_next_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header. + &lmr_ptr->common. ia_list_entry); - dat_status = dapl_lmr_free(lmr_ptr); + dat_status = dapl_lmr_free((struct dat_lmr *)lmr_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): lmr_free(%p) returns %x\n", @@ -213,7 +204,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status); next_sp_ptr = dapl_llist_next_entry(&ia_ptr->psp_list_head, - &sp_ptr->header. + &sp_ptr->common. ia_list_entry); /* Remove CR's from this PSP and clean them up */ @@ -222,18 +213,19 @@ u32 dapl_ia_abrupt_close(struct dapl_ia while (cr_ptr != NULL) { next_cr_ptr = dapl_llist_next_entry( &sp_ptr->cr_list_head, - &cr_ptr->header.ia_list_entry); + &cr_ptr->common.ia_list_entry); /* Remove the CR from the queue & cleanup */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + spin_lock_irqsave(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_sp_remove_cr(sp_ptr, cr_ptr); - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); dapl_cr_free(cr_ptr); cr_ptr = next_cr_ptr; } - dat_status = dapl_psp_free(sp_ptr); + dat_status = dapl_psp_free((struct dat_sp *)sp_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): psp_free(%p) returns %x\n", @@ -246,9 +238,9 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->pz_list_head)); while (pz_ptr != NULL) { next_pz_ptr = dapl_llist_next_entry(&ia_ptr->pz_list_head, - &pz_ptr->header. + &pz_ptr->common. ia_list_entry); - dat_status = dapl_pz_free(pz_ptr); + dat_status = dapl_pz_free((struct dat_pz *)pz_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): pz_free(%p) returns %x\n", @@ -266,11 +258,11 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ? NULL : dapl_llist_peek_head(&ia_ptr->evd_list_head)); while (evd_ptr != NULL) { next_evd_ptr = dapl_llist_next_entry(&ia_ptr->evd_list_head, - &evd_ptr->header. + &evd_ptr->common. ia_list_entry); if (evd_ptr != ia_ptr->async_error_evd) { /* it isn't the async EVD; delete it. */ - dat_status = dapl_evd_free(evd_ptr); + dat_status = dapl_evd_free((struct dat_evd *)evd_ptr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): evd_free(%p) returns %x\n", @@ -289,7 +281,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dat_status = dapl_ia_teardown_callbacks(ia_ptr); atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - dat_status = dapl_evd_free(ia_ptr->async_error_evd); + dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != dat_status) dapl_dbg_log(DAPL_DBG_TYPE_WARN, @@ -404,7 +396,7 @@ u32 dapl_ia_graceful_close(struct dapl_i if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); - cur_dat_status = dapl_evd_free(ia_ptr->async_error_evd); + cur_dat_status = dapl_evd_free((struct dat_evd *)ia_ptr->async_error_evd); if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; @@ -424,8 +416,6 @@ bail: */ void dapl_ia_free(struct dapl_ia *ia_ptr) { - dapl_os_assert(ia_ptr->header.magic == DAPL_MAGIC_IA); - dapl_os_assert(ia_ptr->async_error_evd == NULL); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->lmr_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rmr_list_head)); @@ -434,12 +424,8 @@ void dapl_ia_free(struct dapl_ia *ia_ptr dapl_os_assert(dapl_llist_is_empty(&ia_ptr->psp_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rsp_list_head)); - /* - * deinitialize the header - */ dapl_hca_unlink_ia(ia_ptr->hca_ptr, ia_ptr); - ia_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - /* no need to destroy ia_ptr->header.lock */ + /* no need to destroy ia_ptr->common.lock */ kfree(ia_ptr); } @@ -449,10 +435,10 @@ void dapl_ia_free(struct dapl_ia *ia_ptr */ void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry, ep_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry, ep_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -460,10 +446,10 @@ void dapl_ia_link_ep(struct dapl_ia *ia_ */ void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->ep_list_head, - &ep_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &ep_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -471,10 +457,10 @@ void dapl_ia_unlink_ep(struct dapl_ia *i */ void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry, srq_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry, srq_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -482,10 +468,10 @@ void dapl_ia_link_srq(struct dapl_ia *ia */ void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->srq_list_head, - &srq_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &srq_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -493,10 +479,10 @@ void dapl_ia_unlink_srq(struct dapl_ia * */ void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry, lmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry, lmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -504,10 +490,10 @@ void dapl_ia_link_lmr(struct dapl_ia *ia */ void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->lmr_list_head, - &lmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &lmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -515,10 +501,10 @@ void dapl_ia_unlink_lmr(struct dapl_ia * */ void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry, rmr_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry, rmr_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -526,10 +512,10 @@ void dapl_ia_link_rmr(struct dapl_ia *ia */ void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->rmr_list_head, - &rmr_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &rmr_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -537,10 +523,10 @@ void dapl_ia_unlink_rmr(struct dapl_ia * */ void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry, pz_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry, pz_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -548,10 +534,10 @@ void dapl_ia_link_pz(struct dapl_ia *ia_ */ void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->pz_list_head, - &pz_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &pz_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -559,10 +545,10 @@ void dapl_ia_unlink_pz(struct dapl_ia *i */ void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry, evd_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry, evd_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -570,10 +556,10 @@ void dapl_ia_link_evd(struct dapl_ia *ia */ void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_remove_entry(&ia_ptr->evd_list_head, - &evd_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &evd_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -581,10 +567,10 @@ void dapl_ia_unlink_evd(struct dapl_ia * */ void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->psp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -594,17 +580,17 @@ void dapl_ia_unlink_sp(struct dapl_ia *i { struct dapl_llist_entry **list_head; - if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_PSP) { + if (sp_ptr->sp.type == DAT_SP_TYPE_PSP) { list_head = &ia_ptr->psp_list_head; } else { - dapl_os_assert(sp_ptr->header.handle_type == - DAT_HANDLE_TYPE_RSP); + dapl_os_assert(sp_ptr->sp.type == + DAT_SP_TYPE_RSP); list_head = &ia_ptr->rsp_list_head; } - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); - dapl_llist_remove_entry(list_head, &sp_ptr->header.ia_list_entry); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); + dapl_llist_remove_entry(list_head, &sp_ptr->common.ia_list_entry); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } /* @@ -634,7 +620,7 @@ struct dapl_sp *dapl_ia_sp_search(struct else list_head = &ia_ptr->rsp_list_head; - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); sp_ptr = (dapl_llist_is_empty(list_head) ? NULL : dapl_llist_peek_head(list_head)); @@ -644,10 +630,10 @@ struct dapl_sp *dapl_ia_sp_search(struct break; sp_ptr = dapl_llist_next_entry(list_head, - &sp_ptr->header.ia_list_entry); + &sp_ptr->common.ia_list_entry); } - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); return sp_ptr; } @@ -670,10 +656,10 @@ struct dapl_sp *dapl_ia_sp_search(struct */ void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); + spin_lock_irqsave(&ia_ptr->common.lock, ia_ptr->common.flags); dapl_llist_add_head(&ia_ptr->rsp_list_head, - &sp_ptr->header.ia_list_entry, sp_ptr); - spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); + &sp_ptr->common.ia_list_entry, sp_ptr); + spin_unlock_irqrestore(&ia_ptr->common.lock, ia_ptr->common.flags); } u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, Index: dat-provider/dapl_cr_query.c =================================================================== --- dat-provider/dapl_cr_query.c (revision 2562) +++ dat-provider/dapl_cr_query.c (working copy) @@ -34,16 +34,16 @@ #include "dapl.h" #include "dapl_openib_util.h" -u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) +u32 dapl_cr_query(struct dat_cr *cr, struct dat_cr_param *cr_param) { struct dapl_cr *cr_ptr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", - cr_handle, cr_param); + cr, cr_param); - if (DAPL_BAD_HANDLE(cr_handle, DAPL_MAGIC_CR)) { + if (cr == NULL) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); goto bail; } @@ -53,10 +53,10 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl goto bail; } - cr_ptr = (struct dapl_cr *)cr_handle; + cr_ptr = (struct dapl_cr *)cr; /* obtain the remote IP address */ - (void) dapl_ib_cm_remote_addr((DAT_HANDLE) cr_handle, + (void) dapl_ib_cm_remote_addr((void *) cr, &cr_ptr->remote_ia_address); /* since the arguments are easily accessible, ignore the mask */ Index: dat-provider/dapl_provider.c =================================================================== --- dat-provider/dapl_provider.c (revision 2562) +++ dat-provider/dapl_provider.c (working copy) @@ -71,10 +71,6 @@ static struct dat_provider g_dapl_provid .ia_close_func = &dapl_ia_close, .ia_memtype_hint_func = &dapl_ia_memtype_hint, - .set_consumer_context_func = &dapl_set_consumer_context, - .get_consumer_context_func = &dapl_get_consumer_context, - .get_handle_type_func = &dapl_get_handle_type, - .cr_query_func = &dapl_cr_query, .cr_accept_func = &dapl_cr_accept, .cr_reject_func = &dapl_cr_reject, Index: dat-provider/dapl_openib_cm.c =================================================================== --- dat-provider/dapl_openib_cm.c (revision 2563) +++ dat-provider/dapl_openib_cm.c (working copy) @@ -70,7 +70,7 @@ static void dapl_destroy_cm_id(struct da conn, conn->cm_id); ib_destroy_cm_id(conn->cm_id); if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } } @@ -172,7 +172,7 @@ static int dapl_cm_active_cb_handler(str spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -285,7 +285,7 @@ static int dapl_cm_passive_cb_handler(st spin_unlock_irqrestore(&conn->lock, flags); if (ret) { if (conn->ep) - conn->ep->cm_handle = NULL; + conn->ep->cm = NULL; kfree(conn); } return ret; @@ -413,7 +413,7 @@ error: * Initiate a connection with the passive listener on another node * * Input: - * ep_handle + * ep * remote_ia_address * remote_conn_qual * private_data_size size of private data and structure @@ -428,7 +428,7 @@ error: * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, +u32 dapl_ib_connect(struct dat_ep *ep, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, void *private_data) @@ -438,7 +438,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han struct dapl_cm_id *conn; int status; - ep_ptr = (struct dapl_ep *)ep_handle; + ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->qp == NULL) { printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); return DAT_INTERNAL_ERROR; @@ -460,9 +460,9 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han kfree(conn); return DAT_INTERNAL_ERROR; } - conn->ep->cm_handle = conn; + conn->ep->cm = conn; - ia_ptr = conn->ep->header.owner_ia; + ia_ptr = conn->ep->common.owner_ia; /* Setup QP/CM parameters */ memset(&conn->param, 0, sizeof conn->param); @@ -507,7 +507,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han * Disconnect an EP * * Input: - * ep_handle + * ep * close_flags * * Output: @@ -521,7 +521,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han */ u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags) { - struct dapl_cm_id *conn = ep_ptr->cm_handle; + struct dapl_cm_id *conn = ep_ptr->cm; int status; dapl_dbg_log(DAPL_DBG_TYPE_CM, @@ -537,7 +537,7 @@ u32 dapl_ib_disconnect(struct dapl_ep *e status = ib_send_cm_dreq(conn->cm_id, NULL, 0); if (status) printk(KERN_ERR "dapl_ib_disconnect: CM ID 0x%p " - "status %d\n", ep_ptr->cm_handle, status); + "status %d\n", ep_ptr->cm, status); } return DAT_SUCCESS; @@ -607,14 +607,14 @@ void dapl_ib_reinit_ep(struct dapl_ep *e dapl_dbg_log(DAPL_DBG_TYPE_CM, " >>> dapl_ib_reinit_ep: EP %p\n", ep_ptr); - ia_ptr = ep_ptr->header.owner_ia; + ia_ptr = ep_ptr->common.owner_ia; /* * What if QP is connected? Do we need to destroy the cm_id * here as well, send a DREQ, REJ? What cm_id state are we in? * we may not care if we just destroy the cm_id - * ib_destroy_cm_id(ep_ptr->cm_handle); - * supporting this call may require setting the cm_handle to + * ib_destroy_cm_id(ep_ptr->cm); + * supporting this call may require setting the cm to * something like NULL wherever it is destroyed */ @@ -685,7 +685,7 @@ u32 dapl_ib_setup_conn_listener(struct d * Have the CM remove a connection listener. * * Input: - * ia_handle IA handle + * ia IA handle * ServiceID IB Channel Service ID * * Output: @@ -719,7 +719,7 @@ u32 dapl_ib_remove_conn_listener(struct * Perform necessary steps to reject a connection * * Input: - * cm_handle + * cm * * Output: * none @@ -730,24 +730,24 @@ u32 dapl_ib_remove_conn_listener(struct * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle) +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm) { int status; - if (cm_handle == NULL) { + if (cm == NULL) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_reject_connection: invalid cm handle\n"); return DAT_SUCCESS; } - status = ib_send_cm_rej(cm_handle->cm_id, IB_CM_REJ_CONSUMER_DEFINED, + status = ib_send_cm_rej(cm->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_reject_connection: " "ib_send_cm_rej failed: %d\n", status); return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); } - dapl_destroy_cm_id(cm_handle); + dapl_destroy_cm_id(cm); return DAT_SUCCESS; } @@ -757,8 +757,8 @@ u32 dapl_ib_reject_connection(struct dap * Perform necessary steps to accept a connection * * Input: - * cr_handle - * ep_handle + * cr + * ep * private_data_size * private_data * @@ -771,7 +771,7 @@ u32 dapl_ib_reject_connection(struct dap * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, +u32 dapl_ib_accept_connection(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *prd_ptr) { @@ -783,9 +783,9 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN struct ib_cm_rep_param passive_params; struct dapl_cm_id *conn; - cr_ptr = (struct dapl_cr *)cr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; - ia_ptr = ep_ptr->header.owner_ia; + cr_ptr = (struct dapl_cr *)cr; + ep_ptr = (struct dapl_ep *)ep; + ia_ptr = ep_ptr->common.owner_ia; conn = cr_ptr->ib_cm_handle; /* Obtain size of private data structure & contents */ @@ -810,8 +810,8 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN } } - cr_ptr->param.local_ep_handle = ep_handle; - ep_ptr->cm_handle = conn; + cr_ptr->param.local_ep = ep; + ep_ptr->cm = conn; conn->ep = ep_ptr; memset(&passive_params, 0, sizeof passive_params); @@ -851,7 +851,7 @@ reject: /* * ib_cm_get_remote_gid */ -static int ib_cm_get_remote_gid(struct dapl_cm_id *cm_handle, u8 *remote_gid) +static int ib_cm_get_remote_gid(struct dapl_cm_id *cm, u8 *remote_gid) { return DAT_INTERNAL_ERROR; /* for now!!! */ } @@ -862,7 +862,7 @@ static int ib_cm_get_remote_gid(struct d * Obtain the remote IP address given a connection * * Input: - * cr_handle + * cr * private data structure handle (only if not using ATS) * * Output: @@ -874,7 +874,7 @@ static int ib_cm_get_remote_gid(struct d * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, +u32 dapl_ib_cm_remote_addr(void *dat_handle, struct sockaddr_in6 *remote_ia_address) { struct dapl_ia *ia_ptr; @@ -883,7 +883,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da struct dapl_cr *cr_ptr = (struct dapl_cr *)dat_handle; - ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia_handle; + ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia; rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); if (rc < 0) Index: patches/alt_dat_provider_makefile =================================================================== --- patches/alt_dat_provider_makefile (revision 2563) +++ patches/alt_dat_provider_makefile (working copy) @@ -21,8 +21,6 @@ PROVIDER_MODULES := \ dapl_cr_util \ dapl_ep \ dapl_evd \ - dapl_get_consumer_context \ - dapl_get_handle_type \ dapl_hash \ dapl_hca_util \ dapl_ia_close \ @@ -39,7 +37,6 @@ PROVIDER_MODULES := \ dapl_ring_buffer_util \ dapl_rmr \ dapl_rsp \ - dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ dapl_util Index: patches/kdat.h =================================================================== --- patches/kdat.h (revision 2532) +++ patches/kdat.h (working copy) @@ -33,3 +33,41 @@ #include #define DAT_TIMEOUT_INFINITE DAT_TIMEOUT_MAX + +#define DAT_FALSE FALSE +#define DAT_TRUE TRUE + +typedef void * DAT_HANDLE +typedef struct dat_cr * DAT_CR_HANDLE; +typedef struct dat_ep * DAT_EP_HANDLE; +typedef struct dat_evd * DAT_EVD_HANDLE; +typedef struct dat_ia * DAT_IA_HANDLE; +typedef struct dat_lmr * DAT_LMR_HANDLE; +typedef struct dat_psp * DAT_PSP_HANDLE; +typedef struct dat_pz * DAT_PZ_HANDLE; +typedef struct dat_rmr * DAT_RMR_HANDLE; +typedef struct dat_rsp * DAT_RSP_HANDLE; +typedef struct dat_srq * DAT_SRQ_HANDLE; + +static inline u32 dat_set_consumer_context(DAT_HANDLE handle, + union dat_context context) +{ + struct dat_common *common = (struct dat_common *) handle; + common->context = context; + return DAT_SUCCESS; +} + +static inline u32 dat_get_consumer_context(DAT_HANDLE handle, + union dat_context *context) +{ + struct dat_common *common = (struct dat_common *) handle; + if (context) + *context = common->context; + return DAT_SUCCESS; +} + +static inline u32 dat_get_handle_type(DAT_HANDLE handle, + enum dat_handle_type *type) +{ + return DAT_ERRROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); +} Index: dat/api.c =================================================================== --- dat/api.c (revision 2532) +++ dat/api.c (working copy) @@ -350,7 +350,7 @@ static u32 dat_strerror_minor(u32 value, } u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, + struct dat_evd **async_event_handle, struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; @@ -387,13 +387,12 @@ u32 dat_ia_openv(const char *name, int a return status; } - return ia_open_func(name, async_event_qlen, async_event_handle, - ia_handle); + return ia_open_func(name, async_event_qlen, async_event_handle, ia); } EXPORT_SYMBOL(dat_ia_openv); -u32 dat_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags flags) +u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; @@ -401,17 +400,16 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle const char *ia_name; dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "%s(ia:%x, flags:%x)\n", __func__, ia_handle, flags); + "%s(ia:%x, flags:%x)\n", __func__, ia, flags); - provider = DAT_HANDLE_TO_PROVIDER(ia_handle); + provider = ia->common.provider; ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia_handle, NULL, NULL, + if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); - else if (DAT_SUCCESS != (status = provider->ia_close_func(ia_handle, - flags))) + else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "close function for IA %s failed\n", ia_name); else { Index: dat/dat.h =================================================================== --- dat/dat.h (revision 2532) +++ dat/dat.h (working copy) @@ -333,20 +333,6 @@ enum dat_return_subtype { #define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) #define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) -/* dat handle types */ -enum dat_handle_type { - DAT_HANDLE_TYPE_CR, - DAT_HANDLE_TYPE_EP, - DAT_HANDLE_TYPE_EVD, - DAT_HANDLE_TYPE_IA, - DAT_HANDLE_TYPE_LMR, - DAT_HANDLE_TYPE_PSP, - DAT_HANDLE_TYPE_PZ, - DAT_HANDLE_TYPE_RMR, - DAT_HANDLE_TYPE_RSP, - DAT_HANDLE_TYPE_SRQ -}; - #define DAT_NAME_MAX_LENGTH 256 struct dat_named_attr { @@ -385,21 +371,6 @@ enum dat_completion_flags { #define DAT_TIMEOUT_MAX ULONG_MAX -/* dat handles */ -typedef void *DAT_HANDLE; -typedef DAT_HANDLE DAT_CR_HANDLE; -typedef DAT_HANDLE DAT_EP_HANDLE; -typedef DAT_HANDLE DAT_EVD_HANDLE; -typedef DAT_HANDLE DAT_IA_HANDLE; -typedef DAT_HANDLE DAT_LMR_HANDLE; -typedef DAT_HANDLE DAT_PSP_HANDLE; -typedef DAT_HANDLE DAT_PZ_HANDLE; -typedef DAT_HANDLE DAT_RMR_HANDLE; -typedef DAT_HANDLE DAT_RSP_HANDLE; -typedef DAT_HANDLE DAT_SRQ_HANDLE; - -#define DAT_HANDLE_NULL ((DAT_HANDLE)NULL) - typedef u64 DAT_CONN_QUAL; typedef u64 DAT_PORT_QUAL; @@ -541,8 +512,8 @@ enum dat_mem_priv_flags { /* RMR Arguments */ struct dat_rmr_param { - DAT_IA_HANDLE ia_handle; - DAT_PZ_HANDLE pz_handle; + struct dat_ia *ia; + struct dat_pz *pz; struct dat_lmr_triplet lmr_triplet; enum dat_mem_priv_flags mem_priv; DAT_RMR_CONTEXT rmr_context; @@ -588,17 +559,17 @@ enum dat_ep_state { }; struct dat_ep_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_ep_state ep_state; struct sockaddr *local_ia_address_ptr; DAT_PORT_QUAL local_port_qual; struct sockaddr *remote_ia_address_ptr; DAT_PORT_QUAL remote_port_qual; - DAT_PZ_HANDLE pz_handle; - DAT_EVD_HANDLE recv_evd_handle; - DAT_EVD_HANDLE request_evd_handle; - DAT_EVD_HANDLE connect_evd_handle; - DAT_SRQ_HANDLE srq_handle; + struct dat_pz *pz; + struct dat_evd *recv_evd; + struct dat_evd *request_evd; + struct dat_evd *connect_evd; + struct dat_srq *srq; struct dat_ep_attr ep_attr; }; @@ -654,9 +625,9 @@ struct dat_srq_attr { }; struct dat_srq_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_srq_state srq_state; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; int max_recv_dtos; int max_recv_iov; int low_watermark; @@ -667,25 +638,25 @@ struct dat_srq_param { /* PZ Parameters */ struct dat_pz_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; }; /* PSP Parameters */ struct dat_psp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; enum dat_psp_flags psp_flags; }; /* RSP Parameters */ struct dat_rsp_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; DAT_CONN_QUAL conn_qual; - DAT_EVD_HANDLE evd_handle; - DAT_EP_HANDLE ep_handle; + struct dat_evd *evd; + struct dat_ep *ep; }; /* Connection Request Parameters. @@ -712,11 +683,11 @@ struct dat_cr_param { /* The local Endpoint provided by the Service Point for the requested * connection. It is the only Endpoint that can accept a Connection - * Request on this Service Point. The value DAT_HANDLE_NULL + * Request on this Service Point. The value NULL * represents that there is no associated local Endpoint for the * requested connection. */ - DAT_EP_HANDLE local_ep_handle; + struct dat_ep *local_ep; }; /************************** Events ******************************/ @@ -753,7 +724,7 @@ enum dat_dto_completion_status { /* DTO completion event data */ /* transfered_length is not defined if status is not DAT_SUCCESS */ struct dat_dto_completion_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; DAT_DTO_COOKIE user_cookie; enum dat_dto_completion_status status; u64 transfered_length; @@ -761,25 +732,20 @@ struct dat_dto_completion_event_data { /* RMR bind completion event data */ struct dat_rmr_bind_completion_event_data { - DAT_RMR_HANDLE rmr_handle; + struct dat_rmr *rmr; DAT_RMR_COOKIE user_cookie; enum dat_dto_completion_status status; }; -typedef union dat_sp_handle { - DAT_RSP_HANDLE rsp_handle; - DAT_PSP_HANDLE psp_handle; -} DAT_SP_HANDLE; - /* Connection Request Arrival event data */ struct dat_cr_arrival_event_data { /* Handle to the Service Point that received the Connection Request * from the remote side. If the Service Point was Reserved, sp is - * DAT_HANDLE_NULL because the reserved Service Point is + * NULL because the reserved Service Point is * automatically destroyed upon generating this event. Can be PSP * or RSP. */ - DAT_SP_HANDLE sp_handle; + struct dat_sp *sp; /* Address of the IA on which the Connection Request arrived. */ struct sockaddr *local_ia_address_ptr; @@ -791,30 +757,30 @@ struct dat_cr_arrival_event_data { /* The Connection Request instance created by a Provider for the * arrived Connection Request. Consumers can find out private_data - * passed by a remote Consumer from cr_handle. It is up to a Consumer + * passed by a remote Consumer from cr. It is up to a Consumer * to dat_cr_accept or dat_cr_reject of the Connection Request. */ - DAT_CR_HANDLE cr_handle; + struct dat_cr *cr; }; /* Connection event data */ struct dat_connection_event_data { - DAT_EP_HANDLE ep_handle; + struct dat_ep *ep; int private_data_size; void *private_data; }; /* Async Error event data */ -/* For unaffiliated asynchronous event dat_handle is ia_handle. For - * Endpoint affiliated asynchronous event dat_handle is ep_handle. For - * EVD affiliated asynchronous event dat_handle is evd_handle. For SRQ +/* For unaffiliated asynchronous event dat_handle is ia. For + * Endpoint affiliated asynchronous event dat_handle is ep. For + * EVD affiliated asynchronous event dat_handle is evd. For SRQ * affiliated asynchronous event dat_handle is srq_handle. For Memory - * affiliated asynchronous event dat_handle is either lmr_handle, - * rmr_handle or pz_handle. + * affiliated asynchronous event dat_handle is either lmr, + * rmr_handle or pz. */ struct dat_asynch_error_event_data { - DAT_HANDLE dat_handle; /* either IA, EP, EVD, SRQ, */ + void *dat_handle; /* either IA, EP, EVD, SRQ, */ /* LMR, RMR, or PZ handle */ int reason; /* object specific */ }; @@ -894,7 +860,7 @@ typedef union dat_event_data { struct dat_event { enum dat_event_number event_number; - DAT_EVD_HANDLE evd_handle; + struct dat_evd *evd; DAT_EVENT_DATA event_data; }; @@ -934,7 +900,7 @@ struct dat_provider_info { }; struct dat_evd_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; int evd_qlen; enum dat_upcall_policy upcall_policy; struct dat_upcall_object upcall; @@ -960,7 +926,7 @@ enum dat_mem_optimize_flags { typedef union dat_region_description { void *for_va; - DAT_LMR_HANDLE for_lmr_handle; + struct dat_lmr *for_lmr; void *for_pointer; /* For kdapl only */ void *for_array; /* For kdapl only */ u64 for_pa; /* For kdapl only */ @@ -968,11 +934,11 @@ typedef union dat_region_description { /* LMR Arguments */ struct dat_lmr_param { - DAT_IA_HANDLE ia_handle; + struct dat_ia *ia; enum dat_mem_type mem_type; DAT_REGION_DESCRIPTION region_desc; u64 length; - DAT_PZ_HANDLE pz_handle; + struct dat_pz *pz; enum dat_mem_priv_flags mem_priv; DAT_LMR_CONTEXT lmr_context; DAT_RMR_CONTEXT rmr_context; @@ -1083,178 +1049,172 @@ struct dat_provider_attr { * ****************************************************************/ -typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *); +typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **); -typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *, u32, u32, boolean_t); +typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, struct dat_evd **, + struct dat_ia **, u32, u32, boolean_t); -typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum dat_close_flags); -typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_HANDLE, DAT_EVD_HANDLE *, +typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, struct dat_ia_attr *, struct dat_provider_attr *); -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); - -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); +typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct dat_cr_param *); -typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, enum dat_handle_type *); - -typedef u32 (*DAT_CR_QUERY_FUNC)(DAT_CR_HANDLE, struct dat_cr_param *); - -typedef u32 (*DAT_CR_ACCEPT_FUNC)(DAT_CR_HANDLE, DAT_EP_HANDLE, int, +typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct dat_ep *, int, const void*); -typedef u32 (*DAT_CR_REJECT_FUNC)(DAT_CR_HANDLE); +typedef u32 (*DAT_CR_REJECT_FUNC)(struct dat_cr *); -typedef u32 (*DAT_CR_HANDOFF_FUNC)(DAT_CR_HANDLE, DAT_CONN_QUAL); +typedef u32 (*DAT_CR_HANDOFF_FUNC)(struct dat_cr *, DAT_CONN_QUAL); -typedef u32 (*DAT_EVD_RESIZE_FUNC)(DAT_EVD_HANDLE, int); +typedef u32 (*DAT_EVD_RESIZE_FUNC)(struct dat_evd *, int); -typedef u32 (*DAT_EVD_POST_SE_FUNC)(DAT_EVD_HANDLE, const struct dat_event *); +typedef u32 (*DAT_EVD_POST_SE_FUNC)(struct dat_evd *, const struct dat_event *); -typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(DAT_EVD_HANDLE, struct dat_event *); +typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(struct dat_evd *, struct dat_event *); -typedef u32 (*DAT_EVD_FREE_FUNC)(DAT_EVD_HANDLE); +typedef u32 (*DAT_EVD_FREE_FUNC)(struct dat_evd *); -typedef u32 (*DAT_EVD_QUERY_FUNC)(DAT_EVD_HANDLE, struct dat_evd_param *); +typedef u32 (*DAT_EVD_QUERY_FUNC)(struct dat_evd *, struct dat_evd_param *); -typedef u32 (*DAT_EP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - const struct dat_ep_attr *, DAT_EP_HANDLE *); +typedef u32 (*DAT_EP_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, struct dat_evd *, + struct dat_evd *, struct dat_evd *, + const struct dat_ep_attr *, struct dat_ep **); -typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_SRQ_HANDLE, +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_evd *, struct dat_evd *, + struct dat_evd *, struct dat_srq *, const struct dat_ep_attr *, - DAT_EP_HANDLE *); + struct dat_ep **); -typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_HANDLE, struct dat_ep_param *); +typedef u32 (*DAT_EP_QUERY_FUNC)(struct dat_ep *, struct dat_ep_param *); -typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, enum dat_ep_param_mask, +typedef u32 (*DAT_EP_MODIFY_FUNC)(struct dat_ep *, enum dat_ep_param_mask, const struct dat_ep_param *); -typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, +typedef u32 (*DAT_EP_CONNECT_FUNC)(struct dat_ep *, struct sockaddr *, DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); -typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(struct dat_ep *, struct dat_ep *, unsigned long timeout, int, const void *, enum dat_qos); -typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); +typedef u32 (*DAT_EP_DISCONNECT_FUNC)(struct dat_ep *, enum dat_close_flags); -typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_SEND_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RECV_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RECV_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(DAT_EP_HANDLE, int, +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(struct dat_ep *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, enum dat_ep_state *, +typedef u32 (*DAT_EP_GET_STATUS_FUNC)(struct dat_ep *, enum dat_ep_state *, boolean_t *, boolean_t *); -typedef u32 (*DAT_EP_FREE_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_FREE_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RESET_FUNC)(DAT_EP_HANDLE); +typedef u32 (*DAT_EP_RESET_FUNC)(struct dat_ep *); -typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(DAT_EP_HANDLE, int *, int *); +typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(struct dat_ep *, int *, int *); -typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(DAT_EP_HANDLE, int, int); +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(struct dat_ep *, int, int); -typedef u32 (*DAT_LMR_FREE_FUNC)(DAT_LMR_HANDLE); +typedef u32 (*DAT_LMR_FREE_FUNC)(struct dat_lmr *); -typedef u32 (*DAT_LMR_QUERY_FUNC)(DAT_LMR_HANDLE, struct dat_lmr_param *); +typedef u32 (*DAT_LMR_QUERY_FUNC)(struct dat_lmr *, struct dat_lmr_param *); -typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(DAT_IA_HANDLE, +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(struct dat_ia *, const struct dat_lmr_triplet *, u64); -typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_PZ_HANDLE, DAT_RMR_HANDLE *); +typedef u32 (*DAT_RMR_CREATE_FUNC)(struct dat_pz *, struct dat_rmr * *); -typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RMR_HANDLE, struct dat_rmr_param *); +typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); -typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, - enum dat_mem_priv_flags, DAT_EP_HANDLE, +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, + enum dat_mem_priv_flags, struct dat_ep *, DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); -typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); +typedef u32 (*DAT_RMR_FREE_FUNC)(struct dat_rmr *); -typedef u32 (*DAT_PSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, - enum dat_psp_flags, DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_evd *, + enum dat_psp_flags, struct dat_sp **); -typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL *, - DAT_EVD_HANDLE, enum dat_psp_flags, - DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(struct dat_ia *, DAT_CONN_QUAL *, + struct dat_evd *, enum dat_psp_flags, + struct dat_sp **); -typedef u32 (*DAT_PSP_QUERY_FUNC)(DAT_PSP_HANDLE, struct dat_psp_param *); +typedef u32 (*DAT_PSP_QUERY_FUNC)(struct dat_sp *, struct dat_psp_param *); -typedef u32 (*DAT_PSP_FREE_FUNC)(DAT_PSP_HANDLE); +typedef u32 (*DAT_PSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_RSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, - DAT_EVD_HANDLE, DAT_RSP_HANDLE *); +typedef u32 (*DAT_RSP_CREATE_FUNC)(struct dat_ia *, DAT_CONN_QUAL, struct dat_ep *, + struct dat_evd *, struct dat_sp **); -typedef u32 (*DAT_RSP_QUERY_FUNC)(DAT_RSP_HANDLE, struct dat_rsp_param *); +typedef u32 (*DAT_RSP_QUERY_FUNC)(struct dat_sp *, struct dat_rsp_param *); -typedef u32 (*DAT_RSP_FREE_FUNC)(DAT_RSP_HANDLE); +typedef u32 (*DAT_RSP_FREE_FUNC)(struct dat_sp *); -typedef u32 (*DAT_PZ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE *); +typedef u32 (*DAT_PZ_CREATE_FUNC)(struct dat_ia *, struct dat_pz **); -typedef u32 (*DAT_PZ_QUERY_FUNC)(DAT_PZ_HANDLE, struct dat_pz_param *); +typedef u32 (*DAT_PZ_QUERY_FUNC)(struct dat_pz *, struct dat_pz_param *); -typedef u32 (*DAT_PZ_FREE_FUNC)(DAT_PZ_HANDLE); +typedef u32 (*DAT_PZ_FREE_FUNC)(struct dat_pz *); -typedef u32 (*DAT_SRQ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, - struct dat_srq_attr *, DAT_SRQ_HANDLE *); +typedef u32 (*DAT_SRQ_CREATE_FUNC)(struct dat_ia *, struct dat_pz *, + struct dat_srq_attr *, struct dat_srq **); -typedef u32 (*DAT_SRQ_SET_LW_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_SET_LW_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_FREE_FUNC)(DAT_SRQ_HANDLE); +typedef u32 (*DAT_SRQ_FREE_FUNC)(struct dat_srq *); -typedef u32 (*DAT_SRQ_QUERY_FUNC)(DAT_SRQ_HANDLE, struct dat_srq_param *); +typedef u32 (*DAT_SRQ_QUERY_FUNC)(struct dat_srq *, struct dat_srq_param *); -typedef u32 (*DAT_SRQ_RESIZE_FUNC)(DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_RESIZE_FUNC)(struct dat_srq *, int); -typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DAT_SRQ_HANDLE, int, +typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(struct dat_srq *, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, +typedef u32 (*DAT_LMR_KCREATE_FUNC)(struct dat_ia *, enum dat_mem_type, DAT_REGION_DESCRIPTION, u64, - DAT_PZ_HANDLE, enum dat_mem_priv_flags, + struct dat_pz *, enum dat_mem_priv_flags, enum dat_mem_optimize_flags, - DAT_LMR_HANDLE *, + struct dat_lmr **, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, u64 *, u64 *); -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, u64, +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(struct dat_ia *, enum dat_mem_type, u64, enum dat_mem_optimize_flags, u64 *, u64 *); -typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, enum dat_upcall_policy, +typedef u32 (*DAT_EVD_KCREATE_FUNC) (struct dat_ia *, int, enum dat_upcall_policy, const struct dat_upcall_object *, - enum dat_evd_flags, DAT_EVD_HANDLE *); + enum dat_evd_flags, struct dat_evd **); -typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(struct dat_evd *, enum dat_upcall_policy, const struct dat_upcall_object *); @@ -1267,10 +1227,6 @@ struct dat_provider { DAT_IA_CLOSE_FUNC ia_close_func; DAT_IA_MEMTYPE_HINT_FUNC ia_memtype_hint_func; - DAT_SET_CONSUMER_CONTEXT_FUNC set_consumer_context_func; - DAT_GET_CONSUMER_CONTEXT_FUNC get_consumer_context_func; - DAT_GET_HANDLE_TYPE_FUNC get_handle_type_func; - DAT_CR_QUERY_FUNC cr_query_func; DAT_CR_ACCEPT_FUNC cr_accept_func; DAT_CR_REJECT_FUNC cr_reject_func; @@ -1338,6 +1294,53 @@ struct dat_provider { DAT_SRQ_SET_LW_FUNC srq_set_lw_func; }; +struct dat_common { + struct dat_provider *provider; + union dat_context context; +}; + +struct dat_ia { + struct dat_common common; +}; + +struct dat_evd { + struct dat_common common; +}; + +struct dat_ep { + struct dat_common common; +}; + +struct dat_srq { + struct dat_common common; +}; + +struct dat_pz { + struct dat_common common; +}; + +struct dat_lmr { + struct dat_common common; +}; + +struct dat_rmr { + struct dat_common common; +}; + +enum dat_sp_type { + DAT_SP_TYPE_RSP, + DAT_SP_TYPE_PSP +}; + +struct dat_sp { + struct dat_common common; + enum dat_sp_type type; +}; + +struct dat_cr { + struct dat_common common; +}; + /*************************************************************** * * FUNCTION PROTOTYPES @@ -1359,8 +1362,8 @@ extern u32 dat_registry_remove_provider( * DAT registry functions for consumers */ extern u32 dat_ia_openv(const char *name, int async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, - DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, + struct dat_evd **async_event_handle, + struct dat_ia **ia, u32 dat_major, u32 dat_minor, boolean_t thread_safety); #define dat_ia_open(name, qlen, async_evd, ia) \ @@ -1368,7 +1371,7 @@ extern u32 dat_ia_openv(const char *name DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ DAT_THREADSAFE) -extern u32 dat_ia_close(DAT_IA_HANDLE, enum dat_close_flags); +extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); extern u32 dat_strerror(u32, const char **, const char **); @@ -1378,32 +1381,10 @@ extern u32 dat_registry_list_providers(i /* * inline functions for consumers */ -#define DAT_HANDLE_TO_PROVIDER(handle) (*(struct dat_provider **)(handle)) - #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ - DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) - -static inline u32 dat_set_consumer_context(DAT_HANDLE handle, - union dat_context context) -{ - return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, - context); -} - -static inline u32 dat_get_consumer_context(DAT_HANDLE handle, - union dat_context *context) -{ - return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, - context); -} - -static inline u32 dat_get_handle_type(DAT_HANDLE handle, - enum dat_handle_type *type) -{ - return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); -} + handle->common.provider->func(handle, ##__VA_ARGS__) -static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, +static inline u32 dat_ia_memtype_hint(struct dat_ia *ia, enum dat_mem_type mem_type, u64 length, enum dat_mem_optimize_flags mem_optimize, @@ -1415,7 +1396,7 @@ static inline u32 dat_ia_memtype_hint(DA preferred_alignment); } -static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE *async_evd, +static inline u32 dat_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { @@ -1423,49 +1404,49 @@ static inline u32 dat_ia_query(DAT_IA_HA ia_query_func, ia, async_evd, ia_attr, provider_attr); } -static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, +static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data) { return DAT_CALL_PROVIDER_FUNC( cr_accept_func, cr, ep, private_data_size, private_data); } -static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) +static inline u32 dat_cr_handoff(struct dat_cr *cr, DAT_CONN_QUAL handoff) { return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); } -static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) +static inline u32 dat_cr_query(struct dat_cr *cr, struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } -static inline u32 dat_cr_reject(DAT_CR_HANDLE cr) +static inline u32 dat_cr_reject(struct dat_cr *cr) { return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); } -static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) +static inline u32 dat_evd_dequeue(struct dat_evd *evd, struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } -static inline u32 dat_evd_free(DAT_EVD_HANDLE evd) +static inline u32 dat_evd_free(struct dat_evd *evd) { return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); } -static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, +static inline u32 dat_evd_kcreate(struct dat_ia *ia, int qlen, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall, enum dat_evd_flags flags, - DAT_EVD_HANDLE * evd) + struct dat_evd ** evd) { return DAT_CALL_PROVIDER_FUNC(evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); } -static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_modify_upcall(struct dat_evd *evd, enum dat_upcall_policy policy, const struct dat_upcall_object *upcall) { @@ -1473,23 +1454,23 @@ static inline u32 dat_evd_modify_upcall( upcall); } -static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_post_se(struct dat_evd *evd, const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } -static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) +static inline u32 dat_evd_query(struct dat_evd *evd, struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } -static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, int qlen) +static inline u32 dat_evd_resize(struct dat_evd *evd, int qlen) { return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, +static inline u32 dat_ep_connect(struct dat_ep *ep, struct sockaddr *ia_addr, DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, @@ -1500,36 +1481,36 @@ static inline u32 dat_ep_connect(DAT_EP_ qos, flags); } -static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, +static inline u32 dat_ep_create(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, struct dat_evd *out_evd, + struct dat_evd *connect_evd, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_func, ia, pz, in_evd, out_evd, connect_evd, attr, ep); } -static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, - DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, - DAT_SRQ_HANDLE srq, +static inline u32 dat_ep_create_with_srq(struct dat_ia *ia, struct dat_pz *pz, + struct dat_evd *in_evd, + struct dat_evd *out_evd, + struct dat_evd *connect_evd, + struct dat_srq *srq, const struct dat_ep_attr *attr, - DAT_EP_HANDLE *ep) + struct dat_ep **ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_with_srq_func, ia, pz, in_evd, out_evd, connect_evd, srq, attr, ep); } -static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_disconnect(struct dat_ep *ep, enum dat_close_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } -static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, +static inline u32 dat_ep_dup_connect(struct dat_ep *ep, struct dat_ep *dup_ep, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) @@ -1538,25 +1519,25 @@ static inline u32 dat_ep_dup_connect(DAT private_data_size, private_data, qos); } -static inline u32 dat_ep_free(DAT_EP_HANDLE ep) +static inline u32 dat_ep_free(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, enum dat_ep_state *state, +static inline u32 dat_ep_get_status(struct dat_ep *ep, enum dat_ep_state *state, boolean_t *recv_idle, boolean_t *req_idle) { return DAT_CALL_PROVIDER_FUNC(ep_get_status_func, ep, state, recv_idle, req_idle); } -static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, enum dat_ep_param_mask mask, +static inline u32 dat_ep_modify(struct dat_ep *ep, enum dat_ep_param_mask mask, const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } -static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_read(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1566,7 +1547,7 @@ static inline u32 dat_ep_post_rdma_read( local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_rdma_write(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, @@ -1576,7 +1557,7 @@ static inline u32 dat_ep_post_rdma_write local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, int size, +static inline u32 dat_ep_post_recv(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, enum dat_completion_flags flags) @@ -1585,7 +1566,7 @@ static inline u32 dat_ep_post_recv(DAT_E cookie, flags); } -static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_send(struct dat_ep *ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1595,24 +1576,24 @@ static inline u32 dat_ep_post_send(DAT_E ep_post_send_func, ep, size, local_iov, cookie, flags); } -static inline u32 dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) +static inline u32 dat_ep_query(struct dat_ep *ep, struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } -static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, int *bufs_alloc, +static inline u32 dat_ep_recv_query(struct dat_ep *ep, int *bufs_alloc, int *bufs_avail) { return DAT_CALL_PROVIDER_FUNC(ep_recv_query_func, ep, bufs_alloc, bufs_avail); } -static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) +static inline u32 dat_ep_reset(struct dat_ep *ep) { return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); } -static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, +static inline u32 dat_ep_set_watermark(struct dat_ep *ep, int soft_high_watermark, int hard_high_watermark) { @@ -1620,17 +1601,17 @@ static inline u32 dat_ep_set_watermark(D soft_high_watermark, hard_high_watermark); } -static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) +static inline u32 dat_lmr_query(struct dat_lmr *lmr, struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } -static inline u32 dat_lmr_free(DAT_LMR_HANDLE lmr) +static inline u32 dat_lmr_free(struct dat_lmr *lmr) { return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); } -static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_read(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1638,7 +1619,7 @@ static inline u32 dat_lmr_sync_rdma_read num_iovs); } -static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_write(struct dat_ia *ia, const struct dat_lmr_triplet *iovs, u64 num_iovs) { @@ -1646,20 +1627,20 @@ static inline u32 dat_lmr_sync_rdma_writ num_iovs); } -static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE *rmr) +static inline u32 dat_rmr_create(struct dat_pz *pz, struct dat_rmr * *rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline u32 dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) +static inline u32 dat_rmr_query(struct dat_rmr * rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, +static inline u32 dat_rmr_bind(struct dat_rmr * rmr, const struct dat_lmr_triplet *iov, enum dat_mem_priv_flags mem_flags, - DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, + struct dat_ep *ep, DAT_RMR_COOKIE cookie, enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) { @@ -1667,84 +1648,84 @@ static inline u32 dat_rmr_bind(DAT_RMR_H cookie, comp_flags, context); } -static inline u32 dat_rmr_free(DAT_RMR_HANDLE rmr) +static inline u32 dat_rmr_free(struct dat_rmr * rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) +static inline u32 dat_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd, +static inline u32 dat_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags flags, - DAT_PSP_HANDLE *psp) + struct dat_sp **psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_any_func, ia, conn_qual, evd, flags, psp); } -static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) +static inline u32 dat_psp_query(struct dat_sp *psp, struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } -static inline u32 dat_psp_free(DAT_PSP_HANDLE psp) +static inline u32 dat_psp_free(struct dat_sp *psp) { return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); } -static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, - DAT_RSP_HANDLE *rsp) +static inline u32 dat_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_create_func, ia, conn_qual, ep, evd, rsp); } -static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) +static inline u32 dat_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } -static inline u32 dat_rsp_free(DAT_RSP_HANDLE rsp) +static inline u32 dat_rsp_free(struct dat_sp *rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); } -static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE *pz) +static inline u32 dat_pz_create(struct dat_ia *ia, struct dat_pz **pz) { return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) +static inline u32 dat_pz_query(struct dat_pz *pz, struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } -static inline u32 dat_pz_free(DAT_PZ_HANDLE pz) +static inline u32 dat_pz_free(struct dat_pz *pz) { return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); } -static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, +static inline u32 dat_srq_create(struct dat_ia *ia, struct dat_pz *pz, struct dat_srq_attr *attr, - DAT_SRQ_HANDLE *srq) + struct dat_srq **srq) { return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); } -static inline u32 dat_srq_free(DAT_SRQ_HANDLE srq) +static inline u32 dat_srq_free(struct dat_srq *srq) { return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); } -static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, int num_iovs, +static inline u32 dat_srq_post_recv(struct dat_srq *srq, int num_iovs, struct dat_lmr_triplet *iovs, DAT_DTO_COOKIE cookie) { @@ -1752,34 +1733,34 @@ static inline u32 dat_srq_post_recv(DAT_ cookie); } -static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) +static inline u32 dat_srq_query(struct dat_srq *srq, struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } -static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, int max_recv_dtos) +static inline u32 dat_srq_resize(struct dat_srq *srq, int max_recv_dtos) { return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); } -static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, int low_watermark) +static inline u32 dat_srq_set_lw(struct dat_srq *srq, int low_watermark) { return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, enum dat_mem_type type, +static inline u32 dat_lmr_kcreate(struct dat_ia *ia, enum dat_mem_type type, DAT_REGION_DESCRIPTION region, u64 len, - DAT_PZ_HANDLE pz, + struct dat_pz *pz, enum dat_mem_priv_flags privileges, enum dat_mem_optimize_flags optimization, - DAT_LMR_HANDLE *lmr_handle, + struct dat_lmr **lmr, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { return DAT_CALL_PROVIDER_FUNC(lmr_kcreate_func, ia, type, region, len, - pz, privileges, optimization, lmr_handle, + pz, privileges, optimization, lmr, lmr_context, rmr_context, registered_length, registered_address); } From jlentini at netapp.com Wed Jun 8 12:52:16 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 15:52:16 -0400 (EDT) Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <42A7347B.8060100@ichips.intel.com> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> <42A7347B.8060100@ichips.intel.com> Message-ID: I had all day meetings Monday and Tuesday that left me 2 days behind on email. I've just finished going through this. On Wed, 8 Jun 2005, Sean Hefty wrote: > Tom Duffy wrote: >> On Wed, 2005-06-08 at 12:35 -0400, James Lentini wrote: >> >>> Committed in revision 2572. >> >> Updated patch against 2572. >> >> Signed-off-by: Tom Duffy > > James, is there a specific hold-up for this patch? > > - Sean > From tduffy at sun.com Wed Jun 8 13:16:46 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 13:16:46 -0700 Subject: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: References: <1118093694.5735.13.camel@duffman> <1118101725.30742.2.camel@duffman> Message-ID: <1118261806.31987.21.camel@duffman> On Wed, 2005-06-08 at 15:48 -0400, James Lentini wrote: > I like this update Tom. > > I'd propse a few changes: > > - remove dat_cno because it is not part of the kDAPL API > > - remove DAT_HANDLE_TO_PROVIDER > > - we can provide the functionality of dat_set_consumer_context and > dat_get_consumer_context very easily by creating a dat_common > structure (see my patch). In that case we can remove > the user_context member from the dapl_common structure. > > - I've added the symbols removed from dat.h to kdat.h for legacy > support. > > Let me know what you think. This looks real good, James. I agree with all of your changes. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Wed Jun 8 13:19:41 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 13:19:41 -0700 Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <20050608194831.GB5412@aon.at> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> <42A7347B.8060100@ichips.intel.com> <20050608194831.GB5412@aon.at> Message-ID: <1118261981.7368.1.camel@duffman> On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote: > Tom, just curious.. > Do you plan to iterate over those disturbing > 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally > would appreciate it, since these variants are harder to read for me. What do you want to see instead? if (this) { /* do stuff */ } or were you thinking: if (this != NULL) { /* do stuff */ } > But then i also tend to prefer '* 4711e-0815' over '/ 0x100000' Come again? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From rep.nop at aon.at Wed Jun 8 13:24:45 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Wed, 8 Jun 2005 22:24:45 +0200 Subject: [openib-general] openib-commits administrative Message-ID: <20050608202445.GC5412@aon.at> Hi, May i as ask if diffs sent to the -commits list may be changed to be generated with the --show-c-fun option to diff? Something along the lines of svn diff --diff-cmd diff -x -urNp .. works for me. Knowing the function the change was based against simply makes it easier to follow. PS: yes, OT. But as #openib is completely orphaned (which is a pity, one would expect that folks at least once a week would check the logs they gather from reading there, but well) i know no other addr i could send that plea too. Sorry to the not concerned majority who i had to mux. PPS: having a cia-bot in #openib would be great, too, but i have no illusion that anyone would be willing to setup this. Prove me wrong. thank you, From jlentini at netapp.com Wed Jun 8 13:36:01 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 16:36:01 -0400 (EDT) Subject: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: <1118261806.31987.21.camel@duffman> References: <1118093694.5735.13.camel@duffman> <1118101725.30742.2.camel@duffman> <1118261806.31987.21.camel@duffman> Message-ID: Do you see any additional stability problems after applying this? I'm updating my OpenIB tree to see if that is my problem. james On Wed, 8 Jun 2005, Tom Duffy wrote: > On Wed, 2005-06-08 at 15:48 -0400, James Lentini wrote: >> I like this update Tom. >> >> I'd propse a few changes: >> >> - remove dat_cno because it is not part of the kDAPL API >> >> - remove DAT_HANDLE_TO_PROVIDER >> >> - we can provide the functionality of dat_set_consumer_context and >> dat_get_consumer_context very easily by creating a dat_common >> structure (see my patch). In that case we can remove >> the user_context member from the dapl_common structure. >> >> - I've added the symbols removed from dat.h to kdat.h for legacy >> support. >> >> Let me know what you think. > > This looks real good, James. I agree with all of your changes. > > -tduffy > From mst at mellanox.co.il Wed Jun 8 13:39:16 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 8 Jun 2005 23:39:16 +0300 Subject: [openib-general] Re: [PATCH][SDP] state transition simplification. In-Reply-To: <20050607133616.A27322@topspin.com> References: <20050607133616.A27322@topspin.com> Message-ID: <20050608203916.GC11380@mellanox.co.il> Quoting r. Libor Michalek : > Subject: [PATCH][SDP] state transition simplification. > > > This patch simplifies the state transitions for connection management > within SDP. Previously the connection state was maintained using two > variables 'istate' and 'state' with a combined total of 34 different > values. The patch merges the two into a single variable 'state' and > removes redundancy for a total of 16 state values. Also the file > sdp_wall.c is removed who's original purpose was to keep the two state > variables synchronized. > > The simplification also paves the way for the next step, which is to > use the CM event for transitions instead of the current incorrest use > of CM state. > > The CQ event pending flags were moved from the connection flags field > to a new field in the event lock. > > Finally, I needed to ifdef out one of the send_dreq calls, because > I was seeing a problem when two DREQs would cross in flight and both > sides responded with a DREP. > > 13 files changed, 404 insertions(+), 1045 deletions(-) > > -Libor Works fine for me so far. -- MST From tduffy at sun.com Wed Jun 8 13:36:50 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 13:36:50 -0700 Subject: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: References: <1118093694.5735.13.camel@duffman> <1118101725.30742.2.camel@duffman> <1118261806.31987.21.camel@duffman> Message-ID: <1118263010.8536.2.camel@duffman> On Wed, 2005-06-08 at 16:36 -0400, James Lentini wrote: > Do you see any additional stability problems after applying this? I'm > updating my OpenIB tree to see if that is my problem. Sorry, I just looked at the code, didn't actually test it. Lemme give it a whirl... -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From rep.nop at aon.at Wed Jun 8 13:41:57 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Wed, 8 Jun 2005 22:41:57 +0200 Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <1118261981.7368.1.camel@duffman> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> <42A7347B.8060100@ichips.intel.com> <20050608194831.GB5412@aon.at> <1118261981.7368.1.camel@duffman> Message-ID: <20050608204157.GD5412@aon.at> On Wed, Jun 08, 2005 at 01:19:41PM -0700, Tom Duffy wrote: >On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote: >> 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally >> would appreciate it, since these variants are harder to read for me. > >What do you want to see instead? > >if (this) { > /* do stuff */ >} yes, this. From tduffy at sun.com Wed Jun 8 13:44:50 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 13:44:50 -0700 Subject: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: References: <1118093694.5735.13.camel@duffman> <1118101725.30742.2.camel@duffman> <1118261806.31987.21.camel@duffman> Message-ID: <1118263490.9977.2.camel@duffman> On Wed, 2005-06-08 at 16:36 -0400, James Lentini wrote: > Do you see any additional stability problems after applying this? I'm > updating my OpenIB tree to see if that is my problem. This just in! Don't know if was caused by your patches or mine, but I ran the server on 192.168.0.26, and tried to connect with the client, it didn't connect right away, so I did a control-c. Then, the kernel panic'ed: [root at sins-stinger-10 ~]# ./kdapltest -T Q -d -s 192.168.0.26 -D mthca0a Server Name: 192.168.0.26 Server Net Address: 192.168.0.26 DT_cs_Client: Starting Test ... DT_cs_Client: IA mthca0a opened DT_cs_Client: EP created ***** DAPL Characteristics ***** Provider: mthca0a Version 1.0 DAPL 1.2 Adapter: Generic InfiniBand HCA by Linux Version 0.0 Supporting: 64512 EPs with 65535 DTOs and 0 in RDMA/RDs and 0ut RDMA/RDs each 65408 EVDs of up to 65535 entries (default S/R size is 256/256) IOVs of up to 28 elements 131056 LMRs (and 131056 RMRs) of up to 0xffffffffffffffff bytes Maximum MTU 0x80000000 bytes, RDMA 0x80000000 bytes Maximum Private data size 92 bytes ***** ***** ***** ***** ***** ***** DT_cs_Client: Posting 1 recv buffer DT_cs_Client: Connect Endpoint DT_cs_Client: Await connection ... <--- I DID A CONTROL-C HERE ---> [root at sins-stinger-10 ~]# dapl_path_comp_handler: path resolution failed -110 retry 1802201964!!! dl_ibpa_atht:_c romeqp__henand:dl peren: d epff_pfftr81 000x65ab622b6bbe46b0 6bal6bre6bad6by completed? status 3 dapl_path_comp_handler: path resolution failed -110 retry 1802201965!!! dapl_path_comp_handler: ep_ptr 0x6b6b6b6b6b6b6b6b general protection fault: 0000 [1] SMP CPU 0 Modules linked in: kdapltest ib_dat_provider dat ib_at ib_ipoib ib_sdp ib_cm md5 ipv6 parport_pc lp parport autofs4 nfs lockd rfcomm l2cap bluetooth pcmcia yenta_socket rsrc_nonstatic pcmcia_core sunrpc ext3 jbd dm_mod video container button battery ac ohci_hcd tpm_nsc tpm i2c_amd756 i2c_core ib_mthca ib_sa ib_mad ib_core tg3 floppy xfs exportfs mptscsih mptbase sd_mod scsi_mod Pid: 11881, comm: ib_at_wq/0 Not tainted 2.6.12-rc6openib RIP: 0010:[] {:ib_dat_provider:dapl_evd_connection_callback+67} RSP: 0018:ffff8100218b9dc8 EFLAGS: 00010296 RAX: 6b6b6b6b6b6b6b6b RBX: ffff81005a22be40 RCX: 0000000000004008 RDX: ffff810075baaaf8 RSI: ffffffff883141a0 RDI: 0000000000000048 RBP: ffff81005a22be70 R08: 6b6b6b6b6b6b6b6b R09: 0000000000000000 R10: 0000000000000010 R11: 0000000000000010 R12: ffff81007d9b8c50 R13: ffff81005a22be40 R14: 0000000000000292 R15: ffffffff882f5280 FS: 00002aaaaaad7d60(0000) GS:ffffffff804e7880(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00007fffffd1dce6 CR3: 00000000769db000 CR4: 00000000000006e0 Process ib_at_wq/0 (pid: 11881, threadinfo ffff8100218b8000, task ffff81004167caa0) Stack: ffff81004167caa0 6b6b6b6b6b6b6b6b 0000000000000000 000040080000006e ffff810075baaaf8 ffff81003fdc8f10 0000000000000000 0000000000000003 ffff8100218b9e48 ffffffff88302d18 Call Trace:{:ib_dat_provider:dapl_path_comp_handler+416} {:ib_at:req_comp_work+27} {worker_thread+476} {default_wake_function+0} {keventd_create_kthread+0} {worker_thread+0} {keventd_create_kthread+0} {kthread+217} {schedule_tail+64} {child_rip+8} {keventd_create_kthread+0} {kthread+0} {child_rip+0} Code: 48 8b 80 90 00 00 00 48 89 44 24 28 c7 44 24 30 00 00 00 00 RIP {:ib_dat_provider:dapl_evd_connection_callback+67} RSP <<0>3>geSlneabra cl orprruotpteciotin:on s ftaaurtlt=f: ff00f80010 [072]5b aa<4af>S8,MP l en 51CP2 = 1R eon e:M 0odx5ula2escf l07in1/ke0xd 5ain2c:f0 k71da. teLastst iusb_erda: t_[_a]t( daibpl_i_dpoesibtro iy_b_cms_dpid +0ibxb_c9/m0 xbmde 5[ ibip_dv6at _pparorpviordter_p])c l p<04>f0 p:a rp6bor t6 ba ut6bofs 64b n 6fsb l6bock 6db rf6bco mm6d l 62cba p6b bl 6uebto 6otbh 6bpc m6ciba 6bye aP_sreocv keobtj:<4 s> tarsrtrc=f_nffonf8st10at07ic5b aapc8emc0,ia l_cenor=5e1 2 nRrpedcz : ex0xt3170 jfcbd2a 5/dm0x_m17od0fc v2aid5.eo Lcoasntt aiusneerr: [bu] (ohkmciem_h_acdllo tc+pm0x_n61sc/0 xetp0 m[ xfi2s]c_)am d750060: i2 0c_0co 0re0 i00b_ m0th0ca 00 i b_00sa 00ib _m00ad 0 i1b _c00or ec 0tg a32 fl0o6pp 0y0 xf00s 0ex0p tf01s0 :mp 0ts0c si02h m00ptb 0as0e 00sd _m00od 0 s0c si00_mo 0d2 4>Pi 0d:0 <114>88 82,0 coa3mm : 06ib_ 0at0_w 0q/01 N00ot < taiNentxted o 2bj.6:. s12ta-rrtc6=fopffenf8ib10 075RIbaP:ad 01001, 0:le[]x1 70<4fc><2af5f/ff0xff17ff0f88c230a59c. >L{:asibt _dusater_p: ro[]nec(ktimeonm__calallolbc+ac0xk+6167/0}xe 0 R[xSPfs:] 0)01<48:> f0f80010:0 5900c3 dd0c08 0 E0F LA00GS : 00000 010029 06 0R0AX:<4 6> b601b6 b600b6b a6b06b f6be RB00X: f04fff 081000 05a02 e4010 0:RC X:00 00 000200 00000 000004 00008 2b 00RD f 0ff0f 810000 750b1aa 0af08 R00SI : cbfff 0ff5ff f0488 310041a 00 0RD I: 0000000000000048 RBP: ffff81 Message from syslogd at sins-st0inger-10 at Wed 0Jun 8 13:36:54 2005 ... sins-5stinger-10 kerneal: general protection fault: 00020 [1] SMP 2be70 R08: 6b6b6b6b6b6b6b6b R09: 0000000000000033 R10: 0000000000000010 R11: 0000000000000010 R12: ffff81007d9b8cd0 R13: ffff81005a22be40 R14: 0000000000000292 R15: ffffffff882f5280 FS: 00002aaaaae0ae60(0000) GS:ffffffff804e7900(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00007fffffbd2e96 CR3: 000000007cb2f000 CR4: 00000000000006e0 Process ib_at_wq/1 (pid: 11882, threadinfo ffff810059c3c000, task ffff81004167d190) Stack: 00000000ffffff92 6b6b6b6b6b6b6b6b 0000000000000000 000040087d9b8d10 ffff810075baaaf8 0000000000000001 0000000000000092 0000000000000003 ffff810059c3de48 ffffffff88302d18 Call Trace:{:ib_dat_provider:dapl_path_comp_handler+416} {:ib_at:req_comp_work+27} {worker_thread+476} {default_wake_function+0} {keventd_create_kthread+0} {worker_thread+0} {keventd_create_kthread+0} {kthread+217} {schedule_tail+64} {child_rip+8} {keventd_create_kthread+0} {kthread+0} {child_rip+0} Code: 48 8b 80 90 00 00 00 48 89 44 24 28 c7 44 24 30 00 00 00 00 RIP {:ib_dat_provider:dapl_evd_connection_callback+67} RSP -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Wed Jun 8 13:52:47 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 13:52:47 -0700 Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <20050608204157.GD5412@aon.at> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> <42A7347B.8060100@ichips.intel.com> <20050608194831.GB5412@aon.at> <1118261981.7368.1.camel@duffman> <20050608204157.GD5412@aon.at> Message-ID: <1118263967.9977.5.camel@duffman> On Wed, 2005-06-08 at 22:41 +0200, Bernhard Fischer wrote: > On Wed, Jun 08, 2005 at 01:19:41PM -0700, Tom Duffy wrote: > >On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote: > > >> 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally > >> would appreciate it, since these variants are harder to read for me. > > > >What do you want to see instead? > > > >if (this) { > > /* do stuff */ > >} > > yes, this. I think James would take these types of patches. In any event, going through the code with fine toothed comb will happen after all the major upheavals, I think. *I* want to fix all the major architectural issues that are holding up acceptance before nit-picking. If I have the time or inclination, I will clean up these things. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From troy at scl.ameslab.gov Wed Jun 8 13:57:17 2005 From: troy at scl.ameslab.gov (Troy Benjegerdes) Date: Wed, 08 Jun 2005 15:57:17 -0500 Subject: [openib-general] 2.6.11.11 NFS over IPoIB crash In-Reply-To: <03c22e4542ae1b6cdd72565432143ebe@scl.ameslab.gov> References: <03c22e4542ae1b6cdd72565432143ebe@scl.ameslab.gov> Message-ID: <42A75BAD.9020205@scl.ameslab.gov> We are running NFS over IPOIB, and are getting kernel panics under heavy NFS I/O. This is on a PowerMac G5, and the server is a dual opteron running 2.6.11 with the OpenIB code from subversion. It looks like a bug in nfs.. but we've only seen it using IPoIB... Is it worth trying to reproduce this over gigabit ethernet? Brett Bode wrote: > Well I finally got 2.6.11.11 running this morning, but crashed it too. > I guess I will try updating to the latest OpenIB code in subversion? > Here is the kernel panic: > > Oops: Kernel access of bad area, sig: 11 [#1] > SMP NR_CPUS=2 POWERMAC > Modules linked in: ib_ipoib ib_sa eth1394 ib_mthca ib_mad ib_core > ohci1394 ieee1394 > NIP: C000000000038B58 XER: 00000000 LR: C0000000000390C8 CTR: > C00000000003B340 > REGS: c000000254c845f0 TRAP: 0300 Not tainted (2.6.11.11-G5) > MSR: 9000000000001032 EE: 0 PR: 0 FP: 0 ME: 1 IR/DR: 11 CR: 42028488 > DAR: c00000060045b218 DSISR: 0000000040010000 > TASK: c0000000019047d0[2004] 'IOTest.ppc.x' THREAD: c000000254c84000 > CPU: 1 > GPR00: 0000000600000010 C000000254C84870 C00000000045F280 > C0000000019047D0 > GPR04: C0000000004B4480 0000000000000000 000000003B9A1A60 > C0000000004B4488 > GPR08: C00000000045B208 C000000254C84000 C000000000328C40 > C000000000328C40 > GPR12: 0000000028028428 C00000000037E000 C00000000043B170 > C000000000470340 > GPR16: 0000000000000000 0000000000000001 0000000000000000 > C00000000044EE20 > GPR20: 0000000000000000 0000000000000000 0000000000000003 > 0000000000000000 > GPR24: C00000000FCC4000 0000000000000000 C00000000FD0B030 > 0000000000000001 > GPR28: C0000000004B3B40 0000000000000001 C000000000391A90 > C000000254C84870 > NIP [c000000000038b58] .resched_task+0x38/0xcc > LR [c0000000000390c8] .try_to_wake_up+0x304/0x324 > Call Trace: > [c000000254c848f0] [c0000000000390c8] .try_to_wake_up+0x304/0x324 > [c000000254c849d0] [c00000000003b3e0] .__wake_up_common+0x84/0xe8 > [c000000254c84a90] [c00000000003b4a0] .__wake_up+0x5c/0x94 > [c000000254c84b40] [c000000000057840] .__queue_work+0x80/0xb4 > [c000000254c84be0] [c000000000057908] .queue_work+0x94/0xb0 > [c000000254c84c60] [c000000000280df8] .rpc_make_runnable+0xe0/0x154 > [c000000254c84cf0] [c000000000281324] .__rpc_do_wake_up_task+0xc8/0x24c > [c000000254c84d80] [c000000000281608] .rpc_wake_up_task+0xb0/0xd8 > [c000000254c84e20] [c00000000027e61c] .xprt_complete_rqst+0x90/0x188 > [c000000254c84ed0] [c00000000027ec3c] .udp_data_ready+0x230/0x290 > [c000000254c84fa0] [c000000000236714] .udp_queue_rcv_skb+0x268/0x51c > [c000000254c85040] [c000000000236fc8] .udp_rcv+0x600/0x7f0 > [c000000254c85160] [c000000000209e8c] .ip_local_deliver_finish+0xd4/0x35c > [c000000254c851f0] [c0000000001fa928] .nf_hook_slow+0x184/0x1bc > [c000000254c852d0] [c00000000020a360] .ip_local_deliver+0x24c/0x3bc > [c000000254c85360] [c00000000020a6f4] .ip_rcv_finish+0x224/0x378 > [c000000254c85410] [c0000000001fa928] .nf_hook_slow+0x184/0x1bc > [c000000254c854f0] [c00000000020ad30] .ip_rcv+0x4e8/0x610 > [c000000254c855a0] [c0000000001eac68] .netif_receive_skb+0x2ec/0x3a0 > [c000000254c85650] [c0000000001eae34] .process_backlog+0x118/0x278 > [c000000254c85730] [c0000000001eb07c] .net_rx_action+0xe8/0x24c > [c000000254c857f0] [c000000000047ea8] .__do_softirq+0xdc/0x1b8 > [c000000254c858b0] [c00000000004800c] .do_softirq+0x88/0x90 > [c000000254c85940] [c0000000000480b4] .local_bh_enable+0xa0/0xa4 > [c000000254c859c0] [c0000000001ea274] .dev_queue_xmit+0x2f4/0x3a0 > [c000000254c85a60] [c0000000001f3308] .neigh_connected_output+0x104/0x1a0 > [c000000254c85b00] [c00000000020e824] .ip_finish_output2+0x100/0x2dc > [c000000254c85bc0] [c00000000020fa18] .ip_fragment+0x4d8/0x800 > [c000000254c85cd0] [c00000000024f768] .ip_refrag+0xa4/0xb0 > [c000000254c85d70] [c0000000001fa2a4] .nf_iterate+0x128/0x190 > [c000000254c85e30] [c0000000001fa86c] .nf_hook_slow+0xc8/0x1bc > [c000000254c85f10] [c00000000020ebd8] .ip_finish_output+0x1d8/0x36c > [c000000254c85fd0] [c00000000020fa18] .ip_fragment+0x4d8/0x800 > [c000000254c860e0] [c00000000020e1e4] .dst_output+0x4c/0x80 > [c000000254c86170] [c0000000001fa928] .nf_hook_slow+0x184/0x1bc > [c000000254c86250] [c00000000021162c] .ip_push_pending_frames+0x4ec/0x550 > [c000000254c86300] [c0000000002353b8] > .udp_push_pending_frames+0x16c/0x260 > [c000000254c863c0] [c000000000235af0] .udp_sendmsg+0x644/0x758 > [c000000254c86570] [c00000000023fdd0] .inet_sendmsg+0x88/0xb8 > [c000000254c86610] [c0000000001dba14] .sock_sendmsg+0xdc/0x13c > [c000000254c86810] [c0000000001dbab0] .kernel_sendmsg+0x3c/0x64 > [c000000254c868a0] [c0000000001dff94] .sock_no_sendpage+0x9c/0xb8 > [c000000254c86970] [c000000000235d88] .udp_sendpage+0x184/0x200 > [c000000254c86a60] [c00000000023fe9c] .inet_sendpage+0x9c/0x108 > [c000000254c86b10] [c00000000028cd24] .xdr_sendpages+0x280/0x34c > [c000000254c86c80] [c00000000027fad0] .xprt_transmit+0x170/0x550 > [c000000254c86d50] [c00000000027cfd4] .call_transmit+0x1cc/0x2bc > [c000000254c86e00] [c0000000002822fc] .__rpc_execute+0xf4/0x524 > [c000000254c86f60] [c0000000000ff690] .nfs_execute_write+0x50/0x88 > [c000000254c87000] [c0000000000ffbc0] .nfs_flush_list+0x4f8/0x644 > [c000000254c87120] [c0000000000ffe48] .nfs_flush_inode+0x13c/0x148 > [c000000254c871f0] [c0000000001012e8] .nfs_writepages+0x10c/0x200 > [c000000254c872a0] [c000000000073858] .do_writepages+0x60/0x8c > [c000000254c87320] [c0000000000c41c8] > .__writeback_single_inode+0xa8/0x4b8 > [c000000254c87460] [c0000000000c4de8] .sync_sb_inodes+0x224/0x37c > [c000000254c87530] [c0000000000c529c] .writeback_inodes+0x19c/0x1ac > [c000000254c875d0] [c00000000007302c] > .balance_dirty_pages_ratelimited+0xe8/0x254 > [c000000254c87700] [c00000000006dd90] > .generic_file_buffered_write+0x38c/0x7b4 > [c000000254c878c0] [c00000000006e71c] > .__generic_file_aio_write_nolock+0x2b0/0x498 > [c000000254c87a00] [c00000000006e9a0] .generic_file_aio_write+0x9c/0x1b0 > [c000000254c87ac0] [c0000000000f37bc] .nfs_file_write+0xbc/0x158 > [c000000254c87b70] [c000000000091b48] .do_sync_write+0xcc/0x140 > [c000000254c87ce0] [c000000000091d40] .vfs_write+0x184/0x18c > [c000000254c87d90] [c000000000091e38] .sys_write+0x54/0x9c > Instruction dump: > fbe1fff8 ebc2a958 f8010010 f821ff81 60000000 e9230008 7c3f0b78 e91e8000 > e97e8008 80090010 7d6a5b78 78001f24 <7d28002a> 7d69502e 2fab0000 7c000026 > From tduffy at sun.com Wed Jun 8 14:04:03 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 14:04:03 -0700 Subject: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: <1118263490.9977.2.camel@duffman> References: <1118093694.5735.13.camel@duffman> <1118101725.30742.2.camel@duffman> <1118261806.31987.21.camel@duffman> <1118263490.9977.2.camel@duffman> Message-ID: <1118264643.9977.10.camel@duffman> On Wed, 2005-06-08 at 13:44 -0700, Tom Duffy wrote: > On Wed, 2005-06-08 at 16:36 -0400, James Lentini wrote: > > Do you see any additional stability problems after applying this? I'm > > updating my OpenIB tree to see if that is my problem. > > This just in! I think your patch is fine, at least it doesn't introduce any new bugs, because after a reboot of both machines, restarting the SM, the quit test and the transaction test work fine. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From rep.nop at aon.at Wed Jun 8 14:23:02 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Wed, 8 Jun 2005 23:23:02 +0200 Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <1118263967.9977.5.camel@duffman> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> <42A7347B.8060100@ichips.intel.com> <20050608194831.GB5412@aon.at> <1118261981.7368.1.camel@duffman> <20050608204157.GD5412@aon.at> <1118263967.9977.5.camel@duffman> Message-ID: <20050608212302.GE5412@aon.at> On Wed, Jun 08, 2005 at 01:52:47PM -0700, Tom Duffy wrote: >On Wed, 2005-06-08 at 22:41 +0200, Bernhard Fischer wrote: >> On Wed, Jun 08, 2005 at 01:19:41PM -0700, Tom Duffy wrote: >> >On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote: >> >> >> 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally >> >> would appreciate it, since these variants are harder to read for me. >> > >> >What do you want to see instead? >> > >> >if (this) { >> > /* do stuff */ >> >} >> >> yes, this. > >I think James would take these types of patches. > >In any event, going through the code with fine toothed comb will happen >after all the major upheavals, I think. *I* want to fix all the major >architectural issues that are holding up acceptance before nit-picking. certainly. > >If I have the time or inclination, I will clean up these things. -------------- next part -------------- Index: TODO =================================================================== --- TODO (revision 2572) +++ TODO (working copy) @@ -23,3 +23,7 @@ IB DAT PROVIDER *) cache remote peer's address so that it does not need to be retrieved in dapl_ep_query. *) Replace dapl linked lists with Linux linked lists +*) referencing + http://openib.org/pipermail/openib-general/2005-January/003149.html + and only then replace the checks turned up by + 'grep -r -E "NULL.*!=|!=.*NULL" *' with if(foo)... for better readability From jlentini at netapp.com Wed Jun 8 14:30:01 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 17:30:01 -0400 (EDT) Subject: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: <1118264643.9977.10.camel@duffman> References: <1118093694.5735.13.camel@duffman> <1118101725.30742.2.camel@duffman> <1118261806.31987.21.camel@duffman> <1118263490.9977.2.camel@duffman> <1118264643.9977.10.camel@duffman> Message-ID: The new problem that I'm seeing is below. I don't think that this patch caused this though, so I'll go ahead and commit your patch with my modifications. The oops I see is below. By my calculations, the crash is on line 450 of mthca_cq.c. That line is: entry->wr_id = (*cur_qp)->wrid[wqe_index]; and the resulting instruction that fails is b98: 8b 54 d8 04 mov 0x4(%eax,%ebx,8),%edx Does anyone know which part of the C statement that is? Unable to handle kernel paging request at virtual address 00002014 printing eip: e0a65008 *pde = 1814d067 Oops: 0000 [#1] Modules linked in: kdapltest ib_dat_provider dat ib_cm ib_at ib_ipoib ib_sa md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc i2c_piix4 i2c_core ib_mthca ib_mad ib_core e100 mii floppy sg aic7xxx sd_mod scsi_mod CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010046 (2.6.11-openib) EIP is at mthca_poll_cq+0x368/0x760 [ib_mthca] eax: 00000000 ebx: 00000402 ecx: 00000000 edx: dd6ea000 esi: cdbf7000 edi: cdbf7058 ebp: c0469cc8 esp: c0469c5c ds: 007b es: 007b ss: 0068 Process swapper (pid: 0, threadinfo=c0469000 task=c03bfc20) Stack: 00000000 c046cf1f 00000030 00000000 00000000 00000000 00000000 00000000 cdbf5040 00000000 00000000 00000000 c0469d04 00000000 00000086 dd8af000 c0469d04 00000001 d1a8d9c0 00000000 00000046 00000001 00000000 cdbf7000 Call Trace: [] show_stack+0x7a/0x90 [] show_registers+0x149/0x1c0 [] die+0x126/0x2a0 [] do_page_fault+0x45e/0x644 [] error_code+0x2b/0x30 [] dapl_ib_completion_poll+0x3c/0xd1 [ib_dat_provider] [] dapl_evd_cq_poll_to_event+0x17/0x3b [ib_dat_provider] [] dapl_evd_dequeue+0x277/0x34b [ib_dat_provider] [] dapl_evd_upcall_trigger+0x34/0x66 [ib_dat_provider] [] dapl_evd_dto_callback+0xd4/0xea [ib_dat_provider] [] mthca_cq_event+0x33/0x80 [ib_mthca] [] mthca_eq_int+0x3a4/0x580 [ib_mthca] [] mthca_tavor_interrupt+0x81/0x350 [ib_mthca] [] handle_IRQ_event+0x35/0x70 [] __do_IRQ+0x108/0x340 [] do_IRQ+0x96/0xa0 [] common_interrupt+0x1a/0x20 [] handle_IRQ_event+0x35/0x70 [] __do_IRQ+0x108/0x340 [] do_IRQ+0x5a/0xa0 ======================= [] common_interrupt+0x1a/0x20 [] cpu_idle+0x57/0x60 [] rest_init+0x19/0x20 [] start_kernel+0x17a/0x1f0 [] 0xc010019f Code: 00 00 00 8b 55 b4 0f b6 52 1d 81 e2 80 00 00 00 e9 d3 fd ff ff 8b 45 b4 8d 7e 58 8b 4f 34 8b 58 18 8b 86 e0 00 00 00 0f cb d3 eb <8b> 54 d8 04 8b 04 d8 e9 36 fe ff ff 8b 55 b4 8b 4d c4 8b 42 14 On Wed, 8 Jun 2005, Tom Duffy wrote: > On Wed, 2005-06-08 at 13:44 -0700, Tom Duffy wrote: >> On Wed, 2005-06-08 at 16:36 -0400, James Lentini wrote: >>> Do you see any additional stability problems after applying this? I'm >>> updating my OpenIB tree to see if that is my problem. >> >> This just in! > > I think your patch is fine, at least it doesn't introduce any new bugs, > because after a reboot of both machines, restarting the SM, the quit > test and the transaction test work fine. > > -tduffy > From jlentini at netapp.com Wed Jun 8 14:41:37 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 17:41:37 -0400 (EDT) Subject: [openib-general] Re: [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <1118253907.31223.2.camel@duffman> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> Message-ID: Committed in revision 2573. From jlentini at netapp.com Wed Jun 8 14:50:20 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 17:50:20 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1118246133.19775.9.camel@duffman> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118246133.19775.9.camel@duffman> Message-ID: That would be another way to interpret this: that each time the CM times out, DAPL should re-attempt to establish a connection. However, the original implementation didn't do this and the feedback I've received is that such a feature would not be appropriate for the kernel. If there are kernel applications that want this, we can add it. On Wed, 8 Jun 2005, Tom Duffy wrote: tduffy> On Wed, 2005-06-08 at 11:44 -0400, James Lentini wrote: tduffy> > We interpreted the above to mean "give the connection protocol as tduffy> > much time as it needs to establish a connection, but don't mask tduffy> > errors (no path to the remove node, etc.)". For that reason we changed tduffy> > the variable name to DAT_TIMEOUT_MAX. tduffy> tduffy> Well, let's say the end node is not there yet. Should the CM keep tduffy> trying indefinitely waiting for somebody to show up and respond? tduffy> tduffy> -tduffy tduffy> From bjordan.ics at gmail.com Wed Jun 8 14:53:10 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Wed, 8 Jun 2005 17:53:10 -0400 Subject: [openib-general] shutdown issue with SDP In-Reply-To: <20050608124650.A28180@topspin.com> References: <78d18e20506081029183adcfc@mail.gmail.com> <20050608124650.A28180@topspin.com> Message-ID: <78d18e2050608145322cdb574@mail.gmail.com> On 6/8/05, Libor Michalek wrote: > On Wed, Jun 08, 2005 at 01:29:42PM -0400, William Jordan wrote: > > I'm trying to run netperf over SDP using libsdp. I'm having > > intermittent success, but usually, I get a failure. The client does a > > shutdown (SHUT_WR), but the server gets an ECONNRESET, "Connection > > reset by peer", on the recv instead of a zero length recv. > > > > I'm running 2568. > > > > Anyone seen similar/different results? > > Bill, > > Can you try the patch I sent out yesterday? It significantly simplifies > the connection state machine within SDP. I ran it successfully with a > number of the netperf tests. (TCP_CC, TCP_CRR, TCP_RR, TCP_STREAM) > > Thanks. > > -Libor > That appears to have fixed it. Thanks. -- Bill Jordan SilverStorm Technologies From jlentini at netapp.com Wed Jun 8 14:53:56 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 17:53:56 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1118245813.4477.92.camel@localhost.localdomain> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> Message-ID: On Wed, 8 Jun 2005, Hal Rosenstock wrote: halr> On Wed, 2005-06-08 at 11:44, James Lentini wrote: halr> > We interpreted the above to mean "give the connection protocol as halr> > much time as it needs to establish a connection, but don't mask halr> > errors (no path to the remove node, etc.)". For that reason we changed halr> > the variable name to DAT_TIMEOUT_MAX. halr> halr> But if the REQ is lost, the timeout is really really long (longer than halr> most will wait for an error). If a user doesn't want to wait DAT_TIMEOUT_MAX time, it can pass a smaller amount of time to dat_ep_connect. Does this satisfy your requirements? halr> Transaction test also appears to be using this as well as the halr> quit test. halr> halr> -- Hal halr> From jlentini at netapp.com Wed Jun 8 14:55:02 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 17:55:02 -0400 (EDT) Subject: [openib-general] [PATCHv4] kdapl: remove use of HANDLE's (vs. r2572) In-Reply-To: <1118263967.9977.5.camel@duffman> References: <1117765223.4119.15.camel@duffman> <1118253907.31223.2.camel@duffman> <42A7347B.8060100@ichips.intel.com> <20050608194831.GB5412@aon.at> <1118261981.7368.1.camel@duffman> <20050608204157.GD5412@aon.at> <1118263967.9977.5.camel@duffman> Message-ID: On Wed, 8 Jun 2005, Tom Duffy wrote: > On Wed, 2005-06-08 at 22:41 +0200, Bernhard Fischer wrote: >> On Wed, Jun 08, 2005 at 01:19:41PM -0700, Tom Duffy wrote: >>> On Wed, 2005-06-08 at 21:48 +0200, Bernhard Fischer wrote: >> >>>> 'if (NULL != this)' and 'if (0 != that)' occurances, too? I personally >>>> would appreciate it, since these variants are harder to read for me. >>> >>> What do you want to see instead? >>> >>> if (this) { >>> /* do stuff */ >>> } >> >> yes, this. > > I think James would take these types of patches. I would. > In any event, going through the code with fine toothed comb will happen > after all the major upheavals, I think. *I* want to fix all the major > architectural issues that are holding up acceptance before nit-picking. > > If I have the time or inclination, I will clean up these things. > > -tduffy > From jlentini at netapp.com Wed Jun 8 14:58:33 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 8 Jun 2005 17:58:33 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <42A7193F.4000003@ichips.intel.com> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <42A7193F.4000003@ichips.intel.com> Message-ID: On Wed, 8 Jun 2005, Sean Hefty wrote: > Hal Rosenstock wrote: >> On Wed, 2005-06-08 at 11:44, James Lentini wrote: >> >>> We interpreted the above to mean "give the connection protocol as much >>> time as it needs to establish a connection, but don't mask errors (no path >>> to the remove node, etc.)". For that reason we changed the variable name >>> to DAT_TIMEOUT_MAX. >> >> >> But if the REQ is lost, the timeout is really really long (longer than >> most will wait for an error). Transaction test also appears to be using >> this as well as the quit test. > > My interpretation was that this is a DAPL level timeout and did not > necessarily relate to a timeout for a single CM REQ. That is, there could > still be a different timeout specified to the CM, but the number of retries > could be infinite. If there are kernel users in need of an truly inifinite timeout, we could do that. > Note that I'm not saying that an infinite timeout makes sense, but the use of > TIMEOUT_MAX seems reasonable. To me that indicates that DAPL decides how > long is needed to establish a timeout, and it manages all retries. > > - Sean > From tduffy at sun.com Wed Jun 8 15:54:34 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 15:54:34 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query() Message-ID: <1118271274.9977.41.camel@duffman> I am putting this out there to see what people think about this sort of change. This only does it for one of the functions in dat provider, but I will change it for all if this is accepted. Basically, instead of using a redirected function wrapper, it calls the function directly from the struct (how the rest of the kernel does it). And it gets rid of the typedef DAT_IA_QUERY_FUNC. Signed-off-by: Tom Duffy dat-provider/dapl_provider.c | 2 +- dat/api.c | 4 ++-- dat/dat.h | 15 ++------------- test/dapltest/test/dapl_fft_queryinfo.c | 18 ++++++------------ test/dapltest/test/dapl_performance_util.c | 6 ++---- test/dapltest/test/dapl_test_util.c | 7 +++---- 6 files changed, 16 insertions(+), 36 deletions(-) Index: linux-kernel-return/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_performance_util.c (revision 2573) +++ linux-kernel-return/test/dapltest/test/dapl_performance_util.c (working copy) @@ -64,10 +64,8 @@ DT_Performance_Test_Create ( test_ptr->ia = ia; test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; - ret = dat_ia_query (test_ptr->ia, - NULL, - &test_ptr->ia_attr, - NULL); + ret = test_ptr->ia->common.provider->dat_ia_query(test_ptr->ia, NULL, + &test_ptr->ia_attr, NULL); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ia_query error: %s\n", Index: linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (revision 2573) +++ linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -165,10 +165,8 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_ia_query (ia, - &evd, - &ia_attributes, - &provider_attributes); + rc = ia->common.provider->dat_ia_query(ia, &evd, &ia_attributes, + &provider_attributes); } else if (result_wanted == DAT_INVALID_PARAMETER) { @@ -177,17 +175,13 @@ int DT_queryinfo_basic (Params_t *params * NULL out ia_attr and for the DAT_IA_ATTR_MASK to * have values */ - rc = dat_ia_query (ia, - &evd, - NULL, - &provider_attributes); + rc = ia->common.provider->dat_ia_query(ia, &evd, NULL, + &provider_attributes); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_ia_query (ia, - &evd, - &ia_attributes, - &provider_attributes); + rc = ia->common.provider->dat_ia_query(ia, &evd, &ia_attributes, + &provider_attributes); } } Index: linux-kernel-return/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_test_util.c (revision 2573) +++ linux-kernel-return/test/dapltest/test/dapl_test_util.c (working copy) @@ -44,10 +44,9 @@ DT_query ( Per_Test_Data_t *pt_ptr, phead = pt_ptr->Params.phead; /* Query the IA */ - ret = dat_ia_query (ia, - &async_evd_hdl, - &pt_ptr->ia_attr, - &pt_ptr->provider_attr); + ret = ia->common.provider->dat_ia_query(ia, &async_evd_hdl, + &pt_ptr->ia_attr, + &pt_ptr->provider_attr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_query error: %s\n", Index: linux-kernel-return/dat-provider/dapl_provider.c =================================================================== --- linux-kernel-return/dat-provider/dapl_provider.c (revision 2573) +++ linux-kernel-return/dat-provider/dapl_provider.c (working copy) @@ -67,7 +67,7 @@ static struct dat_provider g_dapl_provid .extension = NULL, .ia_open_func = &dapl_ia_open, - .ia_query_func = &dapl_ia_query, + .dat_ia_query = &dapl_ia_query, .ia_close_func = &dapl_ia_close, .ia_memtype_hint_func = &dapl_ia_memtype_hint, Index: linux-kernel-return/dat/api.c =================================================================== --- linux-kernel-return/dat/api.c (revision 2573) +++ linux-kernel-return/dat/api.c (working copy) @@ -405,8 +405,8 @@ u32 dat_ia_close(struct dat_ia *ia, enum provider = ia->common.provider; ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, - &provider_attr))) + if (DAT_SUCCESS != (status = ia->common.provider->dat_ia_query(ia, NULL, NULL, + &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) Index: linux-kernel-return/dat/dat.h =================================================================== --- linux-kernel-return/dat/dat.h (revision 2573) +++ linux-kernel-return/dat/dat.h (working copy) @@ -1057,10 +1057,6 @@ typedef u32 (*DAT_IA_OPENV_FUNC)(const c typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum dat_close_flags); -typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, - struct dat_ia_attr *, - struct dat_provider_attr *); - typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct dat_cr_param *); typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct dat_ep *, int, @@ -1223,7 +1219,8 @@ struct dat_provider { void *extension; DAT_IA_OPEN_FUNC ia_open_func; - DAT_IA_QUERY_FUNC ia_query_func; + u32 (*dat_ia_query)(struct dat_ia *, struct dat_evd **, + struct dat_ia_attr *, struct dat_provider_attr *); DAT_IA_CLOSE_FUNC ia_close_func; DAT_IA_MEMTYPE_HINT_FUNC ia_memtype_hint_func; @@ -1396,14 +1393,6 @@ static inline u32 dat_ia_memtype_hint(st preferred_alignment); } -static inline u32 dat_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, - struct dat_ia_attr *ia_attr, - struct dat_provider_attr *provider_attr) -{ - return DAT_CALL_PROVIDER_FUNC( - ia_query_func, ia, async_evd, ia_attr, provider_attr); -} - static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data) { From iod00d at hp.com Wed Jun 8 16:40:57 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 8 Jun 2005 16:40:57 -0700 Subject: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050608055030.GA7809@mellanox.co.il> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> <20050601232123.GN25321@esmail.cup.hp.com> <20050604213422.GA23153@mellanox.co.il> <20050608002426.GI15489@esmail.cup.hp.com> <20050608055030.GA7809@mellanox.co.il> Message-ID: <20050608234057.GC20219@esmail.cup.hp.com> On Wed, Jun 08, 2005 at 08:50:30AM +0300, Michael S. Tsirkin wrote: > > The numbers (MByte/cycles) don't mean anything to me. > > Can we report this as cycles/MB (or KB) instead? (aka Service Demand) > > Sure. Patch? Appended. Please apply. > > The other metric I care about is MB/s. > > Dont pass -C then. Doh! Of course. -C makes no sense in this context. It's removed in the patch below. Here is a sample output: grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw -b -n 10000 -s 32768 10.0.0.51 local address: LID 0x06, QPN 0x280406, PSN 0x929ad8 RKey 0xf00434 VAddr 0x600000000001c000 remote address: LID 0x04, QPN 0x270406, PSN 0x61c246, RKey 0xea0434 VAddr 0x600000000001c000 Bandwidth peak (#0 to #7080): 722.421 MB/sec Bandwidth average: 721.997 MB/sec Service Demand peak (#0 to #7080): 2027 cycles/KB Service Demand Avg : 2028 cycles/KB thanks, grant Index: rdma_bw.c =================================================================== --- rdma_bw.c (revision 2570) +++ rdma_bw.c (working copy) @@ -61,10 +61,6 @@ static int page_size; -struct report_options { - int cycles; /* report delta's in cycles, not microsec's */ -}; - struct pingpong_context { struct ibv_context *context; struct ibv_pd *pd; @@ -422,19 +418,14 @@ static void usage(const char *argv0) printf(" -t, --tx-depth= size of tx queue (default 100)\n"); printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); - printf(" -C, --report-cycles report times in cpu cycle units (default seconds)\n"); - printf(" -H, --report-histogram print out all results (default print summary only)\n"); - printf(" -U, --report-unsorted (implies -H) print out unsorted results (default sorted)\n"); } -static void print_report(struct report_options * options, - unsigned int iters, double size, int duplex, +static void print_report(unsigned int iters, int size, int duplex, cycles_t *tposted, cycles_t *tcompleted) { double cycles_to_units; - double tsize; /* Transferred size, in megabytes */ + unsigned long tsize; /* Transferred size, in megabytes */ int i, j; - const char* units; int opt_posted = 0, opt_completed = 0; cycles_t opt_delta; cycles_t t; @@ -453,19 +444,21 @@ static void print_report(struct report_o } } - if (options->cycles) { - cycles_to_units = 1; - units = "cycles"; - } else { - cycles_to_units = get_cpu_mhz() * 1000000; - units = "sec"; - } + cycles_to_units = get_cpu_mhz() * 1000000; tsize = duplex ? 2 : 1; - tsize = tsize * size / 0x100000; + tsize = tsize * size / 1024; - printf("Bandwidth peak (#%d to #%d): %g MByte/%s\n", opt_posted, opt_completed, tsize * cycles_to_units / opt_delta, units); - printf("Bandwidth average: %g MByte/%s\n", tsize * iters * cycles_to_units / (tcompleted[iters - 1] - tposted[0]), units); + printf("Bandwidth peak (#%d to #%d): %g MB/sec\n", + opt_posted, opt_completed, + tsize * cycles_to_units / opt_delta / 1024); + printf("Bandwidth average: %g MB/sec\n", + tsize * iters * cycles_to_units / (tcompleted[iters - 1] - tposted[0]) / 1024); + + printf("Service Demand peak (#%d to #%d): %ld cycles/KB\n", + opt_posted, opt_completed, opt_delta/tsize); + printf("Service Demand Avg : %ld cycles/KB\n", + (tcompleted[iters - 1] - tposted[0])/(tsize * iters)); } @@ -480,14 +473,13 @@ int main(int argc, char *argv[]) char *servername = NULL; int port = 18515; int ib_port = 1; - int size = 1; + int size = 4 * 1024; int tx_depth = 100; int iters = 1000; int scnt, ccnt; int sockfd; int duplex = 0; struct ibv_qp *qp; - struct report_options report = {}; cycles_t *tposted; cycles_t *tcompleted; @@ -504,11 +496,10 @@ int main(int argc, char *argv[]) { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "bidirectional", .has_arg = 0, .val = 'b' }, - { .name = "report-cycles", .has_arg = 0, .val = 'C' }, { 0 } }; - c = getopt_long(argc, argv, "p:d:i:s:n:t:bC", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:s:n:t:b", long_options, NULL); if (c == -1) break; @@ -565,10 +556,6 @@ int main(int argc, char *argv[]) duplex = 1; break; - case 'C': - report.cycles = 1; - break; - default: usage(argv[0]); return 1; @@ -751,7 +738,7 @@ int main(int argc, char *argv[]) write(sockfd, "done", sizeof "done"); close(sockfd); - print_report(&report, iters, size, duplex, tposted, tcompleted); + print_report(iters, size, duplex, tposted, tcompleted); free(tposted); free(tcompleted); From iod00d at hp.com Wed Jun 8 16:45:32 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 8 Jun 2005 16:45:32 -0700 Subject: [openib-general] [PATCH] rdma_lat-09 and results In-Reply-To: <20050602005228.GP25321@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> Message-ID: <20050608234532.GD20219@esmail.cup.hp.com> On Wed, Jun 01, 2005 at 05:52:28PM -0700, Grant Grundler wrote: > Michael, > > Good news: > My next cleanup of rdma_lat.c is working and patch is appended. Michael, Did you want to comment on this patch or apply it? I can resend privately it you don't have the original mail. > Summary of changes below. > > Bad News: > perf is about ~15 cycles slower since the last time I tested. > (Hrm...maybe it's time to cycle power on the TS90 switch again.) Seems with discussion on perf was distracting from the patch. I'll seperate those next time. thanks, grant > Here's with the new rdma_lat.c: > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C > local address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > remote address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > Latency typical: 7140 cycles > Latency best : 6915 cycles > Latency worst : 52915.5 cycles > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ > > And the "client" side: > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C 10.0.0.51 > local address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > remote address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > Latency typical: 7140 cycles > Latency best : 6907 cycles > Latency worst : 94920 cycles > > > The previous set of rdma_lat results are here: > http://openib.org/pipermail/openib-general/2005-May/006721.html > > I'll guess the previous SVN verion was no older than r2229. > > > I get 7140 to 7151 for the original rdma_lat. Usually 7147.5. > I get 7132 to 7155 with my version of rdma_lat. Usually 7140. > No statistically significant differences. > Both essentially agree on the higher result. > Using "-n 10000" gave more consistent results * > > I use "taskset" to bind the rdma_lat test to a CPU. > But it didn't matter which CPU I bound the task to - results > where basically the same. I suspect the "stream" mode just > does not depend on or generating that many interrupts. > > > diffstat rdma_lat.c-09-diff > rdma_lat.c | 395 +++++++++++++++++++++++++++++-------------------------------- > 1 files changed, 188 insertions(+), 207 deletions(-) > > Commit Log entry/Summary of changes: > o move device lookup from main() to pp_find_dev() > o move sockfd handling code to pp_open_port() > o consolidate server/client "key exchange" code path > o enumerate return values in main() > o fixed nit: pp_*_exch_dest was called twice. > Each time it would malloc a new "rem_dest". > Code in pp_open_port() now free()'s the first one. > > Signed-off-by: Grant Grundler > > thanks, > grant > > > > Index: rdma_lat.c > =================================================================== > --- rdma_lat.c (revision 2519) > +++ rdma_lat.c (working copy) > @@ -103,30 +103,40 @@ static uint16_t pp_get_local_lid(struct > return attr.lid; > } > > -static int pp_client_connect(const char *servername, int port) > + > +static int pp_connect_sock(const char *servername, int port) > { > struct addrinfo *res, *t; > struct addrinfo hints = { > + .ai_flags = AI_PASSIVE, /* Server only? */ > .ai_family = AF_UNSPEC, > .ai_socktype = SOCK_STREAM > }; > char *service; > - int n; > int sockfd = -1; > + int n; > > asprintf(&service, "%d", port); > n = getaddrinfo(servername, service, &hints, &res); > > if (n < 0) { > - fprintf(stderr, "%s for %s:%d\n", gai_strerror(n), servername, port); > + fprintf(stderr, "%s for %s:%d\n", gai_strerror(n), > + servername ? servername : "NULL", port); > return n; > } > > for (t = res; t; t = t->ai_next) { > sockfd = socket(t->ai_family, t->ai_socktype, t->ai_protocol); > if (sockfd >= 0) { > - if (!connect(sockfd, t->ai_addr, t->ai_addrlen)) > - break; > + if (servername) { > + if (!connect(sockfd, t->ai_addr, t->ai_addrlen)) > + break; > + } else { > + setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &n, sizeof n); > + if (!bind(sockfd, t->ai_addr, t->ai_addrlen)) > + break; > + } > + > close(sockfd); > sockfd = -1; > } > @@ -134,147 +144,100 @@ static int pp_client_connect(const char > > freeaddrinfo(res); > > - if (sockfd < 0) { > - fprintf(stderr, "Couldn't connect to %s:%d\n", servername, port); > - return sockfd; > + if (sockfd < 0) > + fprintf(stderr, "Couldn't %s to %s:%d\n", > + servername ? "connect" : "bind", > + servername ? servername : "NULL", port); > + > + /* "server" side needs to wait/listen for client to call */ > + if (!servername) { > + int connfd; > + > + listen(sockfd, 1); > + connfd = accept(sockfd, NULL, 0); > + if (connfd < 0) { > + perror("server accept"); > + fprintf(stderr, "accept() failed\n"); > + } > + close(sockfd); > + return connfd; > } > + > return sockfd; > } > > -struct pingpong_dest * pp_client_exch_dest(int sockfd, > - const struct pingpong_dest *my_dest) > +#define KEY_MSG_SIZE (sizeof "0000:000000:000000:00000000:0000000000000000") > +#define KEY_PRINT_FMT "%04x:%06x:%06x:%08x:%016Lx" > + > +static int pp_write_keys(int sockfd, const struct pingpong_dest *my_dest) > { > - struct pingpong_dest *rem_dest = NULL; > - char msg[sizeof "0000:000000:000000:00000000:0000000000000000"]; > - int parsed; > + char msg[KEY_MSG_SIZE]; > + > + sprintf(msg, KEY_PRINT_FMT, my_dest->lid, my_dest->qpn, > + my_dest->psn, my_dest->rkey, my_dest->vaddr); > > - sprintf(msg, "%04x:%06x:%06x:%08x:%016Lx", my_dest->lid, my_dest->qpn, > - my_dest->psn,my_dest->rkey,my_dest->vaddr); > if (write(sockfd, msg, sizeof msg) != sizeof msg) { > perror("client write"); > fprintf(stderr, "Couldn't send local address\n"); > - goto out; > + return 0; > } > > + return 1; > +} > + > +static struct pingpong_dest * pp_read_keys(int sockfd, const struct pingpong_dest *my_dest) > +{ > + struct pingpong_dest *rem_dest = NULL; > + int parsed; > + char msg[KEY_MSG_SIZE]; > + > if (read(sockfd, msg, sizeof msg) != sizeof msg) { > - perror("client read"); > + perror("pp_read_keys"); > fprintf(stderr, "Couldn't read remote address\n"); > - goto out; > + return NULL; > } > > rem_dest = malloc(sizeof *rem_dest); > if (!rem_dest) > - goto out; > + return NULL; > > - parsed = sscanf(msg, "%x:%x:%x:%x:%Lx", &rem_dest->lid, &rem_dest->qpn, > - &rem_dest->psn,&rem_dest->rkey,&rem_dest->vaddr); > + parsed = sscanf(msg, KEY_PRINT_FMT, &rem_dest->lid, &rem_dest->qpn, > + &rem_dest->psn, &rem_dest->rkey, &rem_dest->vaddr); > > if (parsed != 5) { > - fprintf(stderr, "Couldn't parse line <%.*s>\n",(int)sizeof msg, > - msg); > + fprintf(stderr, "Couldn't parse line <%.*s>\n", > + (int)sizeof msg, msg); > free(rem_dest); > - rem_dest = NULL; > - goto out; > - } > -out: > - return rem_dest; > -} > - > -int pp_server_connect(int port) > -{ > - struct addrinfo *res, *t; > - struct addrinfo hints = { > - .ai_flags = AI_PASSIVE, > - .ai_family = AF_UNSPEC, > - .ai_socktype = SOCK_STREAM > - }; > - char *service; > - int sockfd = -1, connfd; > - int n; > - > - asprintf(&service, "%d", port); > - n = getaddrinfo(NULL, service, &hints, &res); > - > - if (n < 0) { > - fprintf(stderr, "%s for port %d\n", gai_strerror(n), port); > - return n; > - } > - > - for (t = res; t; t = t->ai_next) { > - sockfd = socket(t->ai_family, t->ai_socktype, t->ai_protocol); > - if (sockfd >= 0) { > - n = 1; > - > - setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &n, sizeof n); > - > - if (!bind(sockfd, t->ai_addr, t->ai_addrlen)) > - break; > - close(sockfd); > - sockfd = -1; > - } > - } > - > - freeaddrinfo(res); > - > - if (sockfd < 0) { > - fprintf(stderr, "Couldn't listen to port %d\n", port); > - return sockfd; > - } > - > - listen(sockfd, 1); > - connfd = accept(sockfd, NULL, 0); > - if (connfd < 0) { > - perror("server accept"); > - fprintf(stderr, "accept() failed\n"); > - close(sockfd); > - return connfd; > + return NULL; > } > > - close(sockfd); > - return connfd; > + return rem_dest; > } > > -static struct pingpong_dest *pp_server_exch_dest(int connfd, const struct pingpong_dest *my_dest) > +static struct pingpong_dest * pp_exch_dest(int sockfd, const char *servername, > + const struct pingpong_dest *my_dest) > { > - char msg[sizeof "0000:000000:000000:00000000:0000000000000000"]; > struct pingpong_dest *rem_dest = NULL; > - int parsed; > - int n; > - > - n = read(connfd, msg, sizeof msg); > - if (n != sizeof msg) { > - perror("server read"); > - fprintf(stderr, "%d/%d: Couldn't read remote address\n", n, (int) sizeof msg); > - goto out; > - } > > - rem_dest = malloc(sizeof *rem_dest); > - if (!rem_dest) > - goto out; > - > - parsed = sscanf(msg, "%x:%x:%x:%x:%Lx", &rem_dest->lid, &rem_dest->qpn, > - &rem_dest->psn, &rem_dest->rkey, &rem_dest->vaddr); > - if (parsed != 5) { > - fprintf(stderr, "Couldn't parse line <%.*s>\n",(int)sizeof msg, > - msg); > - free(rem_dest); > - rem_dest = NULL; > - goto out; > + if (servername) { > + if (!pp_write_keys(sockfd, my_dest)) > + goto exch_failed; > + rem_dest = pp_read_keys(sockfd, my_dest); > + if (!rem_dest) > + goto exch_failed; > + } else { > + rem_dest = pp_read_keys(sockfd, my_dest); > + if (!rem_dest) > + goto exch_failed; > + if (!pp_write_keys(sockfd, my_dest)) > + goto exch_failed; > } > > - sprintf(msg, "%04x:%06x:%06x:%08x:%016Lx", my_dest->lid, my_dest->qpn, > - my_dest->psn, my_dest->rkey, my_dest->vaddr); > - if (write(connfd, msg, sizeof msg) != sizeof msg) { > - perror("server write"); > - fprintf(stderr, "Couldn't send local address\n"); > - free(rem_dest); > - rem_dest = NULL; > - goto out; > - } > -out: > +exch_failed: > return rem_dest; > } > > + > static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size, > int tx_depth, int port) > { > @@ -424,6 +387,86 @@ static int pp_connect_ctx(struct pingpon > return 0; > } > > +static struct ibv_device * pp_find_dev( const char *ib_devname) > +{ > + struct dlist *dev_list; > + struct ibv_device *ib_dev = NULL; > + > + dev_list = ibv_get_devices(); > + > + dlist_start(dev_list); > + if (!ib_devname) { > + ib_dev = dlist_next(dev_list); > + if (!ib_dev) > + fprintf(stderr, "No IB devices found\n"); > + } else { > + dlist_for_each_data(dev_list, ib_dev, struct ibv_device) > + if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) > + break; > + if (!ib_dev) > + fprintf(stderr, "IB device %s not found\n", ib_devname); > + } > + return ib_dev; > +} > + > + > +static struct pingpong_dest * pp_open_port(struct pingpong_context *ctx, > + const char * servername, int ib_port, int port) > +{ > + char addr_fmt[] = "%8s address: LID %#04x QPN %#06x PSN %#06x RKey %#08x VAddr %#016Lx\n"; > + struct pingpong_dest my_dest; > + struct pingpong_dest *rem_dest; > + int sockfd; > + > + > + /* Create connection between client and server. > + * We do it by exchanging data over a TCP socket connection. */ > + > + my_dest.lid = pp_get_local_lid(ctx, ib_port); > + my_dest.qpn = ctx->qp->qp_num; > + my_dest.psn = lrand48() & 0xffffff; > + if (!my_dest.lid) { > + fprintf(stderr, "Local lid 0x0 detected. Is an SM running?\n"); > + return NULL; > + } > + my_dest.rkey = ctx->mr->rkey; > + my_dest.vaddr = (uintptr_t)ctx->buf + ctx->size; > + > + printf(addr_fmt, "local", my_dest.lid, my_dest.qpn, my_dest.psn, > + my_dest.rkey, my_dest.vaddr); > + > + > + sockfd = pp_connect_sock(servername, port); > + if (sockfd < 0) { > + printf("pp_connect_sock(%s,%d) failed (%d)!\n", > + servername, port, sockfd); > + return NULL; > + } > + > + rem_dest = pp_exch_dest(sockfd, servername, &my_dest); > + if (!rem_dest) > + return NULL; > + > + printf(addr_fmt, "remote", rem_dest->lid, rem_dest->qpn, rem_dest->psn, > + rem_dest->rkey, rem_dest->vaddr); > + > + if (pp_connect_ctx(ctx, ib_port, my_dest.psn, rem_dest)) > + return NULL; > + > + /* An additional handshake is required *after* moving qp to RTR. > + * Arbitrarily reuse exch_dest for this purpose. > + */ > + > + free(rem_dest); > + rem_dest = pp_exch_dest(sockfd, servername, &my_dest); > + > + write(sockfd, "done", sizeof "done"); > + close(sockfd); > + > + return rem_dest; > +} > + > + > static void usage(const char *argv0) > { > printf("Usage:\n"); > @@ -518,27 +561,26 @@ static void print_report(struct report_o > > int main(int argc, char *argv[]) > { > - struct dlist *dev_list; > struct ibv_device *ib_dev; > struct pingpong_context *ctx; > - struct pingpong_dest my_dest; > struct pingpong_dest *rem_dest; > char *ib_devname = NULL; > char *servername = NULL; > - int port = 18515; > - int ib_port = 1; > - int size = 1; > - int tx_depth = 50; > - int iters = 1000; > - int scnt, rcnt, ccnt; > - int sockfd; > + > struct ibv_qp *qp; > struct ibv_send_wr *wr; > volatile char *poll_buf; > volatile char *post_buf; > - struct report_options report = {}; > > - cycles_t *tstamp; > + int port = 18515; > + int ib_port = 1; > + int size = 1; > + int iters = 1000; > + int tx_depth = 50; > + int scnt, rcnt, ccnt; > + > + static struct report_options report = {}; > + static cycles_t *tstamp; > > /* Parameter parsing. */ > while (1) { > @@ -578,25 +620,25 @@ int main(int argc, char *argv[]) > ib_port = strtol(optarg, NULL, 0); > if (ib_port < 0) { > usage(argv[0]); > - return 1; > + return 2; > } > break; > > case 's': > size = strtol(optarg, NULL, 0); > - if (size < 1) { usage(argv[0]); return 1; } > + if (size < 1) { usage(argv[0]); return 3; } > break; > > case 't': > tx_depth = strtol(optarg, NULL, 0); > - if (tx_depth < 1) { usage(argv[0]); return 1; } > + if (tx_depth < 1) { usage(argv[0]); return 4; } > break; > > case 'n': > iters = strtol(optarg, NULL, 0); > if (iters < 2) { > usage(argv[0]); > - return 1; > + return 5; > } > > break; > @@ -615,7 +657,7 @@ int main(int argc, char *argv[]) > > default: > usage(argv[0]); > - return 1; > + return 5; > } > } > > @@ -623,90 +665,26 @@ int main(int argc, char *argv[]) > servername = strdupa(argv[optind]); > else if (optind < argc) { > usage(argv[0]); > - return 1; > + return 6; > } > > > - /* Done with parameter parsing. Perform setup. */ > + /* > + * Done with parameter parsing. Perform setup. > + */ > > srand48(getpid() * time(NULL)); > - > page_size = sysconf(_SC_PAGESIZE); > > - dev_list = ibv_get_devices(); > - > - dlist_start(dev_list); > - if (!ib_devname) { > - ib_dev = dlist_next(dev_list); > - if (!ib_dev) { > - fprintf(stderr, "No IB devices found\n"); > - return 1; > - } > - } else { > - dlist_for_each_data(dev_list, ib_dev, struct ibv_device) > - if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) > - break; > - if (!ib_dev) { > - fprintf(stderr, "IB device %s not found\n", ib_devname); > - return 1; > - } > - } > + ib_dev = pp_find_dev(ib_devname); > > ctx = pp_init_ctx(ib_dev, size, tx_depth, ib_port); > if (!ctx) > - return 1; > - > - /* Create connection between client and server. > - * We do it by exchanging data over a TCP socket connection. */ > - > - my_dest.lid = pp_get_local_lid(ctx, ib_port); > - my_dest.qpn = ctx->qp->qp_num; > - my_dest.psn = lrand48() & 0xffffff; > - if (!my_dest.lid) { > - fprintf(stderr, "Local lid 0x0 detected. Is an SM running?\n"); > - return 1; > - } > - my_dest.rkey = ctx->mr->rkey; > - my_dest.vaddr = (uintptr_t)ctx->buf + ctx->size; > - > - printf(" local address: LID %#04x, QPN %#06x, PSN %#06x " > - "RKey %#08x VAddr %#016Lx\n", > - my_dest.lid, my_dest.qpn, my_dest.psn, > - my_dest.rkey, my_dest.vaddr); > - > - if (servername) { > - sockfd = pp_client_connect(servername, port); > - if (sockfd < 0) > - return 1; > - rem_dest = pp_client_exch_dest(sockfd, &my_dest); > - } else { > - sockfd = pp_server_connect(port); > - if (sockfd < 0) > - return 1; > - rem_dest = pp_server_exch_dest(sockfd, &my_dest); > - } > + return 7; > > + rem_dest = pp_open_port(ctx, servername, ib_port, port); > if (!rem_dest) > - return 1; > - > - printf(" remote address: LID %#04x, QPN %#06x, PSN %#06x, " > - "RKey %#08x VAddr %#016Lx\n", > - rem_dest->lid, rem_dest->qpn, rem_dest->psn, > - rem_dest->rkey, rem_dest->vaddr); > - > - if (pp_connect_ctx(ctx, ib_port, my_dest.psn, rem_dest)) > - return 1; > - > - /* An additional handshake is required *after* moving qp to RTR. > - Arbitrarily reuse exch_dest for this purpose. */ > - if (servername) { > - rem_dest = pp_client_exch_dest(sockfd, &my_dest); > - } else { > - rem_dest = pp_server_exch_dest(sockfd, &my_dest); > - } > - > - write(sockfd, "done", sizeof "done"); > - close(sockfd); > + return 8; > > wr = &ctx->wr; > ctx->list.addr = (uintptr_t) ctx->buf; > @@ -726,7 +704,7 @@ int main(int argc, char *argv[]) > > if (!tstamp) { > perror("malloc"); > - return 1; > + return 9; > } > > /* Done with setup. Start the test. */ > @@ -736,8 +714,8 @@ int main(int argc, char *argv[]) > /* Wait till buffer changes. */ > if (rcnt < iters && !(scnt < 1 && servername)) { > ++rcnt; > - while (*poll_buf != (char)rcnt) { > - } > + while (*poll_buf != (char)rcnt) > + ; > /* Here the data is already in the physical memory. > If we wanted to actually use it, we may need > a read memory barrier here. */ > @@ -751,7 +729,8 @@ int main(int argc, char *argv[]) > if (ibv_post_send(qp, wr, &bad_wr)) { > fprintf(stderr, "Couldn't post send: scnt=%d\n", > scnt); > - return 1; > + free(tstamp); > + return 10; > } > } > > @@ -765,7 +744,8 @@ int main(int argc, char *argv[]) > > if (ne < 0) { > fprintf(stderr, "poll CQ failed %d\n", ne); > - return 1; > + free(tstamp); > + return 11; > } > if (wc.status != IBV_WC_SUCCESS) { > fprintf(stderr, "Completion wth error at %s:\n", > @@ -774,7 +754,8 @@ int main(int argc, char *argv[]) > wc.status, (int) wc.wr_id); > fprintf(stderr, "scnt=%d, rcnt=%d, ccnt=%d\n", > scnt, rcnt, ccnt); > - return 1; > + free(tstamp); > + return 12; > } > } > } > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From iod00d at hp.com Wed Jun 8 16:46:37 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 8 Jun 2005 16:46:37 -0700 Subject: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050608234057.GC20219@esmail.cup.hp.com> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> <20050601232123.GN25321@esmail.cup.hp.com> <20050604213422.GA23153@mellanox.co.il> <20050608002426.GI15489@esmail.cup.hp.com> <20050608055030.GA7809@mellanox.co.il> <20050608234057.GC20219@esmail.cup.hp.com> Message-ID: <20050608234637.GE20219@esmail.cup.hp.com> On Wed, Jun 08, 2005 at 04:40:57PM -0700, Grant Grundler wrote: > Index: rdma_bw.c > =================================================================== > --- rdma_bw.c (revision 2570) > +++ rdma_bw.c (working copy) > @@ -61,10 +61,6 @@ ... Sorry - just realized I forgot the: Signed-off-by: Grant Grundler thanks, grant > > static int page_size; > > -struct report_options { > - int cycles; /* report delta's in cycles, not microsec's */ > -}; > - > struct pingpong_context { > struct ibv_context *context; > struct ibv_pd *pd; > @@ -422,19 +418,14 @@ static void usage(const char *argv0) > printf(" -t, --tx-depth= size of tx queue (default 100)\n"); > printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); > printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); > - printf(" -C, --report-cycles report times in cpu cycle units (default seconds)\n"); > - printf(" -H, --report-histogram print out all results (default print summary only)\n"); > - printf(" -U, --report-unsorted (implies -H) print out unsorted results (default sorted)\n"); > } > > -static void print_report(struct report_options * options, > - unsigned int iters, double size, int duplex, > +static void print_report(unsigned int iters, int size, int duplex, > cycles_t *tposted, cycles_t *tcompleted) > { > double cycles_to_units; > - double tsize; /* Transferred size, in megabytes */ > + unsigned long tsize; /* Transferred size, in megabytes */ > int i, j; > - const char* units; > int opt_posted = 0, opt_completed = 0; > cycles_t opt_delta; > cycles_t t; > @@ -453,19 +444,21 @@ static void print_report(struct report_o > } > } > > - if (options->cycles) { > - cycles_to_units = 1; > - units = "cycles"; > - } else { > - cycles_to_units = get_cpu_mhz() * 1000000; > - units = "sec"; > - } > + cycles_to_units = get_cpu_mhz() * 1000000; > > tsize = duplex ? 2 : 1; > - tsize = tsize * size / 0x100000; > + tsize = tsize * size / 1024; > > - printf("Bandwidth peak (#%d to #%d): %g MByte/%s\n", opt_posted, opt_completed, tsize * cycles_to_units / opt_delta, units); > - printf("Bandwidth average: %g MByte/%s\n", tsize * iters * cycles_to_units / (tcompleted[iters - 1] - tposted[0]), units); > + printf("Bandwidth peak (#%d to #%d): %g MB/sec\n", > + opt_posted, opt_completed, > + tsize * cycles_to_units / opt_delta / 1024); > + printf("Bandwidth average: %g MB/sec\n", > + tsize * iters * cycles_to_units / (tcompleted[iters - 1] - tposted[0]) / 1024); > + > + printf("Service Demand peak (#%d to #%d): %ld cycles/KB\n", > + opt_posted, opt_completed, opt_delta/tsize); > + printf("Service Demand Avg : %ld cycles/KB\n", > + (tcompleted[iters - 1] - tposted[0])/(tsize * iters)); > } > > > @@ -480,14 +473,13 @@ int main(int argc, char *argv[]) > char *servername = NULL; > int port = 18515; > int ib_port = 1; > - int size = 1; > + int size = 4 * 1024; > int tx_depth = 100; > int iters = 1000; > int scnt, ccnt; > int sockfd; > int duplex = 0; > struct ibv_qp *qp; > - struct report_options report = {}; > > cycles_t *tposted; > cycles_t *tcompleted; > @@ -504,11 +496,10 @@ int main(int argc, char *argv[]) > { .name = "iters", .has_arg = 1, .val = 'n' }, > { .name = "tx-depth", .has_arg = 1, .val = 't' }, > { .name = "bidirectional", .has_arg = 0, .val = 'b' }, > - { .name = "report-cycles", .has_arg = 0, .val = 'C' }, > { 0 } > }; > > - c = getopt_long(argc, argv, "p:d:i:s:n:t:bC", long_options, NULL); > + c = getopt_long(argc, argv, "p:d:i:s:n:t:b", long_options, NULL); > if (c == -1) > break; > > @@ -565,10 +556,6 @@ int main(int argc, char *argv[]) > duplex = 1; > break; > > - case 'C': > - report.cycles = 1; > - break; > - > default: > usage(argv[0]); > return 1; > @@ -751,7 +738,7 @@ int main(int argc, char *argv[]) > write(sockfd, "done", sizeof "done"); > close(sockfd); > > - print_report(&report, iters, size, duplex, tposted, tcompleted); > + print_report(iters, size, duplex, tposted, tcompleted); > > free(tposted); > free(tcompleted); > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From iod00d at hp.com Wed Jun 8 17:07:31 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 8 Jun 2005 17:07:31 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query() In-Reply-To: <1118271274.9977.41.camel@duffman> References: <1118271274.9977.41.camel@duffman> Message-ID: <20050609000731.GF20219@esmail.cup.hp.com> On Wed, Jun 08, 2005 at 03:54:34PM -0700, Tom Duffy wrote: > I am putting this out there to see what people think about this sort of > change. This only does it for one of the functions in dat provider, but > I will change it for all if this is accepted. I agree it's the right direction. I would not have guessed dat_ia_query() is really a macro that does pointer chasing. > - ret = dat_ia_query (test_ptr->ia, > - NULL, > - &test_ptr->ia_attr, > - NULL); > + ret = test_ptr->ia->common.provider->dat_ia_query(test_ptr->ia, NULL, > + &test_ptr->ia_attr, NULL); Maybe not reference test_ptr in this line so often? It could read: ret = ia->common.provider->dat_ia_query(ia, NULL, &test_ptr->ia_attr, NULL); > - .ia_query_func = &dapl_ia_query, > + .dat_ia_query = &dapl_ia_query, Yes. I like this alot better too. thanks, grant From caitlin.bestler at gmail.com Wed Jun 8 17:08:05 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 8 Jun 2005 17:08:05 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query() In-Reply-To: <1118271274.9977.41.camel@duffman> References: <1118271274.9977.41.camel@duffman> Message-ID: <469958e0050608170848810ea@mail.gmail.com> On 6/8/05, Tom Duffy wrote: > I am putting this out there to see what people think about this sort of > change. This only does it for one of the functions in dat provider, but > I will change it for all if this is accepted. > > Basically, instead of using a redirected function wrapper, it calls the > function directly from the struct (how the rest of the kernel does it). > > And it gets rid of the typedef DAT_IA_QUERY_FUNC. > Hiding the fact that there was a method table was definitely a cosmetic decision to appeal to the biases of user mode C programmers. I agree that anyone working in the kernel is already quite familiar with calling a function from a struct. A strong argument can be made that a kernel programmer should know that they are making a call through a method table. What I am not following is why you think DAT_IA_QUERY_FUNC is somehow different than all of the other functions in a DAT method table. Is this just a test run? If not, what is the criteria for deciding which methods are exposed and which have their method table underpinnings kept hidden? From tduffy at sun.com Wed Jun 8 18:40:19 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 18:40:19 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query() In-Reply-To: <20050609000731.GF20219@esmail.cup.hp.com> References: <1118271274.9977.41.camel@duffman> <20050609000731.GF20219@esmail.cup.hp.com> Message-ID: <1118281219.30062.2.camel@duffman> On Wed, 2005-06-08 at 17:07 -0700, Grant Grundler wrote: > > - ret = dat_ia_query (test_ptr->ia, > > - NULL, > > - &test_ptr->ia_attr, > > - NULL); > > + ret = test_ptr->ia->common.provider->dat_ia_query(test_ptr->ia, NULL, > > + &test_ptr->ia_attr, NULL); > > Maybe not reference test_ptr in this line so often? > It could read: > ret = ia->common.provider->dat_ia_query(ia, NULL, &test_ptr->ia_attr, > NULL); Yup, cleaner. New patch attached. > > - .ia_query_func = &dapl_ia_query, > > + .dat_ia_query = &dapl_ia_query, > > Yes. I like this alot better too. Or should the name in dat_provider just be ia_query? Any need for the dat_? Signed-off-by: Tom Duffy Index: linux-kernel-return/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_performance_util.c (revision 2573) +++ linux-kernel-return/test/dapltest/test/dapl_performance_util.c (working copy) @@ -64,10 +64,7 @@ DT_Performance_Test_Create ( test_ptr->ia = ia; test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; - ret = dat_ia_query (test_ptr->ia, - NULL, - &test_ptr->ia_attr, - NULL); + ret = ia->common.provider->dat_ia_query(ia, NULL, &test_ptr->ia_attr, NULL); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ia_query error: %s\n", Index: linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (revision 2573) +++ linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -165,10 +165,8 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_ia_query (ia, - &evd, - &ia_attributes, - &provider_attributes); + rc = ia->common.provider->dat_ia_query(ia, &evd, &ia_attributes, + &provider_attributes); } else if (result_wanted == DAT_INVALID_PARAMETER) { @@ -177,17 +175,13 @@ int DT_queryinfo_basic (Params_t *params * NULL out ia_attr and for the DAT_IA_ATTR_MASK to * have values */ - rc = dat_ia_query (ia, - &evd, - NULL, - &provider_attributes); + rc = ia->common.provider->dat_ia_query(ia, &evd, NULL, + &provider_attributes); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_ia_query (ia, - &evd, - &ia_attributes, - &provider_attributes); + rc = ia->common.provider->dat_ia_query(ia, &evd, &ia_attributes, + &provider_attributes); } } Index: linux-kernel-return/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_test_util.c (revision 2573) +++ linux-kernel-return/test/dapltest/test/dapl_test_util.c (working copy) @@ -44,10 +44,9 @@ DT_query ( Per_Test_Data_t *pt_ptr, phead = pt_ptr->Params.phead; /* Query the IA */ - ret = dat_ia_query (ia, - &async_evd_hdl, - &pt_ptr->ia_attr, - &pt_ptr->provider_attr); + ret = ia->common.provider->dat_ia_query(ia, &async_evd_hdl, + &pt_ptr->ia_attr, + &pt_ptr->provider_attr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_query error: %s\n", Index: linux-kernel-return/dat-provider/dapl_provider.c =================================================================== --- linux-kernel-return/dat-provider/dapl_provider.c (revision 2573) +++ linux-kernel-return/dat-provider/dapl_provider.c (working copy) @@ -67,7 +67,7 @@ static struct dat_provider g_dapl_provid .extension = NULL, .ia_open_func = &dapl_ia_open, - .ia_query_func = &dapl_ia_query, + .dat_ia_query = &dapl_ia_query, .ia_close_func = &dapl_ia_close, .ia_memtype_hint_func = &dapl_ia_memtype_hint, Index: linux-kernel-return/dat/api.c =================================================================== --- linux-kernel-return/dat/api.c (revision 2573) +++ linux-kernel-return/dat/api.c (working copy) @@ -405,8 +405,8 @@ u32 dat_ia_close(struct dat_ia *ia, enum provider = ia->common.provider; ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, - &provider_attr))) + if (DAT_SUCCESS != (status = ia->common.provider->dat_ia_query(ia, NULL, NULL, + &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) Index: linux-kernel-return/dat/dat.h =================================================================== --- linux-kernel-return/dat/dat.h (revision 2573) +++ linux-kernel-return/dat/dat.h (working copy) @@ -1057,10 +1057,6 @@ typedef u32 (*DAT_IA_OPENV_FUNC)(const c typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum dat_close_flags); -typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, - struct dat_ia_attr *, - struct dat_provider_attr *); - typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct dat_cr_param *); typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct dat_ep *, int, @@ -1223,7 +1219,8 @@ struct dat_provider { void *extension; DAT_IA_OPEN_FUNC ia_open_func; - DAT_IA_QUERY_FUNC ia_query_func; + u32 (*dat_ia_query)(struct dat_ia *, struct dat_evd **, + struct dat_ia_attr *, struct dat_provider_attr *); DAT_IA_CLOSE_FUNC ia_close_func; DAT_IA_MEMTYPE_HINT_FUNC ia_memtype_hint_func; @@ -1396,14 +1393,6 @@ static inline u32 dat_ia_memtype_hint(st preferred_alignment); } -static inline u32 dat_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, - struct dat_ia_attr *ia_attr, - struct dat_provider_attr *provider_attr) -{ - return DAT_CALL_PROVIDER_FUNC( - ia_query_func, ia, async_evd, ia_attr, provider_attr); -} - static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data) { From tduffy at sun.com Wed Jun 8 18:43:37 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 08 Jun 2005 18:43:37 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query() In-Reply-To: <469958e0050608170848810ea@mail.gmail.com> References: <1118271274.9977.41.camel@duffman> <469958e0050608170848810ea@mail.gmail.com> Message-ID: <1118281417.30062.5.camel@duffman> I said: > > I am putting this out there to see what people think about this sort of > > change. This only does it for one of the functions in dat provider, but > > I will change it for all if this is accepted. Then Caitlin Bestler said: > What I am not following is why you think DAT_IA_QUERY_FUNC > is somehow different than all of the other functions in a DAT method > table. Is this just a test run? If not, what is the criteria for deciding > which methods are exposed and which have their method table > underpinnings kept hidden? Right, dat_ia_query() is not special. Just somewhere to start. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From caitlin.bestler at gmail.com Wed Jun 8 18:58:45 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 8 Jun 2005 18:58:45 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query() In-Reply-To: <1118281417.30062.5.camel@duffman> References: <1118271274.9977.41.camel@duffman> <469958e0050608170848810ea@mail.gmail.com> <1118281417.30062.5.camel@duffman> Message-ID: <469958e0050608185822bd67d9@mail.gmail.com> On 6/8/05, Tom Duffy wrote: > I said: > > > I am putting this out there to see what people think about this sort of > > > change. This only does it for one of the functions in dat provider, but > > > I will change it for all if this is accepted. > Oops, I parsed the header and "sort of change" but missed "in dat provider", which did make the set clear. > Then Caitlin Bestler said: > > What I am not following is why you think DAT_IA_QUERY_FUNC > > is somehow different than all of the other functions in a DAT method > > table. Is this just a test run? If not, what is the criteria for deciding > > which methods are exposed and which have their method table > > underpinnings kept hidden? > > > Right, dat_ia_query() is not special. Just somewhere to start. > Anyway, one extra benefit of this change is that eliminates any risk of breaking kernel code using other existing DAT Providers. And it is easy for someone who has a large body of code using the function calls to use their own macro or inline function if they don't want to change their code. So consistency with other kernel code probably is the best guideline here. From roland at topspin.com Wed Jun 8 20:49:55 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 08 Jun 2005 20:49:55 -0700 Subject: [openib-general] 2.6.11.11 NFS over IPoIB crash In-Reply-To: <42A75BAD.9020205@scl.ameslab.gov> (Troy Benjegerdes's message of "Wed, 08 Jun 2005 15:57:17 -0500") References: <03c22e4542ae1b6cdd72565432143ebe@scl.ameslab.gov> <42A75BAD.9020205@scl.ameslab.gov> Message-ID: <52mzq0chp8.fsf@topspin.com> Troy> We are running NFS over IPOIB, and are getting kernel panics Troy> under heavy NFS I/O. This is on a PowerMac G5, and the Troy> server is a dual opteron running 2.6.11 with the OpenIB code Troy> from subversion. It looks like a bug in nfs.. but we've only Troy> seen it using IPoIB... Is it worth trying to reproduce this Troy> over gigabit ethernet? I don't see anything obviously IPoIB related in the traceback, so it might be worth trying to get it to happen over ethernet. It's probably also worth running it past the Linux NFS mailing list. - R. From kashyapv at us.ibm.com Wed Jun 8 23:51:43 2005 From: kashyapv at us.ibm.com (Vivek Kashyap) Date: Wed, 8 Jun 2005 23:51:43 -0700 (Pacific Daylight Time) Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <6.2.0.14.2.20050607112638.0283e2e0@esmail.cup.hp.com> Message-ID: On Tue, 7 Jun 2005, Michael Krause wrote: > At 09:28 AM 6/7/2005, Fab Tillier wrote: > > > From: Roland Dreier [mailto:roland at topspin.com] > > > Sent: Tuesday, June 07, 2005 8:38 AM > > > > > > Michael> Why not just use the IETF draft for RC / UC based IP over > > > Michael> IB and not worry about creating something new? > > > > > > I think we've come full circle. The original post was a suggestion on > > > how to handle the fact the the connected-mode IPoIB draft requires a > > > network stack to deal with different MTUs for different destinations > > > on the same logical link. Yes, the ipoib wg opinion is that a node should be able to communicate in any IB mode. (I've had to implement such a setup for tokenring in the past - e.g. if the peer was on a ring with smaller MTU ). I beleive this is an issue that needs furhter discussion on the ipoverib discussion list in ietf. Would you restart the discussion there? We can update the draft as needed after a consensus is reached in the wg. > > > >That's right - by implementing IP segmentation in the IPoIB driver when going > >over UD, the driver could expose a single MTU to the network stack, thereby > >removing all the issues related to having per-endpoint MTUs. > > > >Keeping a 2K MTU for RC mode doesn't really take advantage of IB's RC > >capabilities. I'd probably target 64K as the MTU. > > The draft should state a minimum for all RC / UC which should be the TCP > MSS. Whether one does SAR over a UD endpoint independent of the underlying > physical MTU can be done but it should not require end-to-end understanding > of the operation, i.e. the send side tells its local that the TCP MSS is X > while the receive side only posts 2-4 KB buffers. This has been done over > Ethernet for years. > > Mike I suggest we move this discussion to ipoverib at ietf.org (can continue to cc this list) to involve all since there have been suggestions for draft update. Vivek __ Vivek Kashyap Linux Technology Center, IBM From rep.nop at aon.at Thu Jun 9 00:23:49 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Thu, 9 Jun 2005 09:23:49 +0200 Subject: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050608234637.GE20219@esmail.cup.hp.com> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> <20050601232123.GN25321@esmail.cup.hp.com> <20050604213422.GA23153@mellanox.co.il> <20050608002426.GI15489@esmail.cup.hp.com> <20050608055030.GA7809@mellanox.co.il> <20050608234057.GC20219@esmail.cup.hp.com> <20050608234637.GE20219@esmail.cup.hp.com> Message-ID: <20050609072349.GA30843@aon.at> On Wed, Jun 08, 2005 at 04:46:37PM -0700, Grant Grundler wrote: >On Wed, Jun 08, 2005 at 04:40:57PM -0700, Grant Grundler wrote: >> Index: rdma_bw.c >> =================================================================== >> --- rdma_bw.c (revision 2570) >> +++ rdma_bw.c (working copy) >> @@ -61,10 +61,6 @@ >... > >Sorry - just realized I forgot the: >Signed-off-by: Grant Grundler >> - int size = 1; >> + int size = 4 * 1024; Please reflect this change in the help text too. I'd make that at least 4*page_size, but 4k is certainly better than 1 :) From itamar at mellanox.co.il Thu Jun 9 01:37:25 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 9 Jun 2005 11:37:25 +0300 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() Message-ID: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> Hi Tom, It looks very good to me but i want to ask why do we need the dat_common struct ? as we can see in dat.h dat_common is defined as : struct dat_common { struct dat_provider *provider; union dat_context context; }; and is being use as data member of any dat_* struct There are 9 places were we instance dat_common (dat_ia, dat_ep, dat_evd, ...) i think that that dat_* struct should be like this struct dat_ia { struct dat_provider *provider; union dat_context context; }; instaed of struct dat_ia { struct dat_common common; }; if we do it (only 5 lines more in code ) every time we want to call a function it will be ret = ia->provider->dat_ia_query(ia, NULL, instaed of ret = ia->common.provider->dat_ia_query(ia, NULL, i think that this is more readable. and i agree that we dont need the dat_ before the functions name Itamar > -----Original Message----- > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Thursday, June 09, 2005 4:40 AM > To: Grant Grundler > Cc: openib-general at openib.org > Subject: Re: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper > function dat_ia_query() > > > On Wed, 2005-06-08 at 17:07 -0700, Grant Grundler wrote: > > > - ret = dat_ia_query (test_ptr->ia, > > > - NULL, > > > - &test_ptr->ia_attr, > > > - NULL); > > > + ret = > test_ptr->ia->common.provider->dat_ia_query(test_ptr->ia, NULL, > > > + > &test_ptr->ia_attr, NULL); > > > > Maybe not reference test_ptr in this line so often? > > It could read: > > ret = ia->common.provider->dat_ia_query(ia, NULL, > &test_ptr->ia_attr, > > NULL); > > Yup, cleaner. New patch attached. > > > > - .ia_query_func = &dapl_ia_query, > > > + .dat_ia_query = &dapl_ia_query, > > > > Yes. I like this alot better too. > > Or should the name in dat_provider just be ia_query? Any need for the > dat_? > > Signed-off-by: Tom Duffy > > Index: linux-kernel-return/test/dapltest/test/dapl_performance_util.c > =================================================================== > --- > linux-kernel-return/test/dapltest/test/dapl_performance > _util.c (revision 2573) > +++ > linux-kernel-return/test/dapltest/test/dapl_performance > _util.c (working copy) > @@ -64,10 +64,7 @@ DT_Performance_Test_Create ( > test_ptr->ia = ia; > test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; > > - ret = dat_ia_query (test_ptr->ia, > - NULL, > - &test_ptr->ia_attr, > - NULL); > + ret = ia->common.provider->dat_ia_query(ia, NULL, > &test_ptr->ia_attr, NULL); > if ( DAT_SUCCESS != ret) > { > DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ia_query > error: %s\n", > Index: linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c > =================================================================== > --- > linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c > (revision 2573) > +++ > linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c > (working copy) > @@ -165,10 +165,8 @@ int DT_queryinfo_basic (Params_t *params > { > if (result_wanted == DAT_SUCCESS) > { > - rc = dat_ia_query (ia, > - &evd, > - &ia_attributes, > - &provider_attributes); > + rc = ia->common.provider->dat_ia_query(ia, &evd, > &ia_attributes, > + > &provider_attributes); > } > else if (result_wanted == DAT_INVALID_PARAMETER) > { > @@ -177,17 +175,13 @@ int DT_queryinfo_basic (Params_t *params > * NULL out ia_attr and for the DAT_IA_ATTR_MASK to > * have values > */ > - rc = dat_ia_query (ia, > - &evd, > - NULL, > - &provider_attributes); > + rc = ia->common.provider->dat_ia_query(ia, &evd, NULL, > + > &provider_attributes); > } > else if (result_wanted == DAT_INVALID_HANDLE) > { > - rc = dat_ia_query (ia, > - &evd, > - &ia_attributes, > - &provider_attributes); > + rc = ia->common.provider->dat_ia_query(ia, &evd, > &ia_attributes, > + > &provider_attributes); > } > } > > Index: linux-kernel-return/test/dapltest/test/dapl_test_util.c > =================================================================== > --- linux-kernel-return/test/dapltest/test/dapl_test_util.c > (revision 2573) > +++ linux-kernel-return/test/dapltest/test/dapl_test_util.c > (working copy) > @@ -44,10 +44,9 @@ DT_query ( Per_Test_Data_t *pt_ptr, > phead = pt_ptr->Params.phead; > > /* Query the IA */ > - ret = dat_ia_query (ia, > - &async_evd_hdl, > - &pt_ptr->ia_attr, > - &pt_ptr->provider_attr); > + ret = ia->common.provider->dat_ia_query(ia, &async_evd_hdl, > + &pt_ptr->ia_attr, > + &pt_ptr->provider_attr); > if (ret != DAT_SUCCESS) > { > DT_Tdep_PT_Printf (phead, "%s: dat_ia_query error: %s\n", > Index: linux-kernel-return/dat-provider/dapl_provider.c > =================================================================== > --- linux-kernel-return/dat-provider/dapl_provider.c (revision 2573) > +++ linux-kernel-return/dat-provider/dapl_provider.c (working copy) > @@ -67,7 +67,7 @@ static struct dat_provider g_dapl_provid > .extension = NULL, > > .ia_open_func = &dapl_ia_open, > - .ia_query_func = &dapl_ia_query, > + .dat_ia_query = &dapl_ia_query, > .ia_close_func = &dapl_ia_close, > .ia_memtype_hint_func = &dapl_ia_memtype_hint, > > Index: linux-kernel-return/dat/api.c > =================================================================== > --- linux-kernel-return/dat/api.c (revision 2573) > +++ linux-kernel-return/dat/api.c (working copy) > @@ -405,8 +405,8 @@ u32 dat_ia_close(struct dat_ia *ia, enum > provider = ia->common.provider; > ia_name = provider->device_name; > > - if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, > - &provider_attr))) > + if (DAT_SUCCESS != (status = > ia->common.provider->dat_ia_query(ia, NULL, NULL, > + > &provider_attr))) > dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > "dat_ia_query for IA %s > failed\n", ia_name); > else if (DAT_SUCCESS != (status = > provider->ia_close_func(ia, flags))) > Index: linux-kernel-return/dat/dat.h > =================================================================== > --- linux-kernel-return/dat/dat.h (revision 2573) > +++ linux-kernel-return/dat/dat.h (working copy) > @@ -1057,10 +1057,6 @@ typedef u32 (*DAT_IA_OPENV_FUNC)(const c > > typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum > dat_close_flags); > > -typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, > - struct dat_ia_attr *, > - struct dat_provider_attr *); > - > typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct > dat_cr_param *); > > typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct > dat_ep *, int, > @@ -1223,7 +1219,8 @@ struct dat_provider { > void *extension; > > DAT_IA_OPEN_FUNC ia_open_func; > - DAT_IA_QUERY_FUNC ia_query_func; > + u32 (*dat_ia_query)(struct dat_ia *, struct dat_evd **, > + struct dat_ia_attr *, struct > dat_provider_attr *); > DAT_IA_CLOSE_FUNC ia_close_func; > DAT_IA_MEMTYPE_HINT_FUNC ia_memtype_hint_func; > > @@ -1396,14 +1393,6 @@ static inline u32 dat_ia_memtype_hint(st > preferred_alignment); > } > > -static inline u32 dat_ia_query(struct dat_ia *ia, struct > dat_evd **async_evd, > - struct dat_ia_attr *ia_attr, > - struct dat_provider_attr *provider_attr) > -{ > - return DAT_CALL_PROVIDER_FUNC( > - ia_query_func, ia, async_evd, ia_attr, > provider_attr); > -} > - > static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, > int private_data_size, const > void *private_data) > { > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hch at lst.de Thu Jun 9 02:47:59 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 9 Jun 2005 11:47:59 +0200 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> Message-ID: <20050609094759.GA5947@lst.de> On Thu, Jun 09, 2005 at 11:37:25AM +0300, Itamar Rabenstein wrote: > Hi Tom, > > It looks very good to me but i want to ask why do we need the dat_common > struct ? > > as we can see in dat.h dat_common is defined as : > > struct dat_common { > struct dat_provider *provider; > union dat_context context; > }; > and is being use as data member of any dat_* struct > There are 9 places were we instance dat_common > (dat_ia, dat_ep, dat_evd, ...) > > i think that that dat_* struct should be like this > struct dat_ia { > struct dat_provider *provider; > union dat_context context; > }; > instaed of > struct dat_ia { > struct dat_common common; > }; > > if we do it (only 5 lines more in code ) > every time we want to call a function it will be > ret = ia->provider->dat_ia_query(ia, NULL, > instaed of > ret = ia->common.provider->dat_ia_query(ia, NULL, > > i think that this is more readable. It is. B?ut I think it'll change even more later on - having a pointer to the method table in every object is an enormous waste of space, in the end the method should probably be only in a hca-level object, and all other objects should have public pointers to it. From Postmaster at mx.bigpond.com Thu Jun 9 05:27:27 2005 From: Postmaster at mx.bigpond.com (Mail Administrator) Date: Thu, 9 Jun 2005 12:27:27 +0000 Subject: [openib-general] Mail System Error - Returned Mail Message-ID: <20050609122727.ZHUW27377.omta04ps.mx.bigpond.com@omta04ps> This Message was undeliverable due to the following reason: Each of the following recipients was rejected by a remote mail server. The reasons given by the server are included to help you determine why each recipient was rejected. Recipient: Reason: No such user - psmtp Please reply to if you feel this message to be in error. -------------- next part -------------- An embedded message was scrubbed... From: openib-general at openib.org Subject: Here is your documents. Date: Thu, 9 Jun 2005 20:23:24 +0800 Size: 49592 URL: From halr at voltaire.com Thu Jun 9 05:48:10 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 Jun 2005 08:48:10 -0400 Subject: [openib-general] IPoIB MTU Changing Message-ID: <1118321289.4493.26.camel@localhost.localdomain> Hi Roland, I have a question about ipoib_main.c::ipoib_change_mtu: static int ipoib_change_mtu(struct net_device *dev, int new_mtu) { struct ipoib_dev_priv *priv = netdev_priv(dev); if (new_mtu > IPOIB_PACKET_SIZE - IPOIB_ENCAP_LEN) return -EINVAL; Shouldn't the new MTU be compared to the link MTU (minus the encapsulation header length) rather than a hardcoded 2K ? -- Hal From jlentini at netapp.com Thu Jun 9 07:26:46 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 10:26:46 -0400 (EDT) Subject: [openib-general] RE: [patch][kdapl] enable kdapltest -T P In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C4D@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C4D@mtvex01.mtv.mtl.com> Message-ID: On Wed, 8 Jun 2005, Itamar Rabenstein wrote: >>> Current openib gen2 code is not reporting the max cq size >> and i dont think >>> that we should put >>> a fix number . >>> if we want to get the number we need Roland to fill this >> number in mthca but >>> as Roland said before >>> "what real App will meed this number?" >> >> I see the initialization code you are refering to in >> mthca_query_device of mthca_provider.c. >> >> I think a "real app" would use this number in exactly the same way >> that dapltest's performance subtest uses it: >> >> pipeline_length = min(max_cqe, max_qp_wr) >> >> Why would a limit like the one above be unnecessary? > > you can try to printk the value and you will see it is not initialized. > it is not enough to assign value to the output attr struct > if this value is not initialized (;-) Understood. > i dont mind if Roland will implement it but for now it is not implemented. I will ping Roland on this subject. > Please ci the patch for now in order to enable kdapltest -T P > and if Roland will implement it then unremark the lines. > > >>>> >>>> Was this also a problem in the transaction test? >>>> >>> >>> Yes but in order to fall on the bug you need to alloc a >> very small buffer >>> (like 12 byte) >>> and this is only in -T P . >> >> So is it only in the performance test or would it occur in the >> transaction test if I specified a small buffer? > > It will fail for every small buffer (include in -T T ) > >> >> Why do we need to translate the virtual address to a physical address >> in all cases? Will this interact properly with the transaction test's >> -M (memory type) option? > > Current openib gen2 register only physical memory. > the code will work also for any -M option . So I don't make this mistake in the future, can you explain what the underlying problem was? From iod00d at hp.com Thu Jun 9 07:44:32 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 9 Jun 2005 07:44:32 -0700 Subject: [openib-general] Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050609072349.GA30843@aon.at> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> <20050601232123.GN25321@esmail.cup.hp.com> <20050604213422.GA23153@mellanox.co.il> <20050608002426.GI15489@esmail.cup.hp.com> <20050608055030.GA7809@mellanox.co.il> <20050608234057.GC20219@esmail.cup.hp.com> <20050608234637.GE20219@esmail.cup.hp.com> <20050609072349.GA30843@aon.at> Message-ID: <20050609144432.GA22595@esmail.cup.hp.com> On Thu, Jun 09, 2005 at 09:23:49AM +0200, Bernhard Fischer wrote: > > - int size = 1; > > + int size = 4 * 1024; > > Please reflect this change in the help text too. Sorry - my bad. I was testing with that and didn't intend to submit. You've already got a patch outstanding for 4MB. > I'd make that at least 4*page_size, but 4k is certainly better than 1 :) BTW, I'm thinking 4MB is too big for a default. One page seems like a reasonable default. thanks, grant From jlentini at netapp.com Thu Jun 9 08:00:48 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 11:00:48 -0400 (EDT) Subject: [openib-general] Re: [patch][kdapl] enable kdapltest -T P In-Reply-To: <20050602132843.GA3760@mellanox.co.il> References: <20050602132843.GA3760@mellanox.co.il> Message-ID: Committed in revision 2574. On Thu, 2 Jun 2005, Itamar wrote: itamar> in order to enable kdapltest -T P i needed to remark attr that are not set by openib gen2 itamar> 1) ia_attr.max_evd_qlen itamar> 2) ia_attr.max_rdma_read_per_ep_in itamar> 3) ia_attr.max_rdma_read_per_ep_out itamar> itamar> also there was bug in kdapltest memory registration (file dapl_bpool.c) itamar> and there was bug where we free memory and then we try to print part of the memory itamar> itamar> i can run now kdapltest -T P for both RR and RW itamar> itamar> itamar> enable kdapltest -T P itamar> fix bug in memory registration itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: test/dapl_performance_util.c itamar> =================================================================== itamar> --- test/dapl_performance_util.c (revision 2522) itamar> +++ test/dapl_performance_util.c (working copy) itamar> @@ -76,17 +76,20 @@ itamar> } itamar> itamar> pipeline_len = DT_min ( itamar> - DT_min (test_ptr->cmd->num_iterations, itamar> - test_ptr->cmd->pipeline_len), itamar> - DT_min (test_ptr->ia_attr.max_dto_per_ep, itamar> - test_ptr->ia_attr.max_evd_qlen)); itamar> + DT_min (test_ptr->cmd->num_iterations, itamar> + test_ptr->cmd->pipeline_len), itamar> + test_ptr->ia_attr.max_dto_per_ep); itamar> +/* DT_min (test_ptr->ia_attr.max_dto_per_ep, */ itamar> +/* test_ptr->ia_attr.max_evd_qlen) */ itamar> +/* ); */ itamar> itamar> if ( RDMA_READ == test_ptr->cmd->op.transfer_type ) itamar> - { itamar> - pipeline_len = DT_min (pipeline_len, itamar> - test_ptr->ia_attr.max_rdma_read_per_ep_in); itamar> - pipeline_len = DT_min (pipeline_len, itamar> - test_ptr->ia_attr.max_rdma_read_per_ep_out); itamar> + { itamar> +/* max_rdma_read_per_ep_in max_rdma_read_per_ep_out are not set by gen2 */ itamar> +/* pipeline_len = DT_min (pipeline_len, */ itamar> +/* test_ptr->ia_attr.max_rdma_read_per_ep_in); */ itamar> +/* pipeline_len = DT_min (pipeline_len, */ itamar> +/* test_ptr->ia_attr.max_rdma_read_per_ep_out); */ itamar> } itamar> itamar> test_ptr->reqt_evd_length = pipeline_len; itamar> @@ -203,7 +206,7 @@ itamar> DAT_HANDLE_NULL, /* rmr */ itamar> DT_PERF_SYNC_BUFF_SIZE, itamar> 2, /* 2 RMIs */ itamar> - 255, /* FIXME should query for this */ itamar> + 256, /* FIXME should query for this */ itamar> FALSE, itamar> FALSE); itamar> if ( !test_ptr->ep_context.bp ) itamar> @@ -406,8 +409,8 @@ itamar> } itamar> } itamar> itamar> + DT_Tdep_PT_Debug (1, (phead,"Test[" F64x "]: cleanup is done\n", test_ptr->base_port)); itamar> DT_MemListFree (test_ptr->pt_ptr, test_ptr); itamar> - DT_Tdep_PT_Debug (1, (phead,"Test[" F64x "]: cleanup is done\n", test_ptr->base_port)); itamar> itamar> return status; itamar> } itamar> Index: test/dapl_bpool.c itamar> =================================================================== itamar> --- test/dapl_bpool.c (revision 2522) itamar> +++ test/dapl_bpool.c (working copy) itamar> @@ -388,7 +388,7 @@ itamar> u64 itamar> DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index) itamar> { itamar> - return ( bpool_ptr->reg_addr + index * bpool_ptr->seg_size ); itamar> + return (virt_to_phys(DT_Bpool_GetBuffer(bpool_ptr,index))); itamar> } itamar> itamar> /*****************************************************************************/ itamar> -- itamar> Itamar itamar> From jlentini at netapp.com Thu Jun 9 08:03:05 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 11:03:05 -0400 (EDT) Subject: [openib-general] kdaptest wedges server In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C23@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C23@mtvex01.mtv.mtl.com> Message-ID: Itamar, Were you able to get to the bottom of this? james On Thu, 2 Jun 2005, Itamar Rabenstein wrote: itamar>Hi Josh, itamar> itamar>Can you run it with debug option ( -d ) itamar>and tell me what is the last debug print on the server side ? itamar>when you say "kill" can you hit ^c or is it hanging the station ? itamar> itamar>what is system configuration ? (arch , Os ,kernel) itamar> itamar> Itamar itamar> itamar> itamar>> -----Original Message----- itamar>> From: Josh England [mailto:jjengla at sandia.gov] itamar>> Sent: Thursday, June 02, 2005 9:00 PM itamar>> To: openib-general itamar>> Subject: [openib-general] kdaptest wedges server itamar>> itamar>> itamar>> Hi, itamar>> itamar>> The kdapltest program wedges consistently wedges a node. I just tried itamar>> it with Revision 2523, and it runs fine the first couple of itamar>> times. The itamar>> third time took a while, and the fourth time killed the server. itamar>> itamar>> The server is running: itamar>> kdapltest -T S -D mthca0a itamar>> itamar>> The client runs this several times in succession: itamar>> kdapltest -T T -D mthca0a -s client SR 256 server SR 256 itamar>> itamar>> -JE itamar>> itamar>> itamar>> _______________________________________________ itamar>> openib-general mailing list itamar>> openib-general at openib.org itamar>> http://openib.org/mailman/listinfo/openib-general itamar>> itamar>> To unsubscribe, please visit itamar>> http://openib.org/mailman/listinfo/openib-general itamar>> itamar>_______________________________________________ itamar>openib-general mailing list itamar>openib-general at openib.org itamar>http://openib.org/mailman/listinfo/openib-general itamar> itamar>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general itamar> From tduffy at sun.com Thu Jun 9 08:22:11 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 09 Jun 2005 08:22:11 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <20050609094759.GA5947@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> Message-ID: <1118330531.21255.6.camel@duffman> On Thu, 2005-06-09 at 11:47 +0200, Christoph Hellwig wrote: > It is. B?ut I think it'll change even more later on - having a pointer > to the method table in every object is an enormous waste of space, in > the end the method should probably be only in a hca-level object, and > all other objects should have public pointers to it. Can you please explain this a bit more? A 32 or 64 bit pointer is a waste of space? Or are you saying that each provider should just define a struct and have a global pointer to it? Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Thu Jun 9 09:05:55 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 09 Jun 2005 09:05:55 -0700 Subject: [openib-general] Re: IPoIB MTU Changing In-Reply-To: <1118321289.4493.26.camel@localhost.localdomain> (Hal Rosenstock's message of "09 Jun 2005 08:48:10 -0400") References: <1118321289.4493.26.camel@localhost.localdomain> Message-ID: <52ekbbcy70.fsf@topspin.com> Hal> Shouldn't the new MTU be compared to the link MTU (minus the Hal> encapsulation header length) rather than a hardcoded 2K ? Yes, that seems right. - R. From roland at topspin.com Thu Jun 9 09:13:27 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 09 Jun 2005 09:13:27 -0700 Subject: [openib-general] Re: IPoIB MTU Changing In-Reply-To: <52ekbbcy70.fsf@topspin.com> (Roland Dreier's message of "Thu, 09 Jun 2005 09:05:55 -0700") References: <1118321289.4493.26.camel@localhost.localdomain> <52ekbbcy70.fsf@topspin.com> Message-ID: <527jh3cxug.fsf@topspin.com> I take that back. I believe the code is correct as it stands. There are two MTU values we keep track of: admin_mtu, requested by the user, and mcast_mtu, calculated from the link MTU. We use the lower of the two values. This allows the user to set an MTU and not have the setting lost if the broadcast multicast join completes afterwards. - R. From hch at lst.de Thu Jun 9 09:17:22 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 9 Jun 2005 18:17:22 +0200 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <1118330531.21255.6.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> Message-ID: <20050609161722.GA13454@lst.de> On Thu, Jun 09, 2005 at 08:22:11AM -0700, Tom Duffy wrote: > On Thu, 2005-06-09 at 11:47 +0200, Christoph Hellwig wrote: > > It is. B?ut I think it'll change even more later on - having a pointer > > to the method table in every object is an enormous waste of space, in > > the end the method should probably be only in a hca-level object, and > > all other objects should have public pointers to it. > > Can you please explain this a bit more? A 32 or 64 bit pointer is a > waste of space? Or are you saying that each provider should just define > a struct and have a global pointer to it? No. What I mean is that we don't need a pointer to the method table in every single object. We don't have network driver methods in every sk_buff, struct sock and what other objects exist in the network stack. Instead the method table should be in some top-level per-HCA or per-port which most other object will have pointers to anyway (maybe through one or two indirections). This will need quite a bit of revamp of the dat codebase to move more members of the objects to the public part of it, but that should happen anyway. From uploads at doneasy.com Thu Jun 9 10:21:19 2005 From: uploads at doneasy.com (Andre Childress) Date: Thu, 09 Jun 2005 21:21:19 +0400 Subject: [openib-general] Notification: We offer low rates Message-ID: <20041dshlg3.ED8DA244AE@mailhost1l.lists.techtarget.com> Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $400,000 for as little as $400 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.pr0ved.com/signs.asp Best Regards, Salvatore Case to be remov(ed: http://www.pr0ved.com/deletion.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From hch at lst.de Thu Jun 9 09:30:11 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 9 Jun 2005 18:30:11 +0200 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <20050609161722.GA13454@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <20050609161722.GA13454@lst.de> Message-ID: <20050609163011.GA13748@lst.de> Btw, it seems the openib list manager software has a bug in that it doesn't send you a copy of a mail you received a personal copy of. Any chance the list admin could look into this? From halr at voltaire.com Thu Jun 9 09:34:48 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 Jun 2005 12:34:48 -0400 Subject: [openib-general] Re: IPoIB MTU Changing In-Reply-To: <527jh3cxug.fsf@topspin.com> References: <1118321289.4493.26.camel@localhost.localdomain> <52ekbbcy70.fsf@topspin.com> <527jh3cxug.fsf@topspin.com> Message-ID: <1118334888.4479.22.camel@localhost.localdomain> On Thu, 2005-06-09 at 12:13, Roland Dreier wrote: > I take that back. I believe the code is correct as it stands. There > are two MTU values we keep track of: admin_mtu, requested by the user, > and mcast_mtu, calculated from the link MTU. We use the lower of the > two values. This allows the user to set an MTU and not have the > setting lost if the broadcast multicast join completes afterwards. Yes, the multicast MTU could further trim this down but the admin MTU appears to be currently capped at IPOIB_PACKET_SIZE - IPOIB_ENCAP_LEN. That's OK for a 2K link MTU but if the link were negotiated at 4K MTU, shouldn't this be allowed to be larger if requested ? -- Hal From libor at topspin.com Thu Jun 9 09:48:19 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 9 Jun 2005 09:48:19 -0700 Subject: [openib-general] shutdown issue with SDP In-Reply-To: <78d18e2050608145322cdb574@mail.gmail.com>; from bjordan.ics@gmail.com on Wed, Jun 08, 2005 at 05:53:10PM -0400 References: <78d18e20506081029183adcfc@mail.gmail.com> <20050608124650.A28180@topspin.com> <78d18e2050608145322cdb574@mail.gmail.com> Message-ID: <20050609094819.A28821@topspin.com> On Wed, Jun 08, 2005 at 05:53:10PM -0400, William Jordan wrote: > On 6/8/05, Libor Michalek wrote: > > On Wed, Jun 08, 2005 at 01:29:42PM -0400, William Jordan wrote: > > > I'm trying to run netperf over SDP using libsdp. I'm having > > > intermittent success, but usually, I get a failure. The client does a > > > shutdown (SHUT_WR), but the server gets an ECONNRESET, "Connection > > > reset by peer", on the recv instead of a zero length recv. > > > > > > I'm running 2568. > > > > > > Anyone seen similar/different results? > > > > Can you try the patch I sent out yesterday? It significantly simplifies > > the connection state machine within SDP. I ran it successfully with a > > number of the netperf tests. (TCP_CC, TCP_CRR, TCP_RR, TCP_STREAM) > > That appears to have fixed it. Great, I'll commit the changes. -Libor From roland at topspin.com Thu Jun 9 09:48:27 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 09 Jun 2005 09:48:27 -0700 Subject: [openib-general] Re: IPoIB MTU Changing In-Reply-To: <1118334888.4479.22.camel@localhost.localdomain> (Hal Rosenstock's message of "09 Jun 2005 12:34:48 -0400") References: <1118321289.4493.26.camel@localhost.localdomain> <52ekbbcy70.fsf@topspin.com> <527jh3cxug.fsf@topspin.com> <1118334888.4479.22.camel@localhost.localdomain> Message-ID: <52vf4nbhno.fsf@topspin.com> Hal> Yes, the multicast MTU could further trim this down but the Hal> admin MTU appears to be currently capped at IPOIB_PACKET_SIZE Hal> - IPOIB_ENCAP_LEN. That's OK for a 2K link MTU but if the Hal> link were negotiated at 4K MTU, shouldn't this be allowed to Hal> be larger if requested ? No, you would have to change the PACKET_SIZE define to make 4K packets work anyway. - R. From tduffy at sun.com Thu Jun 9 09:51:20 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 09 Jun 2005 09:51:20 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <20050609163011.GA13748@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <20050609161722.GA13454@lst.de> <20050609163011.GA13748@lst.de> Message-ID: <1118335880.21255.11.camel@duffman> On Thu, 2005-06-09 at 18:30 +0200, Christoph Hellwig wrote: > Btw, it seems the openib list manager software has a bug in that it > doesn't send you a copy of a mail you received a personal copy of. Any > chance the list admin could look into this? This is a preference you can set in mailman: http://openib.org/mailman/options/openib-general > Avoid duplicate copies of messages? > > When you are listed explicitly in the To: or Cc: headers of a list > message, you can opt to not receive another copy from the mailing > list. Select Yes to avoid receiving copies from the mailing list; > select No to receive copies. > > If the list has member personalized messages enabled, and you elect to > receive copies, every copy will have a X-Mailman-Copy: yes header > added to it. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Thu Jun 9 09:52:50 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 12:52:50 -0400 (EDT) Subject: [openib-general] RE: [patch][kdapl] enable kdapltest -T P In-Reply-To: References: <91DB792C7985D411BEC300B40080D29CC35C4D@mtvex01.mtv.mtl.com> Message-ID: On Thu, 9 Jun 2005, James Lentini wrote: > On Wed, 8 Jun 2005, Itamar Rabenstein wrote: > >>>> Current openib gen2 code is not reporting the max cq size >>> and i dont think >>>> that we should put >>>> a fix number . >>>> if we want to get the number we need Roland to fill this >>> number in mthca but >>>> as Roland said before >>>> "what real App will meed this number?" >>> >>> I see the initialization code you are refering to in >>> mthca_query_device of mthca_provider.c. >>> >>> I think a "real app" would use this number in exactly the same way >>> that dapltest's performance subtest uses it: >>> >>> pipeline_length = min(max_cqe, max_qp_wr) >>> >>> Why would a limit like the one above be unnecessary? >> >> you can try to printk the value and you will see it is not initialized. >> it is not enough to assign value to the output attr struct >> if this value is not initialized (;-) > > Understood. In preparation for asking Roland about this, I added a print statement as you suggested. I actually see the ib_device_attr structure's max_qp_wr and max_cqe values as being initialized to 65535. After taking a second look at the initialization code in mthca_query_device(), I see that these are indeed the values being specified (and not all 1s as I had thought initially). Why did you think that these were not initialized? 65535 work requests and cqe's seem like perfectly valid values to me. From halr at voltaire.com Thu Jun 9 10:16:53 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 Jun 2005 13:16:53 -0400 Subject: [openib-general] Re: IPoIB MTU Changing In-Reply-To: <52vf4nbhno.fsf@topspin.com> References: <1118321289.4493.26.camel@localhost.localdomain> <52ekbbcy70.fsf@topspin.com> <527jh3cxug.fsf@topspin.com> <1118334888.4479.22.camel@localhost.localdomain> <52vf4nbhno.fsf@topspin.com> Message-ID: <1118337413.4479.42.camel@localhost.localdomain> On Thu, 2005-06-09 at 12:48, Roland Dreier wrote: > Hal> Yes, the multicast MTU could further trim this down but the > Hal> admin MTU appears to be currently capped at IPOIB_PACKET_SIZE > Hal> - IPOIB_ENCAP_LEN. That's OK for a 2K link MTU but if the > Hal> link were negotiated at 4K MTU, shouldn't this be allowed to > Hal> be larger if requested ? > > No, you would have to change the PACKET_SIZE define to make 4K packets > work anyway. OK. The changes go a little deeper than I thought. Would it be worthwhile to accomodate this for HCAs which support 4K MTU ? -- Hal From roland at topspin.com Thu Jun 9 10:26:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 09 Jun 2005 10:26:20 -0700 Subject: [openib-general] Re: IPoIB MTU Changing In-Reply-To: <1118337413.4479.42.camel@localhost.localdomain> (Hal Rosenstock's message of "09 Jun 2005 13:16:53 -0400") References: <1118321289.4493.26.camel@localhost.localdomain> <52ekbbcy70.fsf@topspin.com> <527jh3cxug.fsf@topspin.com> <1118334888.4479.22.camel@localhost.localdomain> <52vf4nbhno.fsf@topspin.com> <1118337413.4479.42.camel@localhost.localdomain> Message-ID: <52d5qvbfwj.fsf@topspin.com> Hal> OK. The changes go a little deeper than I thought. Would it Hal> be worthwhile to accomodate this for HCAs which support 4K Hal> MTU ? Sure. - R. From itamar at mellanox.co.il Thu Jun 9 10:58:34 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 9 Jun 2005 20:58:34 +0300 Subject: [openib-general] kdaptest wedges server Message-ID: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> No , I dont see the problem at my system . Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Thursday, June 09, 2005 6:03 PM > To: Itamar Rabenstein > Cc: Josh England; openib-general > Subject: RE: [openib-general] kdaptest wedges server > > > > Itamar, > > Were you able to get to the bottom of this? > > james > > On Thu, 2 Jun 2005, Itamar Rabenstein wrote: > > itamar>Hi Josh, > itamar> > itamar>Can you run it with debug option ( -d ) > itamar>and tell me what is the last debug print on the server side ? > itamar>when you say "kill" can you hit ^c or is it hanging > the station ? > itamar> > itamar>what is system configuration ? (arch , Os ,kernel) > itamar> > itamar> Itamar > itamar> > itamar> > itamar>> -----Original Message----- > itamar>> From: Josh England [mailto:jjengla at sandia.gov] > itamar>> Sent: Thursday, June 02, 2005 9:00 PM > itamar>> To: openib-general > itamar>> Subject: [openib-general] kdaptest wedges server > itamar>> > itamar>> > itamar>> Hi, > itamar>> > itamar>> The kdapltest program wedges consistently wedges a > node. I just tried > itamar>> it with Revision 2523, and it runs fine the first couple of > itamar>> times. The > itamar>> third time took a while, and the fourth time killed > the server. > itamar>> > itamar>> The server is running: > itamar>> kdapltest -T S -D mthca0a > itamar>> > itamar>> The client runs this several times in succession: > itamar>> kdapltest -T T -D mthca0a -s client SR 256 > server SR 256 > itamar>> > itamar>> -JE > itamar>> > itamar>> > itamar>> _______________________________________________ > itamar>> openib-general mailing list > itamar>> openib-general at openib.org > itamar>> http://openib.org/mailman/listinfo/openib-general > itamar>> > itamar>> To unsubscribe, please visit > itamar>> http://openib.org/mailman/listinfo/openib-general > itamar>> > itamar>_______________________________________________ > itamar>openib-general mailing list > itamar>openib-general at openib.org > itamar>http://openib.org/mailman/listinfo/openib-general > itamar> > itamar>To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > itamar> > From itamar at mellanox.co.il Thu Jun 9 11:01:20 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 9 Jun 2005 21:01:20 +0300 Subject: [openib-general] RE: [patch][kdapl] enable kdapltest -T P Message-ID: <91DB792C7985D411BEC300B40080D29CC35C73@mtvex01.mtv.mtl.com> > > In preparation for asking Roland about this, I added a print > statement > as you suggested. I actually see the ib_device_attr structure's > max_qp_wr and max_cqe values as being initialized to 65535. > After taking a second look at the initialization code in > mthca_query_device(), I see that these are indeed the values being > specified (and not all 1s as I had thought initially). > > Why did you think that these were not initialized? 65535 work > requests > and cqe's seem like perfectly valid values to me. > At the time i worked on opening the flow of kdapltst -T P all 3 attr were not init by mthca if they are now init it's greate ! Itamar From libor at topspin.com Thu Jun 9 10:50:17 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 9 Jun 2005 10:50:17 -0700 Subject: [openib-general] [CM] possible problem with crossing DREQs. Message-ID: <20050609105017.B28821@topspin.com> Sean, I'm seeing an unusual problem when both halves of a connection actively disconnect at the same time. Each connection peer issues a DREQ at the same time, next each receive the DREQ and responds with a DREP, and finally each connection gets a callback for the transition to the idle state. However, at this point it appears that each CM keeps retransmitting DREQ requests, which then seems to interfere with new connection establishment. For now I've ifdef'd out the code which causes the simultaneous disconnect, and all works as expected. However, I'd like to add it back in at some point. -Libor From jlentini at netapp.com Thu Jun 9 10:51:15 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 13:51:15 -0400 (EDT) Subject: [openib-general] RE: [patch][kdapl] enable kdapltest -T P In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C73@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C73@mtvex01.mtv.mtl.com> Message-ID: Can you confirm that removing the #if 0 on line 84 of dapl_performance_util.c works with the current DAT provider and IB stack? james On Thu, 9 Jun 2005, Itamar Rabenstein wrote: >> >> In preparation for asking Roland about this, I added a print >> statement >> as you suggested. I actually see the ib_device_attr structure's >> max_qp_wr and max_cqe values as being initialized to 65535. >> After taking a second look at the initialization code in >> mthca_query_device(), I see that these are indeed the values being >> specified (and not all 1s as I had thought initially). >> >> Why did you think that these were not initialized? 65535 work >> requests >> and cqe's seem like perfectly valid values to me. >> > > At the time i worked on opening the flow of kdapltst -T P all 3 attr were > not init by mthca > if they are now init it's greate ! > > Itamar > From tduffy at sun.com Thu Jun 9 11:17:12 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 09 Jun 2005 11:17:12 -0700 Subject: [openib-general] [PATCHv3] kDAPL: remove dat wrapper function dat_ia_query() In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> Message-ID: <1118341032.32250.6.camel@duffman> On Thu, 2005-06-09 at 11:37 +0300, Itamar Rabenstein wrote: > and i agree that we dont need the dat_ before the functions name Ok, this patch renames the method ia_query (instead of dat_ia_query). James, can you please apply? If you do, I will write a patch to do the rest. We can do the dat_common cleanup later, if desired. Signed-off-by: Tom Duffy Index: linux-kernel-return/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_performance_util.c (revision 2575) +++ linux-kernel-return/test/dapltest/test/dapl_performance_util.c (working copy) @@ -64,10 +64,7 @@ DT_Performance_Test_Create ( test_ptr->ia = ia; test_ptr->cmd = &pt_ptr->Params.u.Performance_Cmd; - ret = dat_ia_query (test_ptr->ia, - NULL, - &test_ptr->ia_attr, - NULL); + ret = ia->common.provider->ia_query(ia, NULL, &test_ptr->ia_attr, NULL); if ( DAT_SUCCESS != ret) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ia_query error: %s\n", Index: linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (revision 2575) +++ linux-kernel-return/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -165,10 +165,8 @@ int DT_queryinfo_basic (Params_t *params { if (result_wanted == DAT_SUCCESS) { - rc = dat_ia_query (ia, - &evd, - &ia_attributes, - &provider_attributes); + rc = ia->common.provider->ia_query(ia, &evd, &ia_attributes, + &provider_attributes); } else if (result_wanted == DAT_INVALID_PARAMETER) { @@ -177,17 +175,13 @@ int DT_queryinfo_basic (Params_t *params * NULL out ia_attr and for the DAT_IA_ATTR_MASK to * have values */ - rc = dat_ia_query (ia, - &evd, - NULL, - &provider_attributes); + rc = ia->common.provider->ia_query(ia, &evd, NULL, + &provider_attributes); } else if (result_wanted == DAT_INVALID_HANDLE) { - rc = dat_ia_query (ia, - &evd, - &ia_attributes, - &provider_attributes); + rc = ia->common.provider->ia_query(ia, &evd, &ia_attributes, + &provider_attributes); } } Index: linux-kernel-return/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel-return/test/dapltest/test/dapl_test_util.c (revision 2575) +++ linux-kernel-return/test/dapltest/test/dapl_test_util.c (working copy) @@ -44,10 +44,8 @@ DT_query ( Per_Test_Data_t *pt_ptr, phead = pt_ptr->Params.phead; /* Query the IA */ - ret = dat_ia_query (ia, - &async_evd_hdl, - &pt_ptr->ia_attr, - &pt_ptr->provider_attr); + ret = ia->common.provider->ia_query(ia, &async_evd_hdl, &pt_ptr->ia_attr, + &pt_ptr->provider_attr); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_query error: %s\n", Index: linux-kernel-return/dat-provider/dapl_provider.c =================================================================== --- linux-kernel-return/dat-provider/dapl_provider.c (revision 2575) +++ linux-kernel-return/dat-provider/dapl_provider.c (working copy) @@ -67,7 +67,7 @@ static struct dat_provider g_dapl_provid .extension = NULL, .ia_open_func = &dapl_ia_open, - .ia_query_func = &dapl_ia_query, + .ia_query = &dapl_ia_query, .ia_close_func = &dapl_ia_close, .ia_memtype_hint_func = &dapl_ia_memtype_hint, Index: linux-kernel-return/dat/api.c =================================================================== --- linux-kernel-return/dat/api.c (revision 2575) +++ linux-kernel-return/dat/api.c (working copy) @@ -405,8 +405,8 @@ u32 dat_ia_close(struct dat_ia *ia, enum provider = ia->common.provider; ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, - &provider_attr))) + if (DAT_SUCCESS != (status = provider->ia_query(ia, NULL, NULL, + &provider_attr))) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "dat_ia_query for IA %s failed\n", ia_name); else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) Index: linux-kernel-return/dat/dat.h =================================================================== --- linux-kernel-return/dat/dat.h (revision 2575) +++ linux-kernel-return/dat/dat.h (working copy) @@ -1057,10 +1057,6 @@ typedef u32 (*DAT_IA_OPENV_FUNC)(const c typedef u32 (*DAT_IA_CLOSE_FUNC)(struct dat_ia *, enum dat_close_flags); -typedef u32 (*DAT_IA_QUERY_FUNC)(struct dat_ia *, struct dat_evd **, - struct dat_ia_attr *, - struct dat_provider_attr *); - typedef u32 (*DAT_CR_QUERY_FUNC)(struct dat_cr *, struct dat_cr_param *); typedef u32 (*DAT_CR_ACCEPT_FUNC)(struct dat_cr *, struct dat_ep *, int, @@ -1223,7 +1219,8 @@ struct dat_provider { void *extension; DAT_IA_OPEN_FUNC ia_open_func; - DAT_IA_QUERY_FUNC ia_query_func; + u32 (*ia_query)(struct dat_ia *, struct dat_evd **, + struct dat_ia_attr *, struct dat_provider_attr *); DAT_IA_CLOSE_FUNC ia_close_func; DAT_IA_MEMTYPE_HINT_FUNC ia_memtype_hint_func; @@ -1396,14 +1393,6 @@ static inline u32 dat_ia_memtype_hint(st preferred_alignment); } -static inline u32 dat_ia_query(struct dat_ia *ia, struct dat_evd **async_evd, - struct dat_ia_attr *ia_attr, - struct dat_provider_attr *provider_attr) -{ - return DAT_CALL_PROVIDER_FUNC( - ia_query_func, ia, async_evd, ia_attr, provider_attr); -} - static inline u32 dat_cr_accept(struct dat_cr *cr, struct dat_ep *ep, int private_data_size, const void *private_data) { From jlentini at netapp.com Thu Jun 9 11:27:24 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 14:27:24 -0400 (EDT) Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <20050609094759.GA5947@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> Message-ID: Tom, I think this change has a lot of merits, but I would prefer to defer this sort of change until we have a more stable provider. Once we have removed the bugs and finished our update of the provider code (removing the custom DAPL data structures, etc.), then I think we should revisit this topic. Does that sound reasonable? james On Thu, 9 Jun 2005, Christoph Hellwig wrote: > On Thu, Jun 09, 2005 at 11:37:25AM +0300, Itamar Rabenstein wrote: >> Hi Tom, >> >> It looks very good to me but i want to ask why do we need the dat_common >> struct ? >> >> as we can see in dat.h dat_common is defined as : >> >> struct dat_common { >> struct dat_provider *provider; >> union dat_context context; >> }; >> and is being use as data member of any dat_* struct >> There are 9 places were we instance dat_common >> (dat_ia, dat_ep, dat_evd, ...) >> >> i think that that dat_* struct should be like this >> struct dat_ia { >> struct dat_provider *provider; >> union dat_context context; >> }; >> instaed of >> struct dat_ia { >> struct dat_common common; >> }; >> >> if we do it (only 5 lines more in code ) >> every time we want to call a function it will be >> ret = ia->provider->dat_ia_query(ia, NULL, >> instaed of >> ret = ia->common.provider->dat_ia_query(ia, NULL, >> >> i think that this is more readable. > > It is. B?ut I think it'll change even more later on - having a pointer > to the method table in every object is an enormous waste of space, in > the end the method should probably be only in a hca-level object, and > all other objects should have public pointers to it. > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From jlentini at netapp.com Thu Jun 9 11:35:00 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 14:35:00 -0400 (EDT) Subject: [openib-general] kdaptest wedges server In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> Message-ID: Josh, Was there any information about what occurred? Loading the the DAT modules with modprobe dat dbg_mask=0xff modprobe ib_dat_provider dbg_mask=0xffff will produce a lot of debug information that might help. james On Thu, 9 Jun 2005, Itamar Rabenstein wrote: > No , I dont see the problem at my system . > > Itamar > >> -----Original Message----- >> From: James Lentini [mailto:jlentini at netapp.com] >> Sent: Thursday, June 09, 2005 6:03 PM >> To: Itamar Rabenstein >> Cc: Josh England; openib-general >> Subject: RE: [openib-general] kdaptest wedges server >> >> >> >> Itamar, >> >> Were you able to get to the bottom of this? >> >> james >> >> On Thu, 2 Jun 2005, Itamar Rabenstein wrote: >> >> itamar>Hi Josh, >> itamar> >> itamar>Can you run it with debug option ( -d ) >> itamar>and tell me what is the last debug print on the server side ? >> itamar>when you say "kill" can you hit ^c or is it hanging >> the station ? >> itamar> >> itamar>what is system configuration ? (arch , Os ,kernel) >> itamar> >> itamar> Itamar >> itamar> >> itamar> >> itamar>> -----Original Message----- >> itamar>> From: Josh England [mailto:jjengla at sandia.gov] >> itamar>> Sent: Thursday, June 02, 2005 9:00 PM >> itamar>> To: openib-general >> itamar>> Subject: [openib-general] kdaptest wedges server >> itamar>> >> itamar>> >> itamar>> Hi, >> itamar>> >> itamar>> The kdapltest program wedges consistently wedges a >> node. I just tried >> itamar>> it with Revision 2523, and it runs fine the first couple of >> itamar>> times. The >> itamar>> third time took a while, and the fourth time killed >> the server. >> itamar>> >> itamar>> The server is running: >> itamar>> kdapltest -T S -D mthca0a >> itamar>> >> itamar>> The client runs this several times in succession: >> itamar>> kdapltest -T T -D mthca0a -s client SR 256 >> server SR 256 >> itamar>> >> itamar>> -JE >> itamar>> >> itamar>> >> itamar>> _______________________________________________ >> itamar>> openib-general mailing list >> itamar>> openib-general at openib.org >> itamar>> http://openib.org/mailman/listinfo/openib-general >> itamar>> >> itamar>> To unsubscribe, please visit >> itamar>> http://openib.org/mailman/listinfo/openib-general >> itamar>> >> itamar>_______________________________________________ >> itamar>openib-general mailing list >> itamar>openib-general at openib.org >> itamar>http://openib.org/mailman/listinfo/openib-general >> itamar> >> itamar>To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> itamar> >> > From roland at topspin.com Thu Jun 9 11:29:28 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 09 Jun 2005 11:29:28 -0700 Subject: [openib-general] Re: uverbs_main cleanup In-Reply-To: <1118094867.4493.72.camel@hal.voltaire.com> (Hal Rosenstock's message of "06 Jun 2005 17:54:28 -0400") References: <1118094867.4493.72.camel@hal.voltaire.com> Message-ID: <52zmtz9yev.fsf@topspin.com> Hal> Hi Roland, Should the order of unregistering the uverbs event Hal> fs and the unmounting be reversed or does this not matter ? I don't think this makes any practical difference but in the interest of consistency between initialization and cleanup order, I made this change. From tduffy at sun.com Thu Jun 9 11:51:32 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 09 Jun 2005 11:51:32 -0700 Subject: [openib-general] [PATCHv6] SDP: Allow SDP to compile on 2.6.12-rc6 Message-ID: <1118343092.32250.15.camel@duffman> This updates the 2.6.12 SDP patch against the state change patch Libor checked into r2577. It also fixes a bug I was seeing regarding freeing of resources when tearing down a connection. I am still trying to track down a bug with this patch that occurs when you try to initiate an active connection and there is no passive listener ready. This can cause a panic on the passive side if ib_sdp is loaded. 2.6.12 final should be out any day now, so please apply when that happens. Signed-off-by: Tom Duffy Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -91,8 +91,8 @@ static int sdp_rcvd_disconnect(struct sd /* * async notification. POLL_HUP on full duplex close only. */ - sdp_inet_wake_generic(conn->sk); - sk_wake_async(conn->sk, 1, band); + sdp_inet_wake_generic(sk_sdp(conn)); + sk_wake_async(sk_sdp(conn), 1, band); return 0; error: @@ -1213,7 +1213,7 @@ int sdp_event_recv(struct sdp_opt *conn, * If data was consumed by the protocol, signal * the user. */ - sdp_inet_wake_recv(conn->sk, conn->byte_strm); + sdp_inet_wake_recv(sk_sdp(conn), conn->byte_strm); /* * It's possible that a new recv buffer advertisment opened up the * recv window and we can flush buffered send data Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -129,9 +129,9 @@ MODULE_PARM_DESC(sdp_debug_level, */ void sdp_inet_wake_send(struct sock *sk) { - struct sdp_opt *conn; + struct sdp_opt *conn = sdp_sk(sk); - if (!sk || !(conn = SDP_GET_CONN(sk))) + if (!sk) return; if (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) && @@ -294,27 +294,22 @@ error: static int sdp_inet_release(struct socket *sock) { struct sdp_opt *conn; - struct sock *sk; + struct sock *sk = sock->sk; int result; long timeout; u32 flags; - if (!sock->sk) { + if (!sk) { sdp_dbg_warn(NULL, "release empty <%d:%d> flags <%08lx>", sock->type, sock->state, sock->flags); return 0; } - sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "RELEASE: linger <%d:%lu> data <%d:%d>", sock_flag(sk, SOCK_LINGER), sk->sk_lingertime, conn->byte_strm, conn->src_recv); - /* - * clear out sock, so we only do this once. - */ - sock->sk = NULL; sdp_conn_lock(conn); conn->shutdown = SHUTDOWN_MASK; @@ -412,6 +407,7 @@ done: * finally drop socket reference. (socket API reference) */ sock_orphan(sk); + sock->sk = NULL; sdp_conn_unlock(conn); sdp_conn_put(conn); @@ -431,7 +427,7 @@ static int sdp_inet_bind(struct socket * int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "BIND: family <%d> addr <%08x:%04x>", addr->sin_family, addr->sin_addr.s_addr, addr->sin_port); @@ -522,7 +518,7 @@ static int sdp_inet_connect(struct socke int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "CONNECT: family <%d> addr <%08x:%04x>", addr->sin_family, addr->sin_addr.s_addr, addr->sin_port); @@ -684,7 +680,7 @@ static int sdp_inet_listen(struct socket int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "LISTEN: addr <%08x:%04x> backlog <%04x>", conn->src_addr, conn->src_port, backlog); @@ -745,7 +741,7 @@ static int sdp_inet_accept(struct socket long timeout; listen_sk = listen_sock->sk; - listen_conn = SDP_GET_CONN(listen_sk); + listen_conn = sdp_sk(listen_sk); sdp_dbg_ctrl(listen_conn, "ACCEPT: addr <%08x:%04x>", listen_conn->src_addr, listen_conn->src_port); @@ -801,7 +797,7 @@ static int sdp_inet_accept(struct socket goto listen_done; } } else { - accept_sk = accept_conn->sk; + accept_sk = sk_sdp(accept_conn); switch (accept_conn->state) { case SDP_CONN_ST_REQ_RECV: @@ -870,7 +866,7 @@ static int sdp_inet_getname(struct socke struct sdp_opt *conn; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "GETNAME: src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -910,7 +906,7 @@ static unsigned int sdp_inet_poll(struct * recheck the falgs on being woken. */ sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "POLL: socket flags <%08lx>", sock->flags); @@ -997,7 +993,7 @@ static int sdp_inet_ioctl(struct socket int value; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "IOCTL: command <%d> argument <%08lx>", cmd, arg); /* @@ -1119,7 +1115,7 @@ static int sdp_inet_setopt(struct socket int result = 0; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "SETSOCKOPT: level <%d> option <%d>", level, optname); @@ -1186,7 +1182,7 @@ static int sdp_inet_getopt(struct socket int len; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "GETSOCKOPT: level <%d> option <%d>", level, optname); @@ -1244,7 +1240,7 @@ static int sdp_inet_shutdown(struct sock int result = 0; struct sdp_opt *conn; - conn = SDP_GET_CONN(sock->sk); + conn = sdp_sk(sock->sk); sdp_dbg_ctrl(conn, "SHUTDOWN: flag <%d>", flag); /* @@ -1352,7 +1348,7 @@ static int sdp_inet_create(struct socket sock->ops = &lnx_stream_ops; sock->state = SS_UNCONNECTED; - sock_graft(conn->sk, sock); + sock_graft(sk_sdp(conn), sock); conn->pid = current->pid; @@ -1379,6 +1375,11 @@ static struct net_proto_family sdp_proto /* * SDP host module load/unload functions */ +struct proto sdp_sk_proto = { + .name = "SDP", + .owner = THIS_MODULE, + .obj_size = sizeof(struct sdp_opt), +}; /* * sdp_init - initialize the sdp host module @@ -1389,6 +1390,12 @@ static int __init sdp_init(void) sdp_dbg_init("SDP module load."); + result = proto_register(&sdp_sk_proto, 1); + if (result < 0) { + sdp_warn("INIT: Error <%d> registering sk proto,", result); + goto error_proto; + } + /* * proc entries */ @@ -1465,6 +1472,8 @@ error_link: error_advt: sdp_main_proc_cleanup(); error_proc: + proto_unregister(&sdp_sk_proto); +error_proto: return result; /* success */ } @@ -1498,6 +1507,8 @@ static void __exit sdp_exit(void) * proc tables */ sdp_main_proc_cleanup(); + + proto_unregister(&sdp_sk_proto); } module_init(sdp_init); Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_read.c =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_read.c (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_read.c (working copy) @@ -185,7 +185,7 @@ int sdp_event_read(struct sdp_opt *conn, */ conn->byte_strm += result; - sdp_inet_wake_recv(conn->sk, conn->byte_strm); + sdp_inet_wake_recv(sk_sdp(conn), conn->byte_strm); } else { if (result < 0) sdp_dbg_warn(conn, "Error <%d> receiving buff", @@ -229,7 +229,7 @@ int sdp_event_read(struct sdp_opt *conn, iocb->flags &= ~(SDP_IOCB_F_ACTIVE | SDP_IOCB_F_RDMA_R); - if (conn->sk->sk_rcvlowat > iocb->post) + if (sk_sdp(conn)->sk_rcvlowat > iocb->post) break; iocb = sdp_iocb_q_get_head(&conn->r_pend); Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_send.c =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_send.c (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_send.c (working copy) @@ -1783,7 +1783,7 @@ static int sdp_inet_write_cancel(struct /* * lock the socket while we operate. */ - conn = SDP_GET_CONN(si->sock->sk); + conn = sdp_sk(si->sock->sk); sdp_conn_lock(conn); sdp_dbg_ctrl(conn, "Cancel Write IOCB. <%08x:%04x> <%08x:%04x>", @@ -1993,7 +1993,7 @@ int sdp_send_flush(struct sdp_opt *conn) /* * see if there is enough buffer to wake/notify writers */ - sdp_inet_wake_send(conn->sk); /* conn->sk->write_space(conn->sk); */ + sdp_inet_wake_send(sk_sdp(conn)); /* conn->sk->write_space(conn->sk); */ return 0; done: @@ -2022,7 +2022,7 @@ int sdp_inet_send(struct kiocb *req, str oob = (msg->msg_flags & MSG_OOB); sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "send state <%04x> size <%Zu> flags <%08x>", conn->state, size, msg->msg_flags); Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_actv.c =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (working copy) @@ -40,6 +40,7 @@ void sdp_cm_actv_error(struct sdp_opt *conn, int error) { int result; + struct sock *sk; /* * error value is positive error. * @@ -92,11 +93,12 @@ void sdp_cm_actv_error(struct sdp_opt *c conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; - if (conn->sk->sk_socket) - conn->sk->sk_socket->state = SS_UNCONNECTED; + sk = sk_sdp(conn); + if (sk->sk_socket) + sk->sk_socket->state = SS_UNCONNECTED; sdp_iocb_q_cancel_all(conn, (0 - error)); - sdp_inet_wake_error(conn->sk); + sdp_inet_wake_error(sk); } /* @@ -113,7 +115,7 @@ static int sdp_cm_actv_establish(struct conn->src_addr, conn->src_port, conn->dst_addr, conn->dst_port); - sk = conn->sk; + sk = sk_sdp(conn); qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL); if (!qp_attr) @@ -541,7 +543,7 @@ int sdp_cm_connect(struct sdp_opt *conn) result = sdp_link_path_lookup(htonl(conn->dst_addr), htonl(conn->src_addr), - conn->sk->sk_bound_dev_if, + sk_sdp(conn)->sk_bound_dev_if, sdp_cm_path_complete, conn, &conn->plid); Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -75,6 +75,8 @@ static u32 sdp_psn_generate(void) void sdp_conn_inet_error(struct sdp_opt *conn, int error) { + struct sock *sk; + sdp_dbg_ctrl(conn, "report connection error <%d>", error); /* * the connection has failed, move to error, and notify anyone @@ -88,11 +90,12 @@ void sdp_conn_inet_error(struct sdp_opt conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; - if (conn->sk->sk_socket) - conn->sk->sk_socket->state = SS_UNCONNECTED; + sk = sk_sdp(conn); + if (sk->sk_socket) + sk->sk_socket->state = SS_UNCONNECTED; sdp_iocb_q_cancel_all(conn, -error); - sdp_inet_wake_error(conn->sk); + sdp_inet_wake_error(sk); } void sdp_conn_abort(struct sdp_opt *conn) @@ -402,7 +405,7 @@ int sdp_inet_port_get(struct sdp_opt *co static s32 rover = -1; unsigned long flags; - sk = conn->sk; + sk = sk_sdp(conn); /* * lock table */ @@ -413,7 +416,7 @@ int sdp_inet_port_get(struct sdp_opt *co if (port > 0) { for (look = dev_root_s.bind_list, port_ok = 1; look; look = look->bind_next) { - srch = look->sk; + srch = sk_sdp(look); /* * 1) same port * 2) linux force reuse is off. @@ -842,17 +845,8 @@ void sdp_conn_destruct(struct sdp_opt *c if (dump) sdp_conn_state_dump(conn); - /* - * free the OS socket structure - */ - if (!conn->sk) - sdp_dbg_warn(conn, "destruct, no socket! continuing."); - else { - sk_free(conn->sk); - conn->sk = NULL; - } - kmem_cache_free(dev_root_s.conn_cache, conn); + sk_free(sk_sdp(conn)); } /* @@ -1197,6 +1191,8 @@ error_attr: return result; } +extern struct proto sdp_sk_proto; + /* * sdp_conn_alloc - allocate a new socket, and init. */ @@ -1206,8 +1202,7 @@ struct sdp_opt *sdp_conn_alloc(int prior struct sock *sk; int result; - sk = sk_alloc(dev_root_s.proto, priority, - sizeof(struct inet_sock), dev_root_s.sock_cache); + sk = sk_alloc(dev_root_s.proto, priority, &sdp_sk_proto, 1); if (!sk) { sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%d>", dev_root_s.proto, priority); @@ -1230,23 +1225,8 @@ struct sdp_opt *sdp_conn_alloc(int prior sk->sk_state_change = sdp_inet_wake_generic; sk->sk_data_ready = sdp_inet_wake_recv; sk->sk_error_report = sdp_inet_wake_error; - /* - * Allocate must be called from process context, since QP - * create/modifies must be in that context. - */ - conn = kmem_cache_alloc(dev_root_s.conn_cache, priority); - if (!conn) { - sdp_dbg_warn(conn, "connection alloc error. <%d>", priority); - result = -ENOMEM; - goto error; - } - memset(conn, 0, sizeof(struct sdp_opt)); - /* - * The STRM interface specific data is map/cast over the TCP specific - * area of the sock. - */ - SDP_SET_CONN(sk, conn); + conn = sdp_sk(sk); SDP_CONN_ST_INIT(conn); conn->cm_id = NULL; @@ -1263,7 +1243,6 @@ struct sdp_opt *sdp_conn_alloc(int prior conn->parent = NULL; conn->pid = 0; - conn->sk = sk; conn->hashent = SDP_DEV_SK_INVALID; conn->flags = 0; conn->shutdown = 0; @@ -1369,7 +1348,7 @@ struct sdp_opt *sdp_conn_alloc(int prior sdp_dbg_warn(conn, "Error <%d> conn table insert <%d:%d>", result, dev_root_s.sk_entry, dev_root_s.sk_size); - goto error_conn; + goto error; } /* * set reference @@ -1383,8 +1362,6 @@ struct sdp_opt *sdp_conn_alloc(int prior * done */ return conn; -error_conn: - kmem_cache_free(dev_root_s.conn_cache, conn); error: sk_free(sk); return NULL; @@ -1553,7 +1530,7 @@ int sdp_proc_dump_conn_data(char *buffer continue; conn = dev_root_s.sk_array[counter]; - sk = conn->sk; + sk = sk_sdp(conn); offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_FORM, conn->hashent, @@ -2038,26 +2015,6 @@ int sdp_conn_table_init(int proto_family goto error_iocb; } - dev_root_s.conn_cache = kmem_cache_create("sdp_conn", - sizeof(struct sdp_opt), - 0, SLAB_HWCACHE_ALIGN, - NULL, NULL); - if (!dev_root_s.conn_cache) { - sdp_warn("Failed to initialize connection cache."); - result = -ENOMEM; - goto error_conn; - } - - dev_root_s.sock_cache = kmem_cache_create("sdp_sock", - sizeof(struct inet_sock), - 0, SLAB_HWCACHE_ALIGN, - NULL, NULL); - if (!dev_root_s.sock_cache) { - sdp_warn("Failed to initialize sock cache."); - result = -ENOMEM; - goto error_sock; - } - /* * start listening */ @@ -2066,7 +2023,7 @@ int sdp_conn_table_init(int proto_family if (!dev_root_s.listen_id) { sdp_warn("Failed to create listen connection identifier."); result = -ENOMEM; - goto error_cm_id; + goto error_conn; } result = ib_cm_listen(dev_root_s.listen_id, @@ -2083,10 +2040,6 @@ int sdp_conn_table_init(int proto_family return 0; error_listen: ib_destroy_cm_id(dev_root_s.listen_id); -error_cm_id: - kmem_cache_destroy(dev_root_s.sock_cache); -error_sock: - kmem_cache_destroy(dev_root_s.conn_cache); error_conn: sdp_main_iocb_cleanup(); error_iocb: @@ -2128,14 +2081,6 @@ void sdp_conn_table_clear(void) */ sdp_main_iocb_cleanup(); /* - * delete conn cache - */ - kmem_cache_destroy(dev_root_s.conn_cache); - /* - * delete sock cache - */ - kmem_cache_destroy(dev_root_s.sock_cache); - /* * stop listening */ ib_destroy_cm_id(dev_root_s.listen_id); Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_recv.c =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -786,7 +786,7 @@ static int sdp_recv_buff_iocb_pending(st */ if (!iocb->len || (!conn->src_recv && - !(conn->sk->sk_rcvlowat > iocb->post))) { + !(sk_sdp(conn)->sk_rcvlowat > iocb->post))) { /* * complete IOCB */ @@ -840,7 +840,7 @@ int sdp_recv_buff(struct sdp_opt *conn, */ if (buff->flags & SDP_BUFF_F_OOB_PEND) { conn->rcv_urg_cnt++; - sdp_inet_wake_urg(conn->sk); + sdp_inet_wake_urg(sk_sdp(conn)); } /* * loop while there are available IOCB's, break if there is no @@ -938,7 +938,7 @@ static int sdp_inet_read_cancel(struct k /* * lock the socket while we operate. */ - conn = SDP_GET_CONN(si->sock->sk); + conn = sdp_sk(si->sock->sk); sdp_conn_lock(conn); sdp_dbg_ctrl(conn, "Cancel Read IOCB. <%08x:%04x> <%08x:%04x>", @@ -1091,7 +1091,7 @@ static int sdp_inet_recv_urg(struct sock int result = 0; u8 value; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); if (sock_flag(sk, SOCK_URGINLINE) || !conn->rcv_urg_cnt) return -EINVAL; @@ -1178,7 +1178,7 @@ int sdp_inet_recv(struct kiocb *req, st struct sdpc_buff_q peek_queue; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "state <%08x> size <%Zu> pending <%d> falgs <%08x>", conn->state, size, conn->byte_strm, flags); Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_conn.h =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -108,16 +108,18 @@ enum sdp_mode { */ #define SDP_MSG_EVENT_TABLE_SIZE 0x20 -/* - * connection handle within a socket. - */ -#define SDP_GET_CONN(sk) \ - (*((struct sdp_opt **)&(sk)->sk_protinfo)) -#define SDP_SET_CONN(sk, conn) \ - (*((struct sdp_opt **)&(sk)->sk_protinfo) = (conn)) +static inline struct sdp_opt *sdp_sk(struct sock *sk) +{ + return (struct sdp_opt *)sk; +} + +static inline struct sock *sk_sdp(struct sdp_opt *conn) +{ + return (struct sock *)conn; +} #define SDP_CONN_SET_ERR(conn, val) \ - ((conn)->error = (conn)->sk->sk_err = (val)) + ((conn)->error = sk_sdp(conn)->sk_err = (val)) #define SDP_CONN_GET_ERR(conn) \ ((conn)->error) @@ -179,10 +181,15 @@ struct sdp_conn_lock { * SDP Connection structure. */ struct sdp_opt { + /* + * inet_sock must be first member of sdp_opt + * NOTE: this depends on inet_sock having struct sock as its + * first member + */ + struct inet_sock in; __s32 hashent; /* connection ID/hash entry */ atomic_t refcnt; /* connection reference count. */ - struct sock *sk; /* * SDP specific data */ @@ -493,7 +500,7 @@ static inline int sdp_conn_error(struct * lock, however the linux socket error, needs to be xchg'd since the * SO_ERROR getsockopt happens outside of the connection lock. */ - int error = xchg(&conn->sk->sk_err, 0); + int error = xchg(&sk_sdp(conn)->sk_err, 0); SDP_CONN_SET_ERR(conn, 0); return -error; Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_pass.c =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (working copy) @@ -87,7 +87,7 @@ int sdp_cm_pass_establish(struct sdp_opt goto error; } - sdp_inet_wake_send(conn->sk); + sdp_inet_wake_send(sk_sdp(conn)); kfree(qp_attr); return 0; @@ -271,8 +271,8 @@ static int sdp_cm_listen_lookup(struct s /* * check backlog */ - listen_sk = listen_conn->sk; - sk = conn->sk; + listen_sk = sk_sdp(listen_conn); + sk = sk_sdp(conn); if (listen_conn->backlog_cnt > listen_conn->backlog_max) { sdp_dbg_warn(listen_conn, @@ -305,13 +305,16 @@ static int sdp_cm_listen_lookup(struct s */ sk->sk_lingertime = listen_sk->sk_lingertime; sk->sk_rcvlowat = listen_sk->sk_rcvlowat; - sk->sk_debug = listen_sk->sk_debug; - sk->sk_localroute = listen_sk->sk_localroute; + if (sock_flag(listen_sk, SOCK_DBG)) + sock_set_flag(sk, SOCK_DBG); + if (sock_flag(listen_sk, SOCK_LOCALROUTE)) + sock_set_flag(sk, SOCK_LOCALROUTE); sk->sk_sndbuf = listen_sk->sk_sndbuf; sk->sk_rcvbuf = listen_sk->sk_rcvbuf; sk->sk_no_check = listen_sk->sk_no_check; sk->sk_priority = listen_sk->sk_priority; - sk->sk_rcvtstamp = listen_sk->sk_rcvtstamp; + if (sock_flag(listen_sk, SOCK_RCVTSTAMP)) + sock_set_flag(sk, SOCK_RCVTSTAMP); sk->sk_rcvtimeo = listen_sk->sk_rcvtimeo; sk->sk_sndtimeo = listen_sk->sk_sndtimeo; sk->sk_reuse = listen_sk->sk_reuse; @@ -450,7 +453,7 @@ int sdp_cm_req_handler(struct ib_cm_id * goto done; } /* - * Lock the new connection before modifyingg it into any tables. + * Lock the new connection before modifying it into any tables. */ sdp_conn_lock(conn); /* Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_dev.h =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (revision 2577) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (working copy) @@ -197,11 +197,6 @@ struct sdev_root { * SDP wide listen */ struct ib_cm_id *listen_id; /* listen handle */ - /* - * cache's - */ - kmem_cache_t *conn_cache; - kmem_cache_t *sock_cache; }; #endif /* _SDP_DEV_H */ From libor at topspin.com Thu Jun 9 11:59:13 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 9 Jun 2005 11:59:13 -0700 Subject: [openib-general] [PATCHv6] SDP: Allow SDP to compile on 2.6.12-rc6 In-Reply-To: <1118343092.32250.15.camel@duffman>; from tduffy@sun.com on Thu, Jun 09, 2005 at 11:51:32AM -0700 References: <1118343092.32250.15.camel@duffman> Message-ID: <20050609115913.A28934@topspin.com> On Thu, Jun 09, 2005 at 11:51:32AM -0700, Tom Duffy wrote: > > I am still trying to track down a bug with this patch that occurs when > you try to initiate an active connection and there is no passive > listener ready. This can cause a panic on the passive side if ib_sdp is > loaded. Do you want to send the stack trace from the panic? One thing to note is when a passive connection is received, the new connection and socket structures are allocated even if there is no one listening. Only once they've been allocated does the check for listner occur, and if there is not one present, then structures are deleted. > 2.6.12 final should be out any day now, so please apply when that > happens. Will do. -Libor From halr at voltaire.com Thu Jun 9 11:57:14 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 Jun 2005 14:57:14 -0400 Subject: [openib-general] [PATCH] [TRIVIAL] [kdapl] CM: Message and commentary changes Message-ID: <1118343434.4510.15.camel@localhost.localdomain> Message and commentary changes to make consistent with ongoing code changes Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== --- dapl_openib_cm.c (revision 2577) +++ dapl_openib_cm.c (working copy) @@ -81,8 +81,7 @@ int status; if (conn->ep->qp == NULL) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rep_recv: invalid qp " - "handle\n"); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rep_recv: invalid qp\n"); goto disc; } @@ -416,6 +415,7 @@ * ep * remote_ia_address * remote_conn_qual + * timeout * private_data_size size of private data and structure * private_data pointats_entry->semer to private data structure * @@ -440,7 +440,7 @@ ep_ptr = (struct dapl_ep *)ep; if (ep_ptr->qp == NULL) { - printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); + printk(KERN_ERR "dapl_ib_connect: qp invalid\n"); return DAT_INTERNAL_ERROR; } @@ -550,7 +550,7 @@ * after the final disconnect callback has occurred. * * Input: - * ep_ptr struct dapl_ep + * ep_ptr pointer to dapl_ep struct * active Indicates active side of connection * * Output: @@ -581,7 +581,7 @@ "on transition to error\n", status); } else dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_disconnect_clean: " - "invalid qp handle\n"); + "invalid qp\n"); } /* @@ -590,7 +590,7 @@ * Move the QP to INIT state again. * * Input: - * ep_ptr DAPL_EP + * ep_ptr pointer to dapl_ep struct * * Output: * none @@ -641,8 +641,8 @@ * Have the CM set up a connection listener. * * Input: - * ib_hca_handle HCA handle - * qp_handle QP handle + * ia_ptr pointer to dapl_ia struct + * sp_ptr pointer to dapl_sp struct * * Output: * none @@ -685,8 +685,8 @@ * Have the CM remove a connection listener. * * Input: - * ia IA handle - * ServiceID IB Channel Service ID + * ia_ptr pointer to dapl_ia struct + * sp_ptr pointer to dapl_sp struct * * Output: * none @@ -719,7 +719,7 @@ * Perform necessary steps to reject a connection * * Input: - * cm + * cm pointer to dapl_cm_id struct * * Output: * none @@ -863,7 +863,7 @@ * * Input: * cr - * private data structure handle (only if not using ATS) + * private data structure pointer (only if not using ATS) * * Output: * remote_ia_address: where to place the remote address From halr at voltaire.com Thu Jun 9 12:11:49 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 Jun 2005 15:11:49 -0400 Subject: [openib-general] Re: [PATCHv3] kDAPL: remove use of HANDLE's (vs. r2564) In-Reply-To: <1118263490.9977.2.camel@duffman> References: <1118093694.5735.13.camel@duffman> <1118101725.30742.2.camel@duffman> <1118261806.31987.21.camel@duffman> <1118263490.9977.2.camel@duffman> Message-ID: <1118344308.4510.27.camel@localhost.localdomain> On Wed, 2005-06-08 at 16:44, Tom Duffy wrote: > This just in! Don't know if was caused by your patches or mine, but I > ran the server on 192.168.0.26, and tried to connect with the client, it > didn't connect right away, so I did a control-c. Then, the kernel > panic'ed: > > [root at sins-stinger-10 ~]# ./kdapltest -T Q -d -s 192.168.0.26 -D mthca0a > Server Name: 192.168.0.26 > Server Net Address: 192.168.0.26 > DT_cs_Client: Starting Test ... > DT_cs_Client: IA mthca0a opened > DT_cs_Client: EP created > ***** DAPL Characteristics ***** > Provider: mthca0a Version 1.0 DAPL 1.2 > Adapter: Generic InfiniBand HCA by Linux Version 0.0 > Supporting: > 64512 EPs with 65535 DTOs and 0 in RDMA/RDs and 0ut RDMA/RDs each > 65408 EVDs of up to 65535 entries (default S/R size is 256/256) > IOVs of up to 28 elements > 131056 LMRs (and 131056 RMRs) of up to 0xffffffffffffffff bytes > Maximum MTU 0x80000000 bytes, RDMA 0x80000000 bytes > Maximum Private data size 92 bytes > ***** ***** ***** ***** ***** ***** > DT_cs_Client: Posting 1 recv buffer > DT_cs_Client: Connect Endpoint > DT_cs_Client: Await connection ... > > <-- I DID A CONTROL-C HERE --> > > [root at sins-stinger-10 ~]# dapl_path_comp_handler: path resolution failed -110 retry 1802201964!!! > dl_ibpa_atht:_c romeqp__henand:dl peren: d epff_pfftr81 000x65ab622b6bbe46b0 6bal6bre6bad6by > completed? status 3 > dapl_path_comp_handler: path resolution failed -110 retry 1802201965!!! > dapl_path_comp_handler: ep_ptr 0x6b6b6b6b6b6b6b6b > general protection fault: 0000 [1] SMP > CPU 0 > Modules linked in: kdapltest ib_dat_provider dat ib_at ib_ipoib ib_sdp ib_cm md5 ipv6 parport_pc lp parport autofs4 nfs lockd rfcomm l2cap bluetooth pcmcia yenta_socket rsrc_nonstatic pcmcia_core sunrpc ext3 jbd dm_mod video container button battery ac ohci_hcd tpm_nsc tpm i2c_amd756 i2c_core ib_mthca ib_sa ib_mad ib_core tg3 floppy xfs exportfs mptscsih mptbase sd_mod scsi_mod > Pid: 11881, comm: ib_at_wq/0 Not tainted 2.6.12-rc6openib > RIP: 0010:[] {:ib_dat_provider:dapl_evd_connection_callback+67} > RSP: 0018:ffff8100218b9dc8 EFLAGS: 00010296 > RAX: 6b6b6b6b6b6b6b6b RBX: ffff81005a22be40 RCX: 0000000000004008 > RDX: ffff810075baaaf8 RSI: ffffffff883141a0 RDI: 0000000000000048 > RBP: ffff81005a22be70 R08: 6b6b6b6b6b6b6b6b R09: 0000000000000000 > R10: 0000000000000010 R11: 0000000000000010 R12: ffff81007d9b8c50 > R13: ffff81005a22be40 R14: 0000000000000292 R15: ffffffff882f5280 > FS: 00002aaaaaad7d60(0000) GS:ffffffff804e7880(0000) knlGS:0000000000000000 > CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > CR2: 00007fffffd1dce6 CR3: 00000000769db000 CR4: 00000000000006e0 > Process ib_at_wq/0 (pid: 11881, threadinfo ffff8100218b8000, task ffff81004167caa0) > Stack: ffff81004167caa0 6b6b6b6b6b6b6b6b 0000000000000000 000040080000006e > ffff810075baaaf8 ffff81003fdc8f10 0000000000000000 0000000000000003 > ffff8100218b9e48 ffffffff88302d18 > Call Trace:{:ib_dat_provider:dapl_path_comp_handler+416} > {:ib_at:req_comp_work+27} {worker_thread+476} > {default_wake_function+0} {keventd_create_kthread+0} > {worker_thread+0} {keventd_create_kthread+0} > {kthread+217} {schedule_tail+64} > {child_rip+8} {keventd_create_kthread+0} > {kthread+0} {child_rip+0} > > > Code: 48 8b 80 90 00 00 00 48 89 44 24 28 c7 44 24 30 00 00 00 00 > RIP {:ib_dat_provider:dapl_evd_connection_callback+67} RSP > <<0>3>geSlneabra cl orprruotpteciotin:on s ftaaurtlt=f: ff00f80010 [072]5b aa<4af>S8,MP l en > 51CP2 = > 1R eon > e:M 0odx5ula2escf l07in1/ke0xd 5ain2c:f0 k71da. > teLastst iusb_erda: t_[_a]t( daibpl_i_dpoesibtro iy_b_cms_dpid +0ibxb_c9/m0 xbmde 5[ ibip_dv6at _pparorpviordter_p])c l > p<04>f0 p:a rp6bor t6 ba ut6bofs 64b n 6fsb l6bock 6db rf6bco mm6d l 62cba p6b bl 6uebto 6otbh 6bpc m6ciba 6bye > aP_sreocv keobtj:<4 s> tarsrtrc=f_nffonf8st10at07ic5b aapc8emc0,ia l_cenor=5e1 2 nRrpedcz : ex0xt3170 jfcbd2a 5/dm0x_m17od0fc v2aid5.eo > Lcoasntt aiusneerr: [bu] (ohkmciem_h_acdllo tc+pm0x_n61sc/0 xetp0 m[ xfi2s]c_)am > d750060: i2 0c_0co 0re0 i00b_ m0th0ca 00 i b_00sa 00ib _m00ad 0 i1b _c00or ec 0tg a32 fl0o6pp 0y0 xf00s 0ex0p > tf01s0 :mp 0ts0c si02h m00ptb 0as0e 00sd _m00od 0 s0c si00_mo 0d2 > 4>Pi 0d:0 <114>88 82,0 coa3mm : 06ib_ 0at0_w 0q/01 N00ot < > taiNentxted o 2bj.6:. s12ta-rrtc6=fopffenf8ib10 > 075RIbaP:ad 01001, 0:le[ ffRe88dz30on9ce:2d 0>]x1 70<4fc><2af5f/ff0xff17ff0f88c230a59c. > >L{:asibt _dusater_p: ro[]nec(ktimeonm__calallolbc+ac0xk+6167/0}xe > 0 R[xSPfs:] 0)01<48:> > f0f80010:0 5900c3 dd0c08 0 E0F LA00GS : 00000 010029 06 > 0R0AX:<4 6> b601b6 b600b6b a6b06b f6be RB00X: f04fff 081000 05a02 > e4010 0:RC X:00 00 000200 00000 000004 00008 2b > 00RD f 0ff0f 810000 750b1aa 0af08 R00SI : cbfff 0ff5ff f0488 310041a 00 0RD > I: 0000000000000048 > RBP: ffff81 > Message from syslogd at sins-st0inger-10 at Wed 0Jun 8 13:36:54 2005 ... > sins-5stinger-10 kerneal: general protection fault: 00020 [1] SMP > 2be70 R08: 6b6b6b6b6b6b6b6b R09: 0000000000000033 > R10: 0000000000000010 R11: 0000000000000010 R12: ffff81007d9b8cd0 > R13: ffff81005a22be40 R14: 0000000000000292 R15: ffffffff882f5280 > FS: 00002aaaaae0ae60(0000) GS:ffffffff804e7900(0000) knlGS:0000000000000000 > CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > CR2: 00007fffffbd2e96 CR3: 000000007cb2f000 CR4: 00000000000006e0 > Process ib_at_wq/1 (pid: 11882, threadinfo ffff810059c3c000, task ffff81004167d190) > Stack: 00000000ffffff92 6b6b6b6b6b6b6b6b 0000000000000000 000040087d9b8d10 > ffff810075baaaf8 0000000000000001 0000000000000092 0000000000000003 > ffff810059c3de48 ffffffff88302d18 > Call Trace:{:ib_dat_provider:dapl_path_comp_handler+416} > {:ib_at:req_comp_work+27} {worker_thread+476} > {default_wake_function+0} {keventd_create_kthread+0} > {worker_thread+0} {keventd_create_kthread+0} > {kthread+217} {schedule_tail+64} > {child_rip+8} {keventd_create_kthread+0} > {kthread+0} {child_rip+0} > > > Code: 48 8b 80 90 00 00 00 48 89 44 24 28 c7 44 24 30 00 00 00 00 > RIP {:ib_dat_provider:dapl_evd_connection_callback+67} RSP This is somewhat garbled but it appears that the connection was destroyed from under the path resolution (during dapl_ib_connect). It looks like there is another case to protect against :-( This one is particularly nasty. -- Hal From tduffy at sun.com Thu Jun 9 12:38:25 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 09 Jun 2005 12:38:25 -0700 Subject: [openib-general] [PATCHv6] SDP: Allow SDP to compile on 2.6.12-rc6 In-Reply-To: <20050609115913.A28934@topspin.com> References: <1118343092.32250.15.camel@duffman> <20050609115913.A28934@topspin.com> Message-ID: <1118345905.20343.22.camel@duffman> On Thu, 2005-06-09 at 11:59 -0700, Libor Michalek wrote: > Do you want to send the stack trace from the panic? One thing to > note is when a passive connection is received, the new connection > and socket structures are allocated even if there is no one listening. > Only once they've been allocated does the check for listner occur, > and if there is not one present, then structures are deleted. Yeah, I saw that. I think it is strange behavior, but alas... Here is the trace: [root at sins-stinger-10 ~]# ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at "/build1/tduffy/openib-work/linux-2.6.12-rc6-openi:352 invalid operand: 0000 [1] SMP CPU 1 Modules linked in: ib_sdp ib_cm md5 ipv6 parport_pc lp parport autofs4 nfs lockd rfcomm l2cap bluetooth pcmcia yenta_socket rsrc_nonstatic pcmcia_core sunrpc ext3 jbd dm_mod video container button battery ac ohci_hcd tpm_nsc tpm i2c_amd756 i2c_core ib_mthca ib_ipoib ib_sa ib_mad ib_core tg3 floppy xfs exportfs mptscsih mptbase sd_mod scsi_mod Pid: 946, comm: ib_cm/1 Not tainted 2.6.12-rc6openib RIP: 0010:[] {sk_alloc+297} RSP: 0018:ffff81001d465c68 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff81001c959790 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 000000000000001b RDI: ffffffff882f4900 RBP: ffffffff882f4500 R08: 0000000000000000 R09: ffff81001c959790 R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff882f3dc0 R13: 0000000000000020 R14: 0000000000000001 R15: 000000000000001b FS: 00002aaaaaad4ea0(0000) GS:ffffffff804e7900(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00002aaaaaaac000 CR3: 000000003e764000 CR4: 00000000000006e0 Process ib_cm/1 (pid: 946, threadinfo ffff81001d464000, task ffff81003fdd0f10) Stack: ffff81001ddee230 0000000000000286 0000000100000001 ffff81003f0ebe8c 0000000000000000 ffff81003d3c3158 ffff810037de9af8 ffff81003d3c3158 ffff81003d3c31b0 ffffffff882e2f33 Call Trace:{:ib_sdp:sdp_conn_alloc+35} {:ib_sdp:sdp_cm_req_handler+2175} {cache_free_debugcheck+715} {:ib_sdp:sdp_cm_event_handler+175} {:ib_cm:cm_process_work+50} {:ib_cm:cm_work_handler+1881} {:ib_cm:cm_work_handler+0} {worker_thread+476} {default_wake_function+0} {keventd_create_kthread+0} {worker_thread+0} {keventd_create_kthread+0} {kthread+217} {schedule_tail+64} {child_rip+8} {keventd_create_kthread+0} {kthread+0} {child_rip+0} Code: 0f 0b 40 06 37 80 ff ff ff ff 60 01 65 8b 04 25 34 00 00 00 RIP {sk_alloc+297} RSP Message from syslogd at sins-stinger-10 at Thu Jun 9 11:16:31 2005 ... sins-stinger-10 kernel: invalid operand: 0000 [1] SMP ---- It is very strange because the buffer pool is allocated in init. I don't understand why sk_alloc() would fail, and how it is getting an invalid operand is beyond me. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Thu Jun 9 12:42:11 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 15:42:11 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [TRIVIAL] [kdapl] CM: Message and commentary changes In-Reply-To: <1118343434.4510.15.camel@localhost.localdomain> References: <1118343434.4510.15.camel@localhost.localdomain> Message-ID: Committed in revision 2578. On Thu, 9 Jun 2005, Hal Rosenstock wrote: halr> Message and commentary changes to make consistent with ongoing code halr> changes halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: dapl_openib_cm.c halr> =================================================================== halr> --- dapl_openib_cm.c (revision 2577) halr> +++ dapl_openib_cm.c (working copy) halr> @@ -81,8 +81,7 @@ halr> int status; halr> halr> if (conn->ep->qp == NULL) { halr> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rep_recv: invalid qp " halr> - "handle\n"); halr> + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rep_recv: invalid qp\n"); halr> goto disc; halr> } halr> halr> @@ -416,6 +415,7 @@ halr> * ep halr> * remote_ia_address halr> * remote_conn_qual halr> + * timeout halr> * private_data_size size of private data and structure halr> * private_data pointats_entry->semer to private data structure halr> * halr> @@ -440,7 +440,7 @@ halr> halr> ep_ptr = (struct dapl_ep *)ep; halr> if (ep_ptr->qp == NULL) { halr> - printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); halr> + printk(KERN_ERR "dapl_ib_connect: qp invalid\n"); halr> return DAT_INTERNAL_ERROR; halr> } halr> halr> @@ -550,7 +550,7 @@ halr> * after the final disconnect callback has occurred. halr> * halr> * Input: halr> - * ep_ptr struct dapl_ep halr> + * ep_ptr pointer to dapl_ep struct halr> * active Indicates active side of connection halr> * halr> * Output: halr> @@ -581,7 +581,7 @@ halr> "on transition to error\n", status); halr> } else halr> dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_disconnect_clean: " halr> - "invalid qp handle\n"); halr> + "invalid qp\n"); halr> } halr> halr> /* halr> @@ -590,7 +590,7 @@ halr> * Move the QP to INIT state again. halr> * halr> * Input: halr> - * ep_ptr DAPL_EP halr> + * ep_ptr pointer to dapl_ep struct halr> * halr> * Output: halr> * none halr> @@ -641,8 +641,8 @@ halr> * Have the CM set up a connection listener. halr> * halr> * Input: halr> - * ib_hca_handle HCA handle halr> - * qp_handle QP handle halr> + * ia_ptr pointer to dapl_ia struct halr> + * sp_ptr pointer to dapl_sp struct halr> * halr> * Output: halr> * none halr> @@ -685,8 +685,8 @@ halr> * Have the CM remove a connection listener. halr> * halr> * Input: halr> - * ia IA handle halr> - * ServiceID IB Channel Service ID halr> + * ia_ptr pointer to dapl_ia struct halr> + * sp_ptr pointer to dapl_sp struct halr> * halr> * Output: halr> * none halr> @@ -719,7 +719,7 @@ halr> * Perform necessary steps to reject a connection halr> * halr> * Input: halr> - * cm halr> + * cm pointer to dapl_cm_id struct halr> * halr> * Output: halr> * none halr> @@ -863,7 +863,7 @@ halr> * halr> * Input: halr> * cr halr> - * private data structure handle (only if not using ATS) halr> + * private data structure pointer (only if not using ATS) halr> * halr> * Output: halr> * remote_ia_address: where to place the remote address halr> halr> halr> From sean.hefty at intel.com Thu Jun 9 12:47:03 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 9 Jun 2005 12:47:03 -0700 Subject: [openib-general] [CM] possible problem with crossing DREQs. In-Reply-To: <20050609105017.B28821@topspin.com> Message-ID: > I'm seeing an unusual problem when both halves of a connection >actively disconnect at the same time. Each connection peer issues >a DREQ at the same time, next each receive the DREQ and responds >with a DREP, and finally each connection gets a callback for the >transition to the idle state. However, at this point it appears >that each CM keeps retransmitting DREQ requests, which then seems >to interfere with new connection establishment. I think that I understand what's happening. Receiving the DREQ changed the state of the cm_id, but did not cancel the previous send. I'm actually out on vacation for a little over two weeks (and will be totally away from e-mail after Friday), but something like the patch below might fix the issue. (Note that I didn't test / compile this.) If it does work for you, feel free to commit it. Signed-off-by: Sean Hefty Index: cm.c =================================================================== --- cm.c (revision 2568) +++ cm.c (working copy) @@ -1813,9 +1813,12 @@ static int cm_dreq_handler(struct cm_wor switch (cm_id_priv->id.state) { case IB_CM_REP_SENT: - case IB_CM_MRA_REP_RCVD: - case IB_CM_ESTABLISHED: case IB_CM_DREQ_SENT: + ib_cancel_mad(cm_id_priv->av.port->mad_agent, + (unsigned long) cm_id_priv->msg); + break; + case IB_CM_ESTABLISHED: + case IB_CM_MRA_REP_RCVD: break; case IB_CM_TIMEWAIT: if (cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg)) From damato at psc.edu Thu Jun 9 13:21:09 2005 From: damato at psc.edu (Joe Damato) Date: Thu, 09 Jun 2005 16:21:09 -0400 Subject: [openib-general] opensm Message-ID: <42A8A4B5.1000004@psc.edu> Hello, I recently started to check out infiniband -- unfortunately I am unable to find documentation explaining how to use the different tools. Most notably, I have read lots of things that mention opensm and/or osmtest, but I have been unable to find documentation that explains how to use these tools, and how to test an infiniband setup. Any pointers to where more information can be found would be greatly appreciated. (I have found plenty of lower-level information regarding the architecture and stuff, right now I am looking for more very high level information, just so I can try to test out the small IB set up I have). ifconfig on a test machine shows ib0 and ib1, both with the same ip address although I have no idea what else needs to be done (more setup possibly?) or how to test. My apologies for wasting anyone's time, as this list seems to be mostly development-related and my question does seem somewhat "trivial" compared to the other traffic on this list. Thanks for your time, Joe Damato From jlentini at netapp.com Thu Jun 9 13:48:22 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 9 Jun 2005 16:48:22 -0400 (EDT) Subject: [openib-general] opensm In-Reply-To: <42A8A4B5.1000004@psc.edu> References: <42A8A4B5.1000004@psc.edu> Message-ID: Hi Joe, If you have a switch with an SM, you can use that one instead of opensm. This is the configuration I use, but only because was the simplest way for me to set things up. Others on this list will be able to help you if you want to use OpenSM. A real quick test to see if you can communicate is to use ping. If you built the IB stack as modules, load the IPoIB driver: modprobe ib_mthca modprobe ib_ipoib Regardless of how you compiled the code, ifconfig the ib interfaces and then try a ping. james On Thu, 9 Jun 2005, Joe Damato wrote: > Hello, > I recently started to check out infiniband -- unfortunately I am unable to > find documentation explaining how to use the different tools. Most notably, I > have read lots of things that mention opensm and/or osmtest, but I have been > unable to find documentation that explains how to use these tools, and how to > test an infiniband setup. Any pointers to where more information can be found > would be greatly appreciated. (I have found plenty of lower-level information > regarding the architecture and stuff, right now I am looking for more very > high level information, just so I can try to test out the small IB set up I > have). ifconfig on a test machine shows ib0 and ib1, both with the same ip > address although I have no idea what else needs to be done (more setup > possibly?) or how to test. My apologies for wasting anyone's time, as this > list seems to be mostly development-related and my question does seem > somewhat "trivial" compared to the other traffic on this list. > > > Thanks for your time, > Joe Damato > > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From halr at voltaire.com Thu Jun 9 13:43:46 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 Jun 2005 16:43:46 -0400 Subject: [openib-general] opensm In-Reply-To: <42A8A4B5.1000004@psc.edu> References: <42A8A4B5.1000004@psc.edu> Message-ID: <1118349825.4510.57.camel@localhost.localdomain> On Thu, 2005-06-09 at 16:21, Joe Damato wrote: > Hello, > I recently started to check out infiniband -- unfortunately I am > unable to find documentation explaining how to use the different tools. > Most notably, I have read lots of things that mention opensm and/or > osmtest, I'm curious as to what mentions osmtest. Are you talking about OpenIB (gen2) or gen1 ? > but I have been unable to find documentation that explains how > to use these tools, The documentation is in userspace/management. There is a README and a user manual and release notes under that in osm/doc. > and how to test an infiniband setup. Not sure what you mean by test an infiniband setup. > Any pointers to > where more information can be found would be greatly appreciated. (I > have found plenty of lower-level information regarding the architecture > and stuff, right now I am looking for more very high level information, > just so I can try to test out the small IB set up I have). Is it back to back HCAs ? > ifconfig on a > test machine shows ib0 and ib1, both with the same ip address Were these manually configured IP addresses ? Are you just looking to run IPoIB ? > although I > have no idea what else needs to be done (more setup possibly?) or how to > test. My apologies for wasting anyone's time, as this list seems to be > mostly development-related and my question does seem somewhat "trivial" > compared to the other traffic on this list. To bring up an IB subnet, you need an SM to bring the ports to active. Then you can start to talk some ULPs. There is info on bringing up IPoIB in linux-kernel/docs. There are ipoib.txt and ipoib_faq.txt. There is also a wiki which was started: https://openib.org/tiki/tiki-index.php?page=MellanoxHcaFirmware Hope this is enough to get you rolling... -- Hal From caitlin.bestler at gmail.com Thu Jun 9 13:58:20 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 9 Jun 2005 13:58:20 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <1118330531.21255.6.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> Message-ID: <469958e0050609135862ace6fc@mail.gmail.com> The reason the method table is pointed at in each object is to avoid having to run long chains of memory references to get the method table. If you have the ep pointer you should be able to get the method table directly, not have to fetch the ia then the hca then the method table. And as you point out it is a *pointer* that is being duplicated (32 or 64 bits) not the method table itself. A DAT provider *could* theoretically use different method tables for different objects but I haven't seen anyone actually do that yet. On 6/9/05, Tom Duffy wrote: > On Thu, 2005-06-09 at 11:47 +0200, Christoph Hellwig wrote: > > It is. B?ut I think it'll change even more later on - having a pointer > > to the method table in every object is an enormous waste of space, in > > the end the method should probably be only in a hca-level object, and > > all other objects should have public pointers to it. > > Can you please explain this a bit more? A 32 or 64 bit pointer is a > waste of space? Or are you saying that each provider should just define > a struct and have a global pointer to it? > > Thanks, > > -tduffy > > > BodyID:15801799.2.n.logpart (stored separately) > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From tduffy at sun.com Thu Jun 9 14:15:35 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 09 Jun 2005 14:15:35 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> Message-ID: <1118351735.20343.37.camel@duffman> On Thu, 2005-06-09 at 14:27 -0400, James Lentini wrote: > Tom, > > I think this change has a lot of merits, but I would prefer to defer > this sort of change until we have a more stable provider. Once we have > removed the bugs and finished our update of the provider code > (removing the custom DAPL data structures, etc.), then I think we > should revisit this topic. > > Does that sound reasonable? Ok, I can wait. I definitely understand the need to stabilize the provider and not introduce any new bugs. I will start playing around again with getting rid of dapl's own linked lists. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From xma at us.ibm.com Thu Jun 9 16:14:17 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 9 Jun 2005 16:14:17 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. Message-ID: I used cyclets_to_units() get Bandwidth peak (#0 to #972): 1852.72 MByte/sec Bandwidth average: 1852.67 MByte/sec Which is far away from the 1X (2.5Gbit/s) throughtput. This per secs cycles value is incorrect. Either change the measurment to gettimeofday() to measure MB/s or measure the how many cycles per sec by doing something like that: #define MEASURE_TIME 16 get_cycles(tbegin); sleep(MEASURE_TIME); get_cycles(tend) cycles_to_units = (tend-tbegin)/MEASURE_TIME to caculate the right MB/s throughput. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Thu Jun 9 16:40:10 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 09 Jun 2005 16:40:10 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: (Shirley Ma's message of "Thu, 9 Jun 2005 16:14:17 -0700") References: Message-ID: <52psuv14md.fsf@topspin.com> Shirley> I used cyclets_to_units() get Bandwidth peak (#0 to Shirley> #972): 1852.72 MByte/sec Shirley> Which is far away from the 1X (2.5Gbit/s) throughtput. Actually it's not that far away -- the 2.5 Gbit/sec is after 8b/10b coding, so the raw data rate is really 2 Gbit/sec. Getting > 92% link utilization is pretty good considering packet header overhead and so on. BTW I'm assuming "MByte/sec" is a typo for "Mbit/sec". - R. From xma at us.ibm.com Thu Jun 9 16:46:18 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 9 Jun 2005 16:46:18 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <52psuv14md.fsf@topspin.com> Message-ID: > BTW I'm assuming "MByte/sec" is a typo for "Mbit/sec". No, that's the reason I said wrong. Using gettimeofday(), the throughtput is around 230MB/sec. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Thu Jun 9 16:48:45 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 09 Jun 2005 16:48:45 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: (Shirley Ma's message of "Thu, 9 Jun 2005 16:46:18 -0700") References: Message-ID: <52k6l31482.fsf@topspin.com> Roland> BTW I'm assuming "MByte/sec" is a typo for "Mbit/sec". Shirley> No, that's the reason I said wrong. Shirley> Using gettimeofday(), the throughtput is around Shirley> 230MB/sec. Well, 230 MB/sec = 230 * 8 Mbit/sec = 1840 Mbit/sec, and 1840 is pretty close to the 1852.67 number in your original mail. So are you sure MByte/sec is not just a typo? - R. From xma at us.ibm.com Thu Jun 9 16:55:37 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 9 Jun 2005 16:55:37 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <52k6l31482.fsf@topspin.com> Message-ID: > So are you sure MByte/sec is not just a typo? Yes. It's easy to test it. I tested on both intel and PPC platforms. The get_cycles() doesn't return the value as we expect. Need to read the architecture to find the register definitions. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From iod00d at hp.com Thu Jun 9 17:33:52 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 9 Jun 2005 17:33:52 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: References: Message-ID: <20050610003352.GE12434@esmail.cup.hp.com> On Thu, Jun 09, 2005 at 04:14:17PM -0700, Shirley Ma wrote: > I used cyclets_to_units() get cycles_to_units is a FP value, not a function. get_cpu_mhz() is probably what you meant. What does "cat /proc/cpuinfo" say on the boxes you see this on? > Bandwidth peak (#0 to #972): 1852.72 MByte/sec > Bandwidth average: 1852.67 MByte/sec Does rdma_lat have the same issue? What does rdma_lat report? > This per secs cycles value is incorrect. Yes, but I'd like to know what /proc/cpuinfo reports since that's what it's based on. > Either change the measurment to > gettimeofday() to measure MB/s or > measure the how many cycles per sec by doing something like that: > > #define MEASURE_TIME 16 > > get_cycles(tbegin); > sleep(MEASURE_TIME); > get_cycles(tend) > cycles_to_units = (tend-tbegin)/MEASURE_TIME > > to caculate the right MB/s throughput. I think adding such a calibration to get_cpu_mhz() so we can warn if /proc/cpuinfo data doesn't agree with gettimeofday() and get_cycles(). Or use gettimeofday() for outside major loops and see how well that correlates to the sum of the get_cycles() calls. thanks, grant From xma at us.ibm.com Thu Jun 9 20:52:10 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 9 Jun 2005 20:52:10 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050610003352.GE12434@esmail.cup.hp.com> Message-ID: > Does rdma_lat have the same issue? > What does rdma_lat report? Latency typical: 0.717812 usec Latency best : 0.705625 usec Latency typical: 1149 cycles Latency best : 1129.5 cycles after modifying the per usec cycles the results are: Latency typical: 5.76884 usec Latency best : 5.66834 usec gettimeofday() results are: Latency typical: 6.000000 usec Latency best : 5.500000 usec Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From iod00d at hp.com Thu Jun 9 22:06:39 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 9 Jun 2005 22:06:39 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: References: <20050610003352.GE12434@esmail.cup.hp.com> Message-ID: <20050610050639.GB2833@esmail.cup.hp.com> On Thu, Jun 09, 2005 at 08:52:10PM -0700, Shirley Ma wrote: > > Does rdma_lat have the same issue? > > What does rdma_lat report? > > Latency typical: 0.717812 usec > Latency best : 0.705625 usec > > Latency typical: 1149 cycles > Latency best : 1129.5 cycles > I'm convinced something is broken and that gettimeofday() works better in your case. Please provide "cat /proc/cpuinfo" output so I can rule out broken firmware and look for something else. > after modifying the per usec cycles > the results are: > > Latency typical: 5.76884 usec > Latency best : 5.66834 usec > > gettimeofday() results are: > > Latency typical: 6.000000 usec > Latency best : 5.500000 usec These result suggest gettimeofday() is not very accurate and less reliable. Note that I currently have patches outstanding on both rdma_bw.c and rdma_lat.c. I'm waiting for Michael to reject, accept, or comment on them. Or can someone else commit changes to those files for me? thanks, grant From itamar at mellanox.co.il Thu Jun 9 22:33:44 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Fri, 10 Jun 2005 08:33:44 +0300 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() Message-ID: <91DB792C7985D411BEC300B40080D29CC35C74@mtvex01.mtv.mtl.com> >> Does that sound reasonable? >Ok, I can wait. I definitely understand the need to stabilize the >provider and not introduce any new bugs. > >I will start playing around again with getting rid of dapl's own linked >lists. > >-tduffy Hi All, James, can you close the issue of merging files into one file ? it is hard to create a big patch when you know that the file you are working on is going to be deleted or merged with other file. I think we only left ia and cr I think that sp+psp+rsp should be one file dapl_sp.c Tom, dapl has many lists and I think that most of them should be kernel lists. I think that the following list : in struct dapl_sp : struct dapl_llist_entry *cr_list_head; /* CR pending queue */ should be replaced with kernel rb tree because we search in this list in the hot path of the connect-disconnect flow. we have agreed in the past that evd ring buffer should be replaced also with kernel lists. I will prepare a patch that will remove dapl_hash (no need for this hash). Itamar From xma at us.ibm.com Thu Jun 9 22:13:13 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 9 Jun 2005 22:13:13 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050610003352.GE12434@esmail.cup.hp.com> Message-ID: >I think adding such a calibration to get_cpu_mhz() >so we can warn if /proc/cpuinfo data doesn't agree with >gettimeofday() and get_cycles(). using below to calculate the cycles_to_unit is more accurate. clock_gettime(); get_cycles(); sleep(5); clock_gettime(); get_cycles(); cycles_to_unit = offset - cycles/ offset - time; Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Thu Jun 9 22:17:33 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 9 Jun 2005 22:17:33 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050610050639.GB2833@esmail.cup.hp.com> Message-ID: > Please provide "cat /proc/cpuinfo" output so I can rule out > broken firmware and look for something else. processor : 0 cpu : PPC970, altivec supported clock : 1600.000000MHz revision : 2.2 processor : 1 cpu : PPC970, altivec supported clock : 1600.000000MHz revision : 2.2 timebase : 199836932 machine : CHRP IBM,8842-21X I modified the get_cpu_mhz() to for PPC. rc = sscanf(buf, "clock : %lf", &m); Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sinate at yahoo.com Fri Jun 10 01:11:32 2005 From: sinate at yahoo.com (Steven Wooding) Date: Fri, 10 Jun 2005 09:11:32 +0100 (BST) Subject: [openib-general] Portability of AIO Message-ID: <20050610081132.95763.qmail@web32511.mail.mud.yahoo.com> Hi, Does anybody know of any plans to make the API of libaio (Native Linux AIO) portable (ie make it the standard AIO API)? Thanks, Steve. --------------------------------- Yahoo! Messenger NEW - crystal clear PC to PCcalling worldwide with voicemail -------------- next part -------------- An HTML attachment was scrubbed... URL: From sinate at yahoo.com Fri Jun 10 01:13:34 2005 From: sinate at yahoo.com (Steven Wooding) Date: Fri, 10 Jun 2005 09:13:34 +0100 (BST) Subject: [openib-general] uverbs performance; ibv_pingpong poll vs sleep In-Reply-To: <527jh4erzs.fsf@topspin.com> Message-ID: <20050610081334.96357.qmail@web32511.mail.mud.yahoo.com> Thanks. I'll give pipeling ago in my app. Cheers, Steve. Roland Dreier wrote: Steven> I have compaired the data rates using ibv_pingpong with Steven> and without the -e option. Therefore, using polling and Steven> waiting the CQ events (sleeping). Steven> Is there any way to trade off the data rate with the CPU Steven> usage (I was thinking of some timeout from polling). I suppose you could have some sort of adaptive polling scheme that spins polling for a while and then sleeps waiting for an event. However, as Michael said, it's probably better to use pipelining and post multiple send work requests. This hides the latency of getting a completion event by keeping the HCA busy. - R. --------------------------------- How much free photo storage do you get? Store your holiday snaps for FREE with Yahoo! Photos. Get Yahoo! Photos -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Fri Jun 10 06:04:55 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 Jun 2005 09:04:55 -0400 Subject: [openib-general] [CM] possible problem with crossing DREQs. In-Reply-To: References: Message-ID: <1118408695.4477.10.camel@localhost.localdomain> On Thu, 2005-06-09 at 15:47, Sean Hefty wrote: > > I'm seeing an unusual problem when both halves of a connection > >actively disconnect at the same time. Each connection peer issues > >a DREQ at the same time, next each receive the DREQ and responds > >with a DREP, and finally each connection gets a callback for the > >transition to the idle state. However, at this point it appears > >that each CM keeps retransmitting DREQ requests, which then seems > >to interfere with new connection establishment. > > I think that I understand what's happening. Receiving the DREQ > changed the state of the cm_id, but did not cancel the previous send. > > I'm actually out on vacation for a little over two weeks (and will > be totally away from e-mail after Friday), but something > like the patch below might fix the issue. (Note that I didn't test / > compile this.) If it does work for you, feel free to commit it. This works for me. My test case is a little different. It is repeated loopback kdapl quit tests but it does resolve the same problem. I am comitting this change. Thanks. -- Hal From halr at voltaire.com Fri Jun 10 06:18:24 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 Jun 2005 09:18:24 -0400 Subject: [openib-general] OpenSM: Support for LinkSpeedActive (LSA) added Message-ID: <1118409504.4477.17.camel@localhost.localdomain> I just added the support for LinkSpeedActive(LSA) for DDR/QDR links into OpenSM. Diagnostics (smpquery portinfo) already supports this. -- Hal From halr at voltaire.com Fri Jun 10 06:31:31 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 Jun 2005 09:31:31 -0400 Subject: [openib-general] A couple of kdapltest oopses Message-ID: <1118410291.4476.4.camel@localhost.localdomain> Hi, First on the client when running transaction test, I see the following: kdapltest -T T -s -D mthca0a -d -i 10000 -w 8 client SR server SR Jun 10 08:58:17 localhost kernel: DT_Mdep_Thread_: page allocation failure. order:0, mode:0x20 Jun 10 08:58:17 localhost kernel: [] __alloc_pages+0x2b2/0x440 Jun 10 08:58:17 localhost kernel: [] kernel_map_pages+0x28/0x70 Jun 10 08:58:17 localhost kernel: [] kmem_getpages+0x31/0xb0 Jun 10 08:58:17 localhost kernel: [] cache_grow+0x139/0x360 Jun 10 08:58:17 localhost kernel: [] cache_alloc_refill+0x153/0x340 Jun 10 08:58:17 localhost kernel: [] dbg_redzone1+0x15/0x30 Jun 10 08:58:17 localhost kernel: [] cache_alloc_debugcheck_after+0x6e/0x1a0 Jun 10 08:58:17 localhost kernel: [] __kmalloc+0xb1/0xe0 Jun 10 08:58:17 localhost kernel: [] DT_Mdep_Malloc+0x25/0x60 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Tdep_PT_Printf+0x16/0x1b0 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Transaction_Run+0x48f/0xb60 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Mdep_wait_object_wakeup+0x1d/0x30 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Transaction_Main+0x1383/0x21a0 [kdapltest] Jun 10 08:58:17 localhost kernel: [] kernel_map_pages+0x28/0x70 Jun 10 08:58:17 localhost kernel: [] cache_free_debugcheck+0x196/0x2d0 Jun 10 08:58:17 localhost kernel: [] DT_Mdep_Thread_Start_Routine+0x1f/0x30 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Mdep_Thread_Start_Routine+0x0/0x30 [kdapltest] Jun 10 08:58:17 localhost kernel: [] kernel_thread_helper+0x5/0x10 Jun 10 08:58:17 localhost kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000004 Jun 10 08:58:17 localhost kernel: printing eip: Jun 10 08:58:17 localhost kernel: c022048b Jun 10 08:58:17 localhost kernel: *pde = 07cb6067 Jun 10 08:58:17 localhost kernel: *pte = 00000000 Jun 10 08:58:17 localhost kernel: Oops: 0002 [#1] Jun 10 08:58:17 localhost kernel: DEBUG_PAGEALLOC Jun 10 08:58:17 localhost kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at dat ib_ipoib ib_sa ib_umad ide_cd cdrom lp ipv6 autofs parport_pc parport uhci_hcd ehci_hcd ib_mthca ib_mad ib_core ohci_hcd eepro100 mii evdev usbcore Jun 10 08:58:17 localhost kernel: CPU: 0 Jun 10 08:58:17 localhost kernel: EIP: 0060:[] Not tainted VLI Jun 10 08:58:17 localhost kernel: EFLAGS: 00010283 (2.6.11.6) Jun 10 08:58:17 localhost kernel: EIP is at vsnprintf+0x4b/0x4c0 Jun 10 08:58:17 localhost kernel: eax: 00000054 ebx: c1d17f78 ecx: 00000000 edx: d0a35eef Jun 10 08:58:17 localhost kernel: esi: 00000004 edi: c1d17f78 ebp: 00000103 esp: ce5bfd84 Jun 10 08:58:17 localhost kernel: ds: 007b es: 007b ss: 0068 Jun 10 08:58:17 localhost kernel: Process DT_Mdep_Thread_ (pid: 6696, threadinfo=ce5be000 task=c3730a90) Jun 10 08:58:17 localhost kernel: Stack: cffff740 00000020 00000000 d0a33995 00000104 00000000 c1d17f78 d0a33995 Jun 10 08:58:17 localhost kernel: 00000104 00000020 c1d17f78 00000000 c1d17f78 c5b58000 d0a3484b 00000004 Jun 10 08:58:17 localhost kernel: 00000100 d0a35eef ce5bfdf4 0000007b ffffff05 c1d17f78 cf34ef78 c60d2060 Jun 10 08:58:17 localhost kernel: Call Trace: Jun 10 08:58:17 localhost kernel: [] DT_Mdep_Malloc+0x25/0x60 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Mdep_Malloc+0x25/0x60 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Tdep_PT_Printf+0x3b/0x1b0 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Transaction_Run+0x48f/0xb60 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Mdep_wait_object_wakeup+0x1d/0x30 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Transaction_Main+0x1383/0x21a0 [kdapltest] Jun 10 08:58:17 localhost kernel: [] kernel_map_pages+0x28/0x70 Jun 10 08:58:17 localhost kernel: [] cache_free_debugcheck+0x196/0x2d0 Jun 10 08:58:17 localhost kernel: [] DT_Mdep_Thread_Start_Routine+0x1f/0x30 [kdapltest] Jun 10 08:58:17 localhost kernel: [] DT_Mdep_Thread_Start_Routine+0x0/0x30 [kdapltest] Jun 10 08:58:17 localhost kernel: [] kernel_thread_helper+0x5/0x10 Jun 10 08:58:17 localhost kernel: Code: f0 48 39 c5 73 0d 89 f2 f7 da bd ff ff ff ff 89 54 24 40 8b 54 24 44 80 3a 00 74 23 8d 74 26 00 0f b6 02 3c 25 74 3d 39 ee 77 06 <88> 06 8b 54 24 44 46 89 d0 42 89 54 24 44 80 78 01 00 75 e1 39 then on the server side, when I try to rmmod kdapltest, I get: Unable to handle kernel paging request at ffffffff88243a05 RIP: [] PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 Oops: 0010 [1] SMP CPU 1 Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT ipt_state ip_conntrack iptable_filter ip_tables video button battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod Pid: 9094, comm: kdapltest Not tainted 2.6.11.6 RIP: 0010:[] [] RSP: 0018:ffff810037787e58 EFLAGS: 00010292 RAX: 0000000000000061 RBX: ffff810035af6d88 RCX: 0000000000000282 RDX: ffff810020712308 RSI: 0000000000000282 RDI: ffff810020711ce0 RBP: ffff810035af6d80 R08: ffff810037786000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: ffff810038daccc0 R13: 0000000000000001 R14: 0000000000000000 R15: ffff8100234ad008 FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9f00(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffff88243a05 CR3: 000000003a33a000 CR4: 00000000000006e0 Process kdapltest (pid: 9094, threadinfo ffff810037786000, task ffff81003cd16860) Stack: ffff810000002000 0000024000000003 ffff810000000100 ffff810000000000 ffff810000000000 0000000000000000 ffff81003d40f4c0 0000000100000000 ffff81001eb44000 ffff8100234ad000 Call Trace:{child_rip+8} {child_rip+0} Code: Bad RIP value. RIP [] RSP CR2: ffffffff88243a05 <1>Unable to handle kernel paging request at ffffffff8825eac8 RIP: {filp_close+73} PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 Oops: 0000 [2] SMP CPU 1 Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT ipt_state ip_conntrack iptable_filter ip_tables video button battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod Pid: 9094, comm: kdapltest Not tainted 2.6.11.6 RIP: 0010:[] {filp_close+73} RSP: 0018:ffff810037787bf8 EFLAGS: 00010286 RAX: ffffffff8825ea60 RBX: ffff81001f24a8c0 RCX: ffff810019109238 RDX: ffff810037e58e70 RSI: ffff810037e58d40 RDI: ffff81001f24a8c0 RBP: 0000000000000000 R08: ffff81003d7ce6c0 R09: ffff810037787bf8 R10: 0000000000000001 R11: 0000000000000000 R12: ffff810037e58d40 R13: 0000000000000001 R14: 0000000000000001 R15: ffff81001f278940 FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9f00(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffff8825eac8 CR3: 000000003a33a000 CR4: 00000000000006e0 Process kdapltest (pid: 9094, threadinfo ffff810037786000, task ffff81003cd16860) Stack: 0000000000000003 0000000000000003 ffff810037e58d40 ffffffff80139723 0000000000000009 ffff810037e58d40 ffff81003cd16eb0 ffff81003cd16860 0000000000000009 ffffffff80139eba Call Trace:{put_files_struct+115} {do_exit+378} {do_unblank_screen+119} {do_page_fault+2047} {thread_return+42} {error_exit+0} {child_rip+8} {child_rip+0} Code: 48 8b 40 68 48 85 c0 74 0e 48 89 df ff d0 85 ed 0f 44 e8 66 RIP {filp_close+73} RSP CR2: ffffffff8825eac8 <1>Unable to handle kernel paging request at ffffffff88245190 RIP: [] PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 Oops: 0010 [3] SMP CPU 0 Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT ipt_state ip_conntrack iptable_filter ip_tables video button battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod Pid: 9119, comm: DT_Mdep_Thread_ Not tainted 2.6.11.6 RIP: 0010:[] [] RSP: 0018:ffff810031907f18 EFLAGS: 00010296 RAX: 0000000000000000 RBX: ffff81001b770000 RCX: 0000000000000006 RDX: 0000000000000008 RSI: 0000000000000008 RDI: 0000000000000003 RBP: ffff81001b77004c R08: ffffffff80508a00 R09: 0000000000000008 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001 R13: ffff810038daccc0 R14: 0000000000000000 R15: ffff8100234ad008 FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9e80(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffff88245190 CR3: 000000003d8b3000 CR4: 00000000000006e0 Process DT_Mdep_Thread_ (pid: 9119, threadinfo ffff810031906000, task ffff81003cd16170) Stack: ffff810037bde1c0 ffff810035af6d80 ffff810038daccc0 0000000000000001 ffff81001b770000 ffffffff88250ff8 ffff810037bde1c0 ffffffff8010f1c7 ffff8100234ad008 ffff81001b770000 Call Trace:{child_rip+8} {flat_send_IPI_mask+0} {child_rip+0} Code: Bad RIP value. RIP [] RSP CR2: ffffffff88245190 -- Hal From halr at voltaire.com Fri Jun 10 06:51:53 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 Jun 2005 09:51:53 -0400 Subject: [openib-general] IPoIB: Waiting for ib0 to become free Message-ID: <1118411513.4476.20.camel@localhost.localdomain> Hi Roland and Troy, Last week, Troy reported the following: On Fri, 2005-06-03 at 16:33, Troy Benjegerdes wrote: > > > Also, I have two machines in a state right now where they are > > > printing out: > > > > > > kernel: unregister_netdevice: waiting for ib0 to become free. > > > Usage count = 1 dev/core.c netdev_wait_allrefs says: * Any protocol or device that holds a reference should register * for netdevice notification, and cleanup and put back the * reference if they receive an UNREGISTER event. Is it correct that IPoIB does not need to register for these events ? If it is, then this must be something else which is using the IPoIB driver causing the reference count to be incremented but not handling these events. Troy, any idea on how to recreate this ? Thanks. -- Hal From jlentini at netapp.com Fri Jun 10 07:36:08 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 10 Jun 2005 10:36:08 -0400 (EDT) Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C74@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C74@mtvex01.mtv.mtl.com> Message-ID: On Fri, 10 Jun 2005, Itamar Rabenstein wrote: > >>> Does that sound reasonable? > >> Ok, I can wait. I definitely understand the need to stabilize the >> provider and not introduce any new bugs. >> >> I will start playing around again with getting rid of dapl's own linked >> lists. >> >> -tduffy > > Hi All, > > James, > can you close the issue of merging files into one file ? > it is hard to create a big patch when you know that the file you are working > on > is going to be deleted or merged with other file. > I think we only left ia and cr Don't let that block you. If you create a patch and I merge the files, I'll fix your patch so it applies. > I think that sp+psp+rsp should be one file dapl_sp.c > > Tom, > dapl has many lists and I think that most of them should be kernel lists. > I think that the following list : > in struct dapl_sp : struct dapl_llist_entry *cr_list_head; /* CR > pending queue */ > should be replaced with kernel rb tree because > we search in this list in the hot path of the connect-disconnect flow. Do we know that this is a performance bottleneck? > we have agreed in the past that evd ring buffer should be replaced also with > kernel lists. If you can replace the ring buffer with a data structure provided by Linux, that is a win. Maintaining the locking guarantees will be the hard part. > I will prepare a patch that will remove dapl_hash (no need for this hash). > > Itamar > > From itamar at mellanox.co.il Fri Jun 10 08:29:41 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Fri, 10 Jun 2005 18:29:41 +0300 Subject: [openib-general] A couple of kdapltest oopses Message-ID: <91DB792C7985D411BEC300B40080D29CC35C75@mtvex01.mtv.mtl.com> is this new ? did you saw it before ? do you see the problem with -i 1000 ? both oops are in the malloc stage. I have seen in the past problems were dapl tried to malloc a very big memory. Itamar > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Friday, June 10, 2005 4:32 PM > To: James Lentini > Cc: openib-general at openib.org > Subject: [openib-general] A couple of kdapltest oopses > > > Hi, > > First on the client when running transaction test, I see the > following: > kdapltest -T T -s -D mthca0a -d -i 10000 -w > 8 client SR server SR > > Jun 10 08:58:17 localhost kernel: DT_Mdep_Thread_: page > allocation failure. order:0, mode:0x20 > Jun 10 08:58:17 localhost kernel: [] > __alloc_pages+0x2b2/0x440 > Jun 10 08:58:17 localhost kernel: [] > kernel_map_pages+0x28/0x70 > Jun 10 08:58:17 localhost kernel: [] > kmem_getpages+0x31/0xb0 > Jun 10 08:58:17 localhost kernel: [] cache_grow+0x139/0x360 > Jun 10 08:58:17 localhost kernel: [] > cache_alloc_refill+0x153/0x340 > Jun 10 08:58:17 localhost kernel: [] dbg_redzone1+0x15/0x30 > Jun 10 08:58:17 localhost kernel: [] > cache_alloc_debugcheck_after+0x6e/0x1a0 > Jun 10 08:58:17 localhost kernel: [] __kmalloc+0xb1/0xe0 > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_Malloc+0x25/0x60 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Tdep_PT_Printf+0x16/0x1b0 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Transaction_Run+0x48f/0xb60 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_wait_object_wakeup+0x1d/0x30 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Transaction_Main+0x1383/0x21a0 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > kernel_map_pages+0x28/0x70 > Jun 10 08:58:17 localhost kernel: [] > cache_free_debugcheck+0x196/0x2d0 > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_Thread_Start_Routine+0x1f/0x30 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_Thread_Start_Routine+0x0/0x30 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > kernel_thread_helper+0x5/0x10 > Jun 10 08:58:17 localhost kernel: Unable to handle kernel > NULL pointer dereference at virtual address 00000004 > Jun 10 08:58:17 localhost kernel: printing eip: > Jun 10 08:58:17 localhost kernel: c022048b > Jun 10 08:58:17 localhost kernel: *pde = 07cb6067 > Jun 10 08:58:17 localhost kernel: *pte = 00000000 > Jun 10 08:58:17 localhost kernel: Oops: 0002 [#1] > Jun 10 08:58:17 localhost kernel: DEBUG_PAGEALLOC > Jun 10 08:58:17 localhost kernel: Modules linked in: > kdapltest ib_dat_provider ib_cm ib_at dat ib_ipoib ib_sa > ib_umad ide_cd cdrom lp ipv6 autofs parport_pc parport > uhci_hcd ehci_hcd ib_mthca ib_mad ib_core ohci_hcd eepro100 > mii evdev usbcore > Jun 10 08:58:17 localhost kernel: CPU: 0 > Jun 10 08:58:17 localhost kernel: EIP: 0060:[] > Not tainted VLI > Jun 10 08:58:17 localhost kernel: EFLAGS: 00010283 (2.6.11.6) > Jun 10 08:58:17 localhost kernel: EIP is at vsnprintf+0x4b/0x4c0 > Jun 10 08:58:17 localhost kernel: eax: 00000054 ebx: > c1d17f78 ecx: 00000000 edx: d0a35eef > Jun 10 08:58:17 localhost kernel: esi: 00000004 edi: > c1d17f78 ebp: 00000103 esp: ce5bfd84 > Jun 10 08:58:17 localhost kernel: ds: 007b es: 007b ss: 0068 > Jun 10 08:58:17 localhost kernel: Process DT_Mdep_Thread_ > (pid: 6696, threadinfo=ce5be000 task=c3730a90) > Jun 10 08:58:17 localhost kernel: Stack: cffff740 00000020 > 00000000 d0a33995 00000104 00000000 c1d17f78 d0a33995 > Jun 10 08:58:17 localhost kernel: 00000104 00000020 > c1d17f78 00000000 c1d17f78 c5b58000 d0a3484b 00000004 > Jun 10 08:58:17 localhost kernel: 00000100 d0a35eef > ce5bfdf4 0000007b ffffff05 c1d17f78 cf34ef78 c60d2060 > Jun 10 08:58:17 localhost kernel: Call Trace: > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_Malloc+0x25/0x60 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_Malloc+0x25/0x60 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Tdep_PT_Printf+0x3b/0x1b0 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Transaction_Run+0x48f/0xb60 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_wait_object_wakeup+0x1d/0x30 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Transaction_Main+0x1383/0x21a0 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > kernel_map_pages+0x28/0x70 > Jun 10 08:58:17 localhost kernel: [] > cache_free_debugcheck+0x196/0x2d0 > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_Thread_Start_Routine+0x1f/0x30 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > DT_Mdep_Thread_Start_Routine+0x0/0x30 [kdapltest] > Jun 10 08:58:17 localhost kernel: [] > kernel_thread_helper+0x5/0x10 > Jun 10 08:58:17 localhost kernel: Code: f0 48 39 c5 73 0d 89 > f2 f7 da bd ff ff ff ff 89 54 24 40 8b 54 24 44 80 3a 00 74 > 23 8d 74 26 00 0f b6 02 3c 25 74 3d 39 ee 77 06 <88> 06 8b 54 > 24 44 46 89 d0 42 89 54 24 44 80 78 01 00 75 e1 39 > > then on the server side, when I try to rmmod kdapltest, I get: > > Unable to handle kernel paging request at ffffffff88243a05 RIP: > [] > PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 > Oops: 0010 [1] SMP > CPU 1 > Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib > ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT > ipt_state ip_conntrack iptable_filter ip_tables video button > battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random > ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot > dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod > Pid: 9094, comm: kdapltest Not tainted 2.6.11.6 > RIP: 0010:[] [] > RSP: 0018:ffff810037787e58 EFLAGS: 00010292 > RAX: 0000000000000061 RBX: ffff810035af6d88 RCX: 0000000000000282 > RDX: ffff810020712308 RSI: 0000000000000282 RDI: ffff810020711ce0 > RBP: ffff810035af6d80 R08: ffff810037786000 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000246 R12: ffff810038daccc0 > R13: 0000000000000001 R14: 0000000000000000 R15: ffff8100234ad008 > FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9f00(0000) > knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: ffffffff88243a05 CR3: 000000003a33a000 CR4: 00000000000006e0 > Process kdapltest (pid: 9094, threadinfo ffff810037786000, > task ffff81003cd16860) > Stack: ffff810000002000 0000024000000003 ffff810000000100 > ffff810000000000 > ffff810000000000 0000000000000000 ffff81003d40f4c0 > 0000000100000000 > ffff81001eb44000 ffff8100234ad000 > Call Trace:{child_rip+8} > {child_rip+0} > > > Code: Bad RIP value. > RIP [] RSP > CR2: ffffffff88243a05 > <1>Unable to handle kernel paging request at ffffffff8825eac8 RIP: > {filp_close+73} > PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 > Oops: 0000 [2] SMP > CPU 1 > Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib > ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT > ipt_state ip_conntrack iptable_filter ip_tables video button > battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random > ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot > dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod > Pid: 9094, comm: kdapltest Not tainted 2.6.11.6 > RIP: 0010:[] {filp_close+73} > RSP: 0018:ffff810037787bf8 EFLAGS: 00010286 > RAX: ffffffff8825ea60 RBX: ffff81001f24a8c0 RCX: ffff810019109238 > RDX: ffff810037e58e70 RSI: ffff810037e58d40 RDI: ffff81001f24a8c0 > RBP: 0000000000000000 R08: ffff81003d7ce6c0 R09: ffff810037787bf8 > R10: 0000000000000001 R11: 0000000000000000 R12: ffff810037e58d40 > R13: 0000000000000001 R14: 0000000000000001 R15: ffff81001f278940 > FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9f00(0000) > knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: ffffffff8825eac8 CR3: 000000003a33a000 CR4: 00000000000006e0 > Process kdapltest (pid: 9094, threadinfo ffff810037786000, > task ffff81003cd16860) > Stack: 0000000000000003 0000000000000003 ffff810037e58d40 > ffffffff80139723 > 0000000000000009 ffff810037e58d40 ffff81003cd16eb0 > ffff81003cd16860 > 0000000000000009 ffffffff80139eba > Call Trace:{put_files_struct+115} > {do_exit+378} > {do_unblank_screen+119} > {do_page_fault+2047} > {thread_return+42} > {error_exit+0} > {child_rip+8} > {child_rip+0} > > > Code: 48 8b 40 68 48 85 c0 74 0e 48 89 df ff d0 85 ed 0f 44 e8 66 > RIP {filp_close+73} RSP > CR2: ffffffff8825eac8 > <1>Unable to handle kernel paging request at ffffffff88245190 RIP: > [] > PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 > Oops: 0010 [3] SMP > CPU 0 > Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib > ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT > ipt_state ip_conntrack iptable_filter ip_tables video button > battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random > ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot > dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod > Pid: 9119, comm: DT_Mdep_Thread_ Not tainted 2.6.11.6 > RIP: 0010:[] [] > RSP: 0018:ffff810031907f18 EFLAGS: 00010296 > RAX: 0000000000000000 RBX: ffff81001b770000 RCX: 0000000000000006 > RDX: 0000000000000008 RSI: 0000000000000008 RDI: 0000000000000003 > RBP: ffff81001b77004c R08: ffffffff80508a00 R09: 0000000000000008 > R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001 > R13: ffff810038daccc0 R14: 0000000000000000 R15: ffff8100234ad008 > FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9e80(0000) > knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: ffffffff88245190 CR3: 000000003d8b3000 CR4: 00000000000006e0 > Process DT_Mdep_Thread_ (pid: 9119, threadinfo > ffff810031906000, task ffff81003cd16170) > Stack: ffff810037bde1c0 ffff810035af6d80 ffff810038daccc0 > 0000000000000001 > ffff81001b770000 ffffffff88250ff8 ffff810037bde1c0 > ffffffff8010f1c7 > ffff8100234ad008 ffff81001b770000 > Call Trace:{child_rip+8} > {flat_send_IPI_mask+0} > {child_rip+0} > > Code: Bad RIP value. > RIP [] RSP > CR2: ffffffff88245190 > > -- Hal > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From iod00d at hp.com Fri Jun 10 08:24:40 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 10 Jun 2005 08:24:40 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: References: <20050610050639.GB2833@esmail.cup.hp.com> Message-ID: <20050610152440.GA10802@esmail.cup.hp.com> On Thu, Jun 09, 2005 at 10:17:33PM -0700, Shirley Ma wrote: > > Please provide "cat /proc/cpuinfo" output so I can rule out > > broken firmware and look for something else. > > processor : 0 > cpu : PPC970, altivec supported > clock : 1600.000000MHz > revision : 2.2 Ah no wonder...every other arch uses "cpu MHz". > I modified the get_cpu_mhz() to for PPC. > rc = sscanf(buf, "clock : %lf", &m); Is 1600Mhz correct? I'm assuming it is. Does the attached patch work for you? It will allow PPC to change to "cpu MHz" and it will still work. Or accomodate other arches that might have cloned PPC code. grant Index: get_clock.c =================================================================== --- get_clock.c (revision 2572) +++ get_clock.c (working copy) @@ -48,8 +48,11 @@ double m; int rc; rc = sscanf(buf, "cpu MHz : %lf", &m); - if (rc != 1) - continue; + if (rc != 1) { /* blech...PPC does it different */ + rc = sscanf(buf, "clock : %lf", &m); + if (rc != 1) + continue; + } if (mhz == 0.0) { mhz = m; continue; From roland at topspin.com Fri Jun 10 08:27:46 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 10 Jun 2005 08:27:46 -0700 Subject: [openib-general] IPoIB: Waiting for ib0 to become free In-Reply-To: <1118411513.4476.20.camel@localhost.localdomain> (Hal Rosenstock's message of "10 Jun 2005 09:51:53 -0400") References: <1118411513.4476.20.camel@localhost.localdomain> Message-ID: <527jh21bbh.fsf@topspin.com> Hal> dev/core.c netdev_wait_allrefs says: * Any protocol or device Hal> that holds a reference should register * for netdevice Hal> notification, and cleanup and put back the * reference if Hal> they receive an UNREGISTER event. Hal> Is it correct that IPoIB does not need to register for these Hal> events ? If it is, then this must be something else which is Hal> using the IPoIB driver causing the reference count to be Hal> incremented but not handling these events. Troy, any idea on Hal> how to recreate this ? IPoIB doesn't need to handle these events, since it is the one doing the unregistering. - R. From halr at voltaire.com Fri Jun 10 09:07:23 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 Jun 2005 12:07:23 -0400 Subject: [openib-general] A couple of kdapltest oopses In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C75@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C75@mtvex01.mtv.mtl.com> Message-ID: <1118419642.4476.31.camel@localhost.localdomain> On Fri, 2005-06-10 at 11:29, Itamar Rabenstein wrote: > is this new ? did you saw it before ? I pretty sure I did. > do you see the problem with -i 1000 ? Yes, but it takes more repeats of the command to make it occur. > both oops are in the malloc stage. > I have seen in the past problems were dapl tried to malloc a very big > memory. Or is it some malloc'd memory is not being returned and eventually runs out ? Shouldn't the failure be more graceful too ? -- Hal > Itamar > > > > --Original Message-- > > From: Hal Rosenstock [mailto:halr at voltaire.com] > > Sent: Friday, June 10, 2005 4:32 PM > > To: James Lentini > > Cc: openib-general at openib.org > > Subject: [openib-general] A couple of kdapltest oopses > > > > > > Hi, > > > > First on the client when running transaction test, I see the > > following: > > kdapltest -T T -s -D mthca0a -d -i 10000 -w > > 8 client SR server SR > > > > Jun 10 08:58:17 localhost kernel: DT_Mdep_Thread_: page > > allocation failure. order:0, mode:0x20 > > Jun 10 08:58:17 localhost kernel: [] > > __alloc_pages+0x2b2/0x440 > > Jun 10 08:58:17 localhost kernel: [] > > kernel_map_pages+0x28/0x70 > > Jun 10 08:58:17 localhost kernel: [] > > kmem_getpages+0x31/0xb0 > > Jun 10 08:58:17 localhost kernel: [] cache_grow+0x139/0x360 > > Jun 10 08:58:17 localhost kernel: [] > > cache_alloc_refill+0x153/0x340 > > Jun 10 08:58:17 localhost kernel: [] dbg_redzone1+0x15/0x30 > > Jun 10 08:58:17 localhost kernel: [] > > cache_alloc_debugcheck_after+0x6e/0x1a0 > > Jun 10 08:58:17 localhost kernel: [] __kmalloc+0xb1/0xe0 > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_Malloc+0x25/0x60 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Tdep_PT_Printf+0x16/0x1b0 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Transaction_Run+0x48f/0xb60 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_wait_object_wakeup+0x1d/0x30 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Transaction_Main+0x1383/0x21a0 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > kernel_map_pages+0x28/0x70 > > Jun 10 08:58:17 localhost kernel: [] > > cache_free_debugcheck+0x196/0x2d0 > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_Thread_Start_Routine+0x1f/0x30 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_Thread_Start_Routine+0x0/0x30 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > kernel_thread_helper+0x5/0x10 > > Jun 10 08:58:17 localhost kernel: Unable to handle kernel > > NULL pointer dereference at virtual address 00000004 > > Jun 10 08:58:17 localhost kernel: printing eip: > > Jun 10 08:58:17 localhost kernel: c022048b > > Jun 10 08:58:17 localhost kernel: *pde = 07cb6067 > > Jun 10 08:58:17 localhost kernel: *pte = 00000000 > > Jun 10 08:58:17 localhost kernel: Oops: 0002 [#1] > > Jun 10 08:58:17 localhost kernel: DEBUG_PAGEALLOC > > Jun 10 08:58:17 localhost kernel: Modules linked in: > > kdapltest ib_dat_provider ib_cm ib_at dat ib_ipoib ib_sa > > ib_umad ide_cd cdrom lp ipv6 autofs parport_pc parport > > uhci_hcd ehci_hcd ib_mthca ib_mad ib_core ohci_hcd eepro100 > > mii evdev usbcore > > Jun 10 08:58:17 localhost kernel: CPU: 0 > > Jun 10 08:58:17 localhost kernel: EIP: 0060:[] > > Not tainted VLI > > Jun 10 08:58:17 localhost kernel: EFLAGS: 00010283 (2.6.11.6) > > Jun 10 08:58:17 localhost kernel: EIP is at vsnprintf+0x4b/0x4c0 > > Jun 10 08:58:17 localhost kernel: eax: 00000054 ebx: > > c1d17f78 ecx: 00000000 edx: d0a35eef > > Jun 10 08:58:17 localhost kernel: esi: 00000004 edi: > > c1d17f78 ebp: 00000103 esp: ce5bfd84 > > Jun 10 08:58:17 localhost kernel: ds: 007b es: 007b ss: 0068 > > Jun 10 08:58:17 localhost kernel: Process DT_Mdep_Thread_ > > (pid: 6696, threadinfo=ce5be000 task=c3730a90) > > Jun 10 08:58:17 localhost kernel: Stack: cffff740 00000020 > > 00000000 d0a33995 00000104 00000000 c1d17f78 d0a33995 > > Jun 10 08:58:17 localhost kernel: 00000104 00000020 > > c1d17f78 00000000 c1d17f78 c5b58000 d0a3484b 00000004 > > Jun 10 08:58:17 localhost kernel: 00000100 d0a35eef > > ce5bfdf4 0000007b ffffff05 c1d17f78 cf34ef78 c60d2060 > > Jun 10 08:58:17 localhost kernel: Call Trace: > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_Malloc+0x25/0x60 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_Malloc+0x25/0x60 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Tdep_PT_Printf+0x3b/0x1b0 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Transaction_Run+0x48f/0xb60 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_wait_object_wakeup+0x1d/0x30 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Transaction_Main+0x1383/0x21a0 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > kernel_map_pages+0x28/0x70 > > Jun 10 08:58:17 localhost kernel: [] > > cache_free_debugcheck+0x196/0x2d0 > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_Thread_Start_Routine+0x1f/0x30 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > DT_Mdep_Thread_Start_Routine+0x0/0x30 [kdapltest] > > Jun 10 08:58:17 localhost kernel: [] > > kernel_thread_helper+0x5/0x10 > > Jun 10 08:58:17 localhost kernel: Code: f0 48 39 c5 73 0d 89 > > f2 f7 da bd ff ff ff ff 89 54 24 40 8b 54 24 44 80 3a 00 74 > > 23 8d 74 26 00 0f b6 02 3c 25 74 3d 39 ee 77 06 <88> 06 8b 54 > > 24 44 46 89 d0 42 89 54 24 44 80 78 01 00 75 e1 39 > > > > then on the server side, when I try to rmmod kdapltest, I get: > > > > Unable to handle kernel paging request at ffffffff88243a05 RIP: > > [] > > PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 > > Oops: 0010 [1] SMP > > CPU 1 > > Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib > > ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT > > ipt_state ip_conntrack iptable_filter ip_tables video button > > battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random > > ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot > > dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod > > Pid: 9094, comm: kdapltest Not tainted 2.6.11.6 > > RIP: 0010:[] [] > > RSP: 0018:ffff810037787e58 EFLAGS: 00010292 > > RAX: 0000000000000061 RBX: ffff810035af6d88 RCX: 0000000000000282 > > RDX: ffff810020712308 RSI: 0000000000000282 RDI: ffff810020711ce0 > > RBP: ffff810035af6d80 R08: ffff810037786000 R09: 0000000000000000 > > R10: 0000000000000000 R11: 0000000000000246 R12: ffff810038daccc0 > > R13: 0000000000000001 R14: 0000000000000000 R15: ffff8100234ad008 > > FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9f00(0000) > > knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > > CR2: ffffffff88243a05 CR3: 000000003a33a000 CR4: 00000000000006e0 > > Process kdapltest (pid: 9094, threadinfo ffff810037786000, > > task ffff81003cd16860) > > Stack: ffff810000002000 0000024000000003 ffff810000000100 > > ffff810000000000 > > ffff810000000000 0000000000000000 ffff81003d40f4c0 > > 0000000100000000 > > ffff81001eb44000 ffff8100234ad000 > > Call Trace:{child_rip+8} > > {child_rip+0} > > > > > > Code: Bad RIP value. > > RIP [] RSP > > CR2: ffffffff88243a05 > > <1>Unable to handle kernel paging request at ffffffff8825eac8 RIP: > > {filp_close+73} > > PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 > > Oops: 0000 [2] SMP > > CPU 1 > > Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib > > ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT > > ipt_state ip_conntrack iptable_filter ip_tables video button > > battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random > > ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot > > dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod > > Pid: 9094, comm: kdapltest Not tainted 2.6.11.6 > > RIP: 0010:[] {filp_close+73} > > RSP: 0018:ffff810037787bf8 EFLAGS: 00010286 > > RAX: ffffffff8825ea60 RBX: ffff81001f24a8c0 RCX: ffff810019109238 > > RDX: ffff810037e58e70 RSI: ffff810037e58d40 RDI: ffff81001f24a8c0 > > RBP: 0000000000000000 R08: ffff81003d7ce6c0 R09: ffff810037787bf8 > > R10: 0000000000000001 R11: 0000000000000000 R12: ffff810037e58d40 > > R13: 0000000000000001 R14: 0000000000000001 R15: ffff81001f278940 > > FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9f00(0000) > > knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > > CR2: ffffffff8825eac8 CR3: 000000003a33a000 CR4: 00000000000006e0 > > Process kdapltest (pid: 9094, threadinfo ffff810037786000, > > task ffff81003cd16860) > > Stack: 0000000000000003 0000000000000003 ffff810037e58d40 > > ffffffff80139723 > > 0000000000000009 ffff810037e58d40 ffff81003cd16eb0 > > ffff81003cd16860 > > 0000000000000009 ffffffff80139eba > > Call Trace:{put_files_struct+115} > > {do_exit+378} > > {do_unblank_screen+119} > > {do_page_fault+2047} > > {thread_return+42} > > {error_exit+0} > > {child_rip+8} > > {child_rip+0} > > > > > > Code: 48 8b 40 68 48 85 c0 74 0e 48 89 df ff d0 85 ed 0f 44 e8 66 > > RIP {filp_close+73} RSP > > CR2: ffffffff8825eac8 > > <1>Unable to handle kernel paging request at ffffffff88245190 RIP: > > [] > > PGD 103027 PUD 105027 PMD 3baeb067 PTE 0 > > Oops: 0010 [3] SMP > > CPU 0 > > Modules linked in: ib_dat_provider ib_cm ib_at dat ib_ipoib > > ib_sa parport_pc lp parport autofs4 sunrpc ipt_REJECT > > ipt_state ip_conntrack iptable_filter ip_tables video button > > battery ac md5 ipv6 ohci_hcd i2c_amd8111 i2c_core hw_random > > ib_mthca ib_mad ib_core e100 mii tg3 floppy dm_snapshot > > dm_zero dm_mirror ext3 jbd dm_mod sata_sil libata sd_mod scsi_mod > > Pid: 9119, comm: DT_Mdep_Thread_ Not tainted 2.6.11.6 > > RIP: 0010:[] [] > > RSP: 0018:ffff810031907f18 EFLAGS: 00010296 > > RAX: 0000000000000000 RBX: ffff81001b770000 RCX: 0000000000000006 > > RDX: 0000000000000008 RSI: 0000000000000008 RDI: 0000000000000003 > > RBP: ffff81001b77004c R08: ffffffff80508a00 R09: 0000000000000008 > > R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001 > > R13: ffff810038daccc0 R14: 0000000000000000 R15: ffff8100234ad008 > > FS: 00002aaaaaaccec0(0000) GS:ffffffff804c9e80(0000) > > knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > > CR2: ffffffff88245190 CR3: 000000003d8b3000 CR4: 00000000000006e0 > > Process DT_Mdep_Thread_ (pid: 9119, threadinfo > > ffff810031906000, task ffff81003cd16170) > > Stack: ffff810037bde1c0 ffff810035af6d80 ffff810038daccc0 > > 0000000000000001 > > ffff81001b770000 ffffffff88250ff8 ffff810037bde1c0 > > ffffffff8010f1c7 > > ffff8100234ad008 ffff81001b770000 > > Call Trace:{child_rip+8} > > {flat_send_IPI_mask+0} > > {child_rip+0} > > > > Code: Bad RIP value. > > RIP [] RSP > > CR2: ffffffff88245190 > > > > -- Hal > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > From iod00d at hp.com Fri Jun 10 09:45:37 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 10 Jun 2005 09:45:37 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: References: <20050610003352.GE12434@esmail.cup.hp.com> Message-ID: <20050610164537.GB10802@esmail.cup.hp.com> On Thu, Jun 09, 2005 at 10:13:13PM -0700, Shirley Ma wrote: > >I think adding such a calibration to get_cpu_mhz() > >so we can warn if /proc/cpuinfo data doesn't agree with > >gettimeofday() and get_cycles(). > > using below to calculate the cycles_to_unit is more accurate. > > clock_gettime(); > get_cycles(); > sleep(5); > clock_gettime(); > get_cycles(); While I agree in general with the concept, I don't agree with this algorithm for two reaons: o I don't trust get_cycles() to measure anything more than 1s. Problem is on some 32-bit arches, they may only have (or only read) a 32-bit cycle counter. This can wrap in as little as 2 or 3 seconds. This obviously isn't a problem with 64-bit counters. o Trivial but annoying - adding 5 seconds per run means a scripted batch job could take minutes longer. > cycles_to_unit = offset - cycles/ offset - time; I think we can effectively do the same thing by summing the recorded "get_cycles" values and compare that sum to before/after gettimeofday() readings. I'll submit a patch later today that effectively does that. grant From tduffy at sun.com Fri Jun 10 10:23:57 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 10 Jun 2005 10:23:57 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C74@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C74@mtvex01.mtv.mtl.com> Message-ID: <1118424237.26791.4.camel@duffman> On Fri, 2005-06-10 at 08:33 +0300, Itamar Rabenstein wrote: > I think that sp+psp+rsp should be one file dapl_sp.c You are correct. I should have done this the first time. Here is a patch to fix this. Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_ia.c =================================================================== --- linux-kernel/dat-provider/dapl_ia.c (revision 2585) +++ linux-kernel/dat-provider/dapl_ia.c (working copy) @@ -35,7 +35,7 @@ #include "dapl_evd.h" #include "dapl_hca_util.h" #include "dapl_openib_util.h" -#include "dapl_sp_util.h" +#include "dapl_sp.h" #include "dapl_cr.h" struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, Index: linux-kernel/dat-provider/Makefile =================================================================== --- linux-kernel/dat-provider/Makefile (revision 2585) +++ linux-kernel/dat-provider/Makefile (working copy) @@ -28,13 +28,10 @@ PROVIDER_MODULES := \ dapl_lmr \ dapl_mr_util \ dapl_provider \ - dapl_sp_util \ - dapl_psp \ dapl_pz \ dapl_ring_buffer_util \ dapl_rmr \ - dapl_rsp \ - dapl_sp_util \ + dapl_sp \ dapl_srq \ dapl_util Index: linux-kernel/dat-provider/dapl_psp.c =================================================================== --- linux-kernel/dat-provider/dapl_psp.c (revision 2585) +++ linux-kernel/dat-provider/dapl_psp.c (working copy) @@ -1,445 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_sp_util.h" -#include "dapl_ia.h" -#include "dapl_openib_util.h" - -/* - * dapl_psp_create_any - * - * Create a persistent Public Service Point that can recieve multiple - * requests for connections and generate multiple connection request - * instances that wil be delivered to the specified Event Dispatcher - * in a notification event. Differs from dapl_psp_create() in that - * the conn_qual is selected by the implementation and returned to - * the user. - * - * Input: - * ia - * evd - * psp_flags - * - * Output: - * conn_qual - * psp - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_HANDLE - * DAT_INVALID_PARAMETER - * DAT_CONN_QUAL_IN_USE - * DAT_MODEL_NOT_SUPPORTED - */ -u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, - struct dat_evd *evd, enum dat_psp_flags psp_flags, - struct dat_sp **psp) -{ - static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - struct dapl_evd *evd_ptr; - u32 status = DAT_SUCCESS; - int i; - - ia_ptr = (struct dapl_ia *)ia; - - if (!ia_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (!evd) { - status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (psp == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - if (conn_qual == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - evd_ptr = (struct dapl_evd *)evd; - if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (psp_flags != DAT_PSP_CONSUMER_FLAG && - psp_flags != DAT_PSP_PROVIDER_FLAG) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - - /* Allocate PSP */ - sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); - if (sp_ptr == NULL) { - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - - /* - * Fill out the args for a PSP - */ - sp_ptr->ia = ia; - sp_ptr->evd = evd; - sp_ptr->psp_flags = psp_flags; - sp_ptr->ep = NULL; - - /* - * Take a reference on the EVD handle - */ - atomic_inc(&evd_ptr->evd_ref_count); - - /* Link it onto the IA */ - dapl_ia_link_psp(ia_ptr, sp_ptr); - - /* - * Set up a listener for a connection. Connections can arrive - * even before this call returns! - */ - sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; - sp_ptr->listening = TRUE; - - /* - * If we have a big number of tries and we still haven't - * found a service_ID we can use, bail out with an error, - * something is wrong! - */ - for (i = 0, sp_ptr->conn_qual = hint_conn_qual; i < 100000; i++) { - status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); - if (DAT_SUCCESS == status) - break; - else - sp_ptr->conn_qual++; - } - hint_conn_qual = sp_ptr->conn_qual + 1; - - if (status != DAT_SUCCESS) { - atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd = NULL; - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_psp_create cannot set up conn listener: %x\n", - status); - - goto bail; - } - - /* - * Return handle to the user - */ - *conn_qual = sp_ptr->conn_qual; - *psp = (struct dat_sp *) sp_ptr; - -bail: - return status; -} - -/* - * dapl_psp_create - * - * Create a persistent Public Service Point that can recieve multiple - * requests for connections and generate multiple connection request - * instances that wil be delivered to the specified Event Dispatcher - * in a notification event. - * - * Input: - * ia - * conn_qual - * evd - * psp_flags - * - * Output: - * psp - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_PARAMETER - * DAT_CONN_QUAL_IN_USE - * DAT_MODEL_NOT_SUPPORTED - */ -u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, - struct dat_evd *evd, enum dat_psp_flags psp_flags, - struct dat_sp **psp) -{ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - struct dapl_evd *evd_ptr; - boolean_t sp_found; - u32 status = DAT_SUCCESS; - - ia_ptr = (struct dapl_ia *)ia; - - if (!ia_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (!evd) { - status = DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (psp == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - - evd_ptr = (struct dapl_evd *)evd; - if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (psp_flags != DAT_PSP_CONSUMER_FLAG && - psp_flags != DAT_PSP_PROVIDER_FLAG) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - - /* - * See if we have a quiescent listener to use for this PSP, else - * create one and set it listening - */ - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, TRUE); - sp_found = TRUE; - if (sp_ptr == NULL) { - /* Allocate PSP */ - sp_found = FALSE; - sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); - if (sp_ptr == NULL) { - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - } else if (sp_ptr->listening == TRUE) { - status = DAT_ERROR(DAT_CONN_QUAL_IN_USE, 0); - goto bail; - } - - /* - * Fill out the args for a PSP - */ - sp_ptr->ia = ia; - sp_ptr->conn_qual = conn_qual; - sp_ptr->evd = evd; - sp_ptr->psp_flags = psp_flags; - sp_ptr->ep = NULL; - - /* - * Take a reference on the EVD handle - */ - atomic_inc(&evd_ptr->evd_ref_count); - - /* - * Set up a listener for a connection. Connections can arrive - * even before this call returns! - */ - sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; - sp_ptr->listening = TRUE; - - /* - * If this is a new sp we need to add it to the IA queue, and set up - * a conn_listener. - */ - if (sp_found == FALSE) { - /* - * Link it onto the IA before enabling it to receive conn - * requests - */ - dapl_ia_link_psp(ia_ptr, sp_ptr); - - status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); - - if (status != DAT_SUCCESS) { - /* - * Have a problem setting up the connection, something - * wrong! Decrements the EVD refcount & release it. - */ - atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd = NULL; - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - - dapl_dbg_log(DAPL_DBG_TYPE_CM, "dapl_psp_create " - "setup_conn_listener failed: %x\n", - status); - - goto bail; - } - } - - /* - * Return handle to the user - */ - *psp = (struct dat_sp *) sp_ptr; - -bail: - return status; -} - -/* - * dapl_psp_free - * - * Destroy a specific instance of a Service Point. - * - * Input: - * psp - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ -u32 dapl_psp_free(struct dat_sp *psp) -{ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - DAPL_SP_STATE save_state; - u32 status = DAT_SUCCESS; - - sp_ptr = (struct dapl_sp *)psp; - /* - * Verify handle - */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp); - - if (!sp_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); - goto bail; - } - - ia_ptr = sp_ptr->common.owner_ia; - /* - * Remove the connection listener if it has been established - * and there are no current connections in progress. - * If we defer removing the sp it becomes something of a zombie - * container until the last connection is disconnected, after - * which it will be cleaned up. - */ - spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); - - sp_ptr->listening = FALSE; - - /* - * Release reference on EVD. If an error was encountered in a previous - * free the evd will be NULL - */ - if (sp_ptr->evd) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> - evd_ref_count); - sp_ptr->evd = NULL; - } - - /* - * Release the base resource if there are no outstanding - * connections; else the last disconnect on this PSP will free it - * up. The PSP is used to contain CR records for each connection, - * which contain information necessary to disconnect. - */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, - ">>> dapl_psp_free: state %d cr_list_count %d\n", - sp_ptr->state, sp_ptr->cr_list_count); - if ((sp_ptr->state == DAPL_SP_STATE_PSP_LISTENING || - sp_ptr->state == DAPL_SP_STATE_PSP_PENDING) && - sp_ptr->cr_list_count == 0) { - save_state = sp_ptr->state; - sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->common.lock, - sp_ptr->common.flags); - - status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); - if (status != DAT_SUCCESS) { - /* revert to entry state on error */ - sp_ptr->state = save_state; - goto bail; - } - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - } else { - /* - * The PSP is now in the pending state, where it will sit until - * the last connection terminates or the app uses the same - * ServiceID again, which will reactivate it. - */ - sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->common.lock, - sp_ptr->common.flags); - dapl_dbg_log(DAPL_DBG_TYPE_CM, - ">>> dapl_psp_free: PSP PENDING\n"); - } - -bail: - return status; -} - -u32 dapl_psp_query(struct dat_sp *psp, struct dat_psp_param *psp_param) -{ - struct dapl_sp *sp_ptr; - u32 status; - - if (!psp || ((struct dapl_sp *)psp)->listening != TRUE) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); - goto bail; - } - - if (NULL == psp_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - sp_ptr = (struct dapl_sp *)psp; - - psp_param->ia = sp_ptr->ia; - psp_param->conn_qual = sp_ptr->conn_qual; - psp_param->evd = sp_ptr->evd; - psp_param->psp_flags = sp_ptr->psp_flags; - - status = DAT_SUCCESS; - -bail: - return status; -} Index: linux-kernel/dat-provider/dapl_cr.c =================================================================== --- linux-kernel/dat-provider/dapl_cr.c (revision 2585) +++ linux-kernel/dat-provider/dapl_cr.c (working copy) @@ -35,7 +35,7 @@ #include "dapl_evd.h" #include "dapl_ia.h" #include "dapl_openib_util.h" -#include "dapl_sp_util.h" +#include "dapl_sp.h" /* * Create a CR. Part of the passive side of a connection Index: linux-kernel/dat-provider/dapl_rsp.c =================================================================== --- linux-kernel/dat-provider/dapl_rsp.c (revision 2585) +++ linux-kernel/dat-provider/dapl_rsp.c (working copy) @@ -1,314 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_sp_util.h" -#include "dapl_ia.h" -#include "dapl_ep.h" -#include "dapl_openib_util.h" - -/* - * dapl_rsp_create - * - * Create a Resereved Service Point with the specified Endpoint - * that generates at most one Connection Request that is - * delivered to the specified Event Dispatcher in a notification - * event - * - * Input: - * ia - * conn_qual - * ep - * evd - * - * Output: - * rsp - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_PARAMETER - * DAT_INVALID_STATE - * DAT_CONN_QUAL_IN_USE - */ -u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, - struct dat_ep *ep, struct dat_evd *evd, - struct dat_sp **rsp) -{ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - struct dapl_evd *evd_ptr; - struct dapl_ep *ep_ptr; - boolean_t sp_found; - u32 status = DAT_SUCCESS; - - ia_ptr = (struct dapl_ia *)ia; - - dapl_dbg_log(DAPL_DBG_TYPE_CM, - ">>> dapl_rsp_free conn_qual: %x EP: %p\n", - conn_qual, ep); - - if (!ia_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (!ep) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - if (!evd) { - status = DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (rsp == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - - ep_ptr = (struct dapl_ep *)ep; - if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { - status = DAT_ERROR(DAT_INVALID_STATE, - dapl_ep_state_subtype(ep_ptr)); - goto bail; - } - - evd_ptr = (struct dapl_evd *)evd; - if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, FALSE); - sp_found = TRUE; - if (sp_ptr == NULL) { - sp_found = FALSE; - - /* Allocate RSP */ - sp_ptr = dapl_sp_alloc(ia_ptr, FALSE); - if (sp_ptr == NULL) { - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - } - - /* - * Fill out the RSP args - */ - sp_ptr->ia = ia; - sp_ptr->conn_qual = conn_qual; - sp_ptr->evd = evd; - sp_ptr->psp_flags = 0; - sp_ptr->ep = ep; - - /* - * Take a reference on the EVD handle - */ - atomic_inc(&evd_ptr->evd_ref_count); - - /* - * Update the EP state indicating the provider now owns it - */ - ep_ptr->param.ep_state = DAT_EP_STATE_RESERVED; - - /* - * Set up a listener for a connection. Connections can arrive - * even before this call returns! - */ - sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; - sp_ptr->listening = TRUE; - - if (sp_found == FALSE) { - /* Link it onto the IA */ - dapl_ia_link_rsp(ia_ptr, sp_ptr); - - status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); - - if (status != DAT_SUCCESS) { - /* - * Have a problem setting up the connection, something - * wrong! Decrements the EVD refcount & release it. Set - * the state to FREE, so we know the call failed. - */ - atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd = NULL; - sp_ptr->state = DAPL_SP_STATE_FREE; - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - - dapl_dbg_log(DAPL_DBG_TYPE_CM, - "--> dapl_rsp_create setup_conn_listener failed: %x\n", - status); - - goto bail; - } - } - - /* - * Return handle to the user - */ - *rsp = (struct dat_sp *)sp_ptr; - -bail: - return status; -} - -/* - * dapl_rsp_free - * - * Destroy a specific instance of a Reserved Service Point. - * - * Input: - * rsp - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - */ -u32 dapl_rsp_free(struct dat_sp *rsp) -{ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - struct dapl_ep *ep_ptr; - u32 status = DAT_SUCCESS; - - sp_ptr = (struct dapl_sp *)rsp; - /* - * Verify handle - */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp); - if (!sp_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); - goto bail; - } - - ia_ptr = sp_ptr->common.owner_ia; - - /* - * Remove the connection listener if there are no connections. If - * we defer removing the sp it becomes something of a zombie - * container until disconnection, after which it will be cleaned up. - */ - spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); - - /* - * Make sure we don't leave a dangling EP. If the state is still - * RESERVED then the RSP still owns it. - */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep; - if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) - ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - sp_ptr->ep = NULL; - - /* - * Release reference on EVD. If an error was encountered in a previous - * free the evd will be NULL - */ - if (sp_ptr->evd) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> - evd_ref_count); - sp_ptr->evd = NULL; - } - - /* - * Release the base resource if there are no outstanding connections; - * else the last disconnect on this RSP will free it up. The RSP - * is used to contain CR records for each connection, which - * contain information necessary to disconnect. - * sp_ptr->listening will be TRUE if there has never been a - * connection event, and FALSE if a connection attempt resulted - * in a reject. - */ - if (sp_ptr->cr_list_count == 0) { - /* This RSP has never been used. Clean it up */ - sp_ptr->listening = FALSE; - sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->common.lock, - sp_ptr->common.flags); - - status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); - if (status != DAT_SUCCESS) { - sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; - goto bail; - } - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - } else { - /* - * The RSP is now in the pending state, where it will sit until - * the connection terminates or the app uses the same - * ServiceID again, which will reactivate it. - */ - sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->common.lock, - sp_ptr->common.flags); - } - -bail: - return status; -} - -u32 dapl_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *rsp_param) -{ - struct dapl_sp *sp_ptr; - u32 status; - - if (!rsp) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); - goto bail; - } - - if (NULL == rsp_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - sp_ptr = (struct dapl_sp *)rsp; - - /* - * Fill in the RSP params - */ - rsp_param->ia = sp_ptr->ia; - rsp_param->conn_qual = sp_ptr->conn_qual; - rsp_param->evd = sp_ptr->evd; - rsp_param->ep = sp_ptr->ep; - - status = DAT_SUCCESS; - -bail: - return status; -} Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2585) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -39,7 +39,7 @@ #include "dapl_openib_dto.h" #include "dapl_openib_util.h" #include "dapl_ring_buffer_util.h" -#include "dapl_sp_util.h" +#include "dapl_sp.h" static void dapl_ep_dealloc(struct dapl_ep *ep_ptr) { Index: linux-kernel/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel/dat-provider/dapl_sp_util.c (revision 2585) +++ linux-kernel/dat-provider/dapl_sp_util.c (working copy) @@ -1,240 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_sp_util.h" -#include "dapl_cr.h" - -/* - * Local definitions - */ - -/* - * dapl_sp_alloc - * - * alloc and initialize an SP - * - * Input: - * IA INFO struct ptr - * - * Output: - * sp_ptr - * - * Returns: - * NULL - * pointer to sp info struct - * - */ -struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp) -{ - struct dapl_sp *sp_ptr; - - /* Allocate EP */ - sp_ptr = kmalloc(sizeof *sp_ptr, GFP_ATOMIC); - if (!sp_ptr) - return NULL; - - /* zero the structure */ - memset(sp_ptr, 0, sizeof *sp_ptr); - - sp_ptr->sp.common.provider = ia_ptr->ia.common.provider; - if (is_psp) - sp_ptr->sp.type = DAT_SP_TYPE_PSP; - else - sp_ptr->sp.type = DAT_SP_TYPE_RSP; - sp_ptr->common.owner_ia = ia_ptr; - dapl_llist_init_entry(&sp_ptr->common.ia_list_entry); - spin_lock_init(&sp_ptr->common.lock); - - /* - * Initialize the Body (set to NULL above) - */ - dapl_llist_init_head(&sp_ptr->cr_list_head); - - return sp_ptr; -} - -/* - * dapl_sp_dealloc - * - * Free the passed in SP structure. - * - * Input: - * entry point pointer - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_sp_dealloc(struct dapl_sp *sp_ptr) -{ - dapl_os_assert(dapl_llist_is_empty(&sp_ptr->cr_list_head)); - - kfree(sp_ptr); -} - -/* - * dapl_cr_link_cr - * - * Add a cr to a PSP structure - * - * Input: - * sp_ptr - * cr_ptr - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) -{ - spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); - dapl_llist_add_tail(&sp_ptr->cr_list_head, - &cr_ptr->common.ia_list_entry, cr_ptr); - sp_ptr->cr_list_count++; - spin_unlock_irqrestore(&sp_ptr->common.lock, sp_ptr->common.flags); -} - -/* - * dapl_sp_search_cr - * - * Search for a CR on the PSP cr_list with a matching cm_handle. When - * found, remove it from the list and update fields. - * - * Must be called with the sp_ptr lock taken. - * - * Input: - * sp_ptr - * ib_cm_handle - * - * Output: - * none - * - * Returns: - * cr_ptr Pointer to matching DAPL_CR - * - */ -struct dapl_cr *dapl_sp_search_cr(struct dapl_sp *sp_ptr, - struct dapl_cm_id *ib_cm_handle) -{ - struct dapl_cr *cr_ptr; - - if (dapl_llist_is_empty(&sp_ptr->cr_list_head)) - return NULL; - - cr_ptr = (struct dapl_cr *)dapl_llist_peek_head(&sp_ptr->cr_list_head); - - do { - if (cr_ptr->ib_cm_handle == ib_cm_handle) - return cr_ptr; - - cr_ptr = cr_ptr->common.ia_list_entry.flink->data; - } while ((void *)cr_ptr != (void *)sp_ptr->cr_list_head->data); - - return NULL; -} - -/* - * dapl_sp_remove_cr - * - * Remove the CR from the PSP. Done prior to freeing the CR resource. - * - * Must be called with the sp_ptr lock taken. - * - * Input: - * sp_ptr - * cr_ptr - * - * Output: - * none - * - * Returns: - * void - * - */ -void dapl_sp_remove_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) -{ - if (dapl_llist_is_empty(&sp_ptr->cr_list_head)) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "***dapl_sp_remove_cr: removing from empty queue! sp %p\n", - sp_ptr); - return; - } - - dapl_llist_remove_entry(&sp_ptr->cr_list_head, - &cr_ptr->common.ia_list_entry); - sp_ptr->cr_list_count--; -} - -/* - * dapl_sp_remove_ep - * - * Remove a CR from a PSP, given an EP. - * - * - * Input: - * ep_ptr - * - * Output: - * none - * - * Returns: - * void - * - */ -void dapl_sp_remove_ep(struct dapl_ep *ep_ptr) -{ - struct dapl_sp *sp_ptr; - struct dapl_cr *cr_ptr; - - cr_ptr = ep_ptr->cr_ptr; - - if (cr_ptr != NULL) { - sp_ptr = cr_ptr->sp_ptr; - - spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); - - /* Remove the CR from the queue */ - dapl_sp_remove_cr(sp_ptr, cr_ptr); - - spin_unlock_irqrestore(&sp_ptr->common.lock, - sp_ptr->common.flags); - - /* free memory outside of the lock */ - dapl_cr_free(cr_ptr); - } -} Index: linux-kernel/dat-provider/dapl_sp.c =================================================================== --- linux-kernel/dat-provider/dapl_sp.c (revision 0) +++ linux-kernel/dat-provider/dapl_sp.c (revision 0) @@ -0,0 +1,931 @@ +/* + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * + * This Software is licensed under one of the following licenses: + * + * 1) under the terms of the "Common Public License 1.0" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/cpl.php. + * + * 2) under the terms of the "The BSD License" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/bsd-license.php. + * + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a + * copy of which is available from the Open Source Initiative, see + * http://www.opensource.org/licenses/gpl-license.php. + * + * Licensee has the right to choose one of the above licenses. + * + * Redistributions of source code must retain the above copyright + * notice and one of the license notices. + * + * Redistributions in binary form must reproduce both the above copyright + * notice, one of the license notices in the documentation + * and/or other materials provided with the distribution. + */ + +/* + * $Id: dapl_sp_util.c 2584 2005-06-10 15:31:30Z jlentini $ + */ + +#include "dapl.h" +#include "dapl_sp.h" +#include "dapl_cr.h" +#include "dapl_ep.h" +#include "dapl_ia.h" +#include "dapl_openib_util.h" + +/* + * Local definitions + */ + +/* + * dapl_sp_alloc + * + * alloc and initialize an SP + * + * Input: + * IA INFO struct ptr + * + * Output: + * sp_ptr + * + * Returns: + * NULL + * pointer to sp info struct + * + */ +struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp) +{ + struct dapl_sp *sp_ptr; + + /* Allocate EP */ + sp_ptr = kmalloc(sizeof *sp_ptr, GFP_ATOMIC); + if (!sp_ptr) + return NULL; + + /* zero the structure */ + memset(sp_ptr, 0, sizeof *sp_ptr); + + sp_ptr->sp.common.provider = ia_ptr->ia.common.provider; + if (is_psp) + sp_ptr->sp.type = DAT_SP_TYPE_PSP; + else + sp_ptr->sp.type = DAT_SP_TYPE_RSP; + sp_ptr->common.owner_ia = ia_ptr; + dapl_llist_init_entry(&sp_ptr->common.ia_list_entry); + spin_lock_init(&sp_ptr->common.lock); + + /* + * Initialize the Body (set to NULL above) + */ + dapl_llist_init_head(&sp_ptr->cr_list_head); + + return sp_ptr; +} + +/* + * dapl_sp_dealloc + * + * Free the passed in SP structure. + * + * Input: + * entry point pointer + * + * Output: + * none + * + * Returns: + * none + * + */ +void dapl_sp_dealloc(struct dapl_sp *sp_ptr) +{ + dapl_os_assert(dapl_llist_is_empty(&sp_ptr->cr_list_head)); + + kfree(sp_ptr); +} + +/* + * dapl_cr_link_cr + * + * Add a cr to a PSP structure + * + * Input: + * sp_ptr + * cr_ptr + * + * Output: + * none + * + * Returns: + * none + * + */ +void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) +{ + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); + dapl_llist_add_tail(&sp_ptr->cr_list_head, + &cr_ptr->common.ia_list_entry, cr_ptr); + sp_ptr->cr_list_count++; + spin_unlock_irqrestore(&sp_ptr->common.lock, sp_ptr->common.flags); +} + +/* + * dapl_sp_search_cr + * + * Search for a CR on the PSP cr_list with a matching cm_handle. When + * found, remove it from the list and update fields. + * + * Must be called with the sp_ptr lock taken. + * + * Input: + * sp_ptr + * ib_cm_handle + * + * Output: + * none + * + * Returns: + * cr_ptr Pointer to matching DAPL_CR + * + */ +struct dapl_cr *dapl_sp_search_cr(struct dapl_sp *sp_ptr, + struct dapl_cm_id *ib_cm_handle) +{ + struct dapl_cr *cr_ptr; + + if (dapl_llist_is_empty(&sp_ptr->cr_list_head)) + return NULL; + + cr_ptr = (struct dapl_cr *)dapl_llist_peek_head(&sp_ptr->cr_list_head); + + do { + if (cr_ptr->ib_cm_handle == ib_cm_handle) + return cr_ptr; + + cr_ptr = cr_ptr->common.ia_list_entry.flink->data; + } while ((void *)cr_ptr != (void *)sp_ptr->cr_list_head->data); + + return NULL; +} + +/* + * dapl_sp_remove_cr + * + * Remove the CR from the PSP. Done prior to freeing the CR resource. + * + * Must be called with the sp_ptr lock taken. + * + * Input: + * sp_ptr + * cr_ptr + * + * Output: + * none + * + * Returns: + * void + * + */ +void dapl_sp_remove_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) +{ + if (dapl_llist_is_empty(&sp_ptr->cr_list_head)) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "***dapl_sp_remove_cr: removing from empty queue! sp %p\n", + sp_ptr); + return; + } + + dapl_llist_remove_entry(&sp_ptr->cr_list_head, + &cr_ptr->common.ia_list_entry); + sp_ptr->cr_list_count--; +} + +/* + * dapl_sp_remove_ep + * + * Remove a CR from a PSP, given an EP. + * + * + * Input: + * ep_ptr + * + * Output: + * none + * + * Returns: + * void + * + */ +void dapl_sp_remove_ep(struct dapl_ep *ep_ptr) +{ + struct dapl_sp *sp_ptr; + struct dapl_cr *cr_ptr; + + cr_ptr = ep_ptr->cr_ptr; + + if (cr_ptr != NULL) { + sp_ptr = cr_ptr->sp_ptr; + + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); + + /* Remove the CR from the queue */ + dapl_sp_remove_cr(sp_ptr, cr_ptr); + + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + + /* free memory outside of the lock */ + dapl_cr_free(cr_ptr); + } +} + +/* + * dapl_rsp_create + * + * Create a Resereved Service Point with the specified Endpoint + * that generates at most one Connection Request that is + * delivered to the specified Event Dispatcher in a notification + * event + * + * Input: + * ia + * conn_qual + * ep + * evd + * + * Output: + * rsp + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_PARAMETER + * DAT_INVALID_STATE + * DAT_CONN_QUAL_IN_USE + */ +u32 dapl_rsp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_ep *ep, struct dat_evd *evd, + struct dat_sp **rsp) +{ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; + struct dapl_ep *ep_ptr; + boolean_t sp_found; + u32 status = DAT_SUCCESS; + + ia_ptr = (struct dapl_ia *)ia; + + dapl_dbg_log(DAPL_DBG_TYPE_CM, + ">>> dapl_rsp_free conn_qual: %x EP: %p\n", + conn_qual, ep); + + if (!ia_ptr) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + if (!ep) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); + goto bail; + } + if (!evd) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (rsp == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); + goto bail; + } + + ep_ptr = (struct dapl_ep *)ep; + if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { + status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(ep_ptr)); + goto bail; + } + + evd_ptr = (struct dapl_evd *)evd; + if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, FALSE); + sp_found = TRUE; + if (sp_ptr == NULL) { + sp_found = FALSE; + + /* Allocate RSP */ + sp_ptr = dapl_sp_alloc(ia_ptr, FALSE); + if (sp_ptr == NULL) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } + } + + /* + * Fill out the RSP args + */ + sp_ptr->ia = ia; + sp_ptr->conn_qual = conn_qual; + sp_ptr->evd = evd; + sp_ptr->psp_flags = 0; + sp_ptr->ep = ep; + + /* + * Take a reference on the EVD handle + */ + atomic_inc(&evd_ptr->evd_ref_count); + + /* + * Update the EP state indicating the provider now owns it + */ + ep_ptr->param.ep_state = DAT_EP_STATE_RESERVED; + + /* + * Set up a listener for a connection. Connections can arrive + * even before this call returns! + */ + sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; + sp_ptr->listening = TRUE; + + if (sp_found == FALSE) { + /* Link it onto the IA */ + dapl_ia_link_rsp(ia_ptr, sp_ptr); + + status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); + + if (status != DAT_SUCCESS) { + /* + * Have a problem setting up the connection, something + * wrong! Decrements the EVD refcount & release it. Set + * the state to FREE, so we know the call failed. + */ + atomic_dec(&evd_ptr->evd_ref_count); + sp_ptr->evd = NULL; + sp_ptr->state = DAPL_SP_STATE_FREE; + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_CM, + "--> dapl_rsp_create setup_conn_listener failed: %x\n", + status); + + goto bail; + } + } + + /* + * Return handle to the user + */ + *rsp = (struct dat_sp *)sp_ptr; + +bail: + return status; +} + +/* + * dapl_rsp_free + * + * Destroy a specific instance of a Reserved Service Point. + * + * Input: + * rsp + * + * Output: + * none + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_HANDLE + */ +u32 dapl_rsp_free(struct dat_sp *rsp) +{ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_ep *ep_ptr; + u32 status = DAT_SUCCESS; + + sp_ptr = (struct dapl_sp *)rsp; + /* + * Verify handle + */ + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp); + if (!sp_ptr) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); + goto bail; + } + + ia_ptr = sp_ptr->common.owner_ia; + + /* + * Remove the connection listener if there are no connections. If + * we defer removing the sp it becomes something of a zombie + * container until disconnection, after which it will be cleaned up. + */ + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); + + /* + * Make sure we don't leave a dangling EP. If the state is still + * RESERVED then the RSP still owns it. + */ + ep_ptr = (struct dapl_ep *)sp_ptr->ep; + if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) + ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; + sp_ptr->ep = NULL; + + /* + * Release reference on EVD. If an error was encountered in a previous + * free the evd will be NULL + */ + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> + evd_ref_count); + sp_ptr->evd = NULL; + } + + /* + * Release the base resource if there are no outstanding connections; + * else the last disconnect on this RSP will free it up. The RSP + * is used to contain CR records for each connection, which + * contain information necessary to disconnect. + * sp_ptr->listening will be TRUE if there has never been a + * connection event, and FALSE if a connection attempt resulted + * in a reject. + */ + if (sp_ptr->cr_list_count == 0) { + /* This RSP has never been used. Clean it up */ + sp_ptr->listening = FALSE; + sp_ptr->state = DAPL_SP_STATE_FREE; + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + + status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); + if (status != DAT_SUCCESS) { + sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; + goto bail; + } + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + } else { + /* + * The RSP is now in the pending state, where it will sit until + * the connection terminates or the app uses the same + * ServiceID again, which will reactivate it. + */ + sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + } + +bail: + return status; +} + +u32 dapl_rsp_query(struct dat_sp *rsp, struct dat_rsp_param *rsp_param) +{ + struct dapl_sp *sp_ptr; + u32 status; + + if (!rsp) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); + goto bail; + } + + if (NULL == rsp_param) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto bail; + } + + sp_ptr = (struct dapl_sp *)rsp; + + /* + * Fill in the RSP params + */ + rsp_param->ia = sp_ptr->ia; + rsp_param->conn_qual = sp_ptr->conn_qual; + rsp_param->evd = sp_ptr->evd; + rsp_param->ep = sp_ptr->ep; + + status = DAT_SUCCESS; + +bail: + return status; +} + +/* + * dapl_psp_create_any + * + * Create a persistent Public Service Point that can recieve multiple + * requests for connections and generate multiple connection request + * instances that wil be delivered to the specified Event Dispatcher + * in a notification event. Differs from dapl_psp_create() in that + * the conn_qual is selected by the implementation and returned to + * the user. + * + * Input: + * ia + * evd + * psp_flags + * + * Output: + * conn_qual + * psp + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_HANDLE + * DAT_INVALID_PARAMETER + * DAT_CONN_QUAL_IN_USE + * DAT_MODEL_NOT_SUPPORTED + */ +u32 dapl_psp_create_any(struct dat_ia *ia, DAT_CONN_QUAL *conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) +{ + static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; + u32 status = DAT_SUCCESS; + int i; + + ia_ptr = (struct dapl_ia *)ia; + + if (!ia_ptr) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + if (!evd) { + status = + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (psp == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); + goto bail; + } + if (conn_qual == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + goto bail; + } + + evd_ptr = (struct dapl_evd *)evd; + if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (psp_flags != DAT_PSP_CONSUMER_FLAG && + psp_flags != DAT_PSP_PROVIDER_FLAG) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); + goto bail; + } + + /* Allocate PSP */ + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); + if (sp_ptr == NULL) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } + + /* + * Fill out the args for a PSP + */ + sp_ptr->ia = ia; + sp_ptr->evd = evd; + sp_ptr->psp_flags = psp_flags; + sp_ptr->ep = NULL; + + /* + * Take a reference on the EVD handle + */ + atomic_inc(&evd_ptr->evd_ref_count); + + /* Link it onto the IA */ + dapl_ia_link_psp(ia_ptr, sp_ptr); + + /* + * Set up a listener for a connection. Connections can arrive + * even before this call returns! + */ + sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; + sp_ptr->listening = TRUE; + + /* + * If we have a big number of tries and we still haven't + * found a service_ID we can use, bail out with an error, + * something is wrong! + */ + for (i = 0, sp_ptr->conn_qual = hint_conn_qual; i < 100000; i++) { + status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); + if (DAT_SUCCESS == status) + break; + else + sp_ptr->conn_qual++; + } + hint_conn_qual = sp_ptr->conn_qual + 1; + + if (status != DAT_SUCCESS) { + atomic_dec(&evd_ptr->evd_ref_count); + sp_ptr->evd = NULL; + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "dapl_psp_create cannot set up conn listener: %x\n", + status); + + goto bail; + } + + /* + * Return handle to the user + */ + *conn_qual = sp_ptr->conn_qual; + *psp = (struct dat_sp *) sp_ptr; + +bail: + return status; +} + +/* + * dapl_psp_create + * + * Create a persistent Public Service Point that can recieve multiple + * requests for connections and generate multiple connection request + * instances that wil be delivered to the specified Event Dispatcher + * in a notification event. + * + * Input: + * ia + * conn_qual + * evd + * psp_flags + * + * Output: + * psp + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_PARAMETER + * DAT_CONN_QUAL_IN_USE + * DAT_MODEL_NOT_SUPPORTED + */ +u32 dapl_psp_create(struct dat_ia *ia, DAT_CONN_QUAL conn_qual, + struct dat_evd *evd, enum dat_psp_flags psp_flags, + struct dat_sp **psp) +{ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; + boolean_t sp_found; + u32 status = DAT_SUCCESS; + + ia_ptr = (struct dapl_ia *)ia; + + if (!ia_ptr) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + if (!evd) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (psp == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); + goto bail; + } + + evd_ptr = (struct dapl_evd *)evd; + if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (psp_flags != DAT_PSP_CONSUMER_FLAG && + psp_flags != DAT_PSP_PROVIDER_FLAG) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); + goto bail; + } + + /* + * See if we have a quiescent listener to use for this PSP, else + * create one and set it listening + */ + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, TRUE); + sp_found = TRUE; + if (sp_ptr == NULL) { + /* Allocate PSP */ + sp_found = FALSE; + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); + if (sp_ptr == NULL) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } + } else if (sp_ptr->listening == TRUE) { + status = DAT_ERROR(DAT_CONN_QUAL_IN_USE, 0); + goto bail; + } + + /* + * Fill out the args for a PSP + */ + sp_ptr->ia = ia; + sp_ptr->conn_qual = conn_qual; + sp_ptr->evd = evd; + sp_ptr->psp_flags = psp_flags; + sp_ptr->ep = NULL; + + /* + * Take a reference on the EVD handle + */ + atomic_inc(&evd_ptr->evd_ref_count); + + /* + * Set up a listener for a connection. Connections can arrive + * even before this call returns! + */ + sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; + sp_ptr->listening = TRUE; + + /* + * If this is a new sp we need to add it to the IA queue, and set up + * a conn_listener. + */ + if (sp_found == FALSE) { + /* + * Link it onto the IA before enabling it to receive conn + * requests + */ + dapl_ia_link_psp(ia_ptr, sp_ptr); + + status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); + + if (status != DAT_SUCCESS) { + /* + * Have a problem setting up the connection, something + * wrong! Decrements the EVD refcount & release it. + */ + atomic_dec(&evd_ptr->evd_ref_count); + sp_ptr->evd = NULL; + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_CM, "dapl_psp_create " + "setup_conn_listener failed: %x\n", + status); + + goto bail; + } + } + + /* + * Return handle to the user + */ + *psp = (struct dat_sp *) sp_ptr; + +bail: + return status; +} + +/* + * dapl_psp_free + * + * Destroy a specific instance of a Service Point. + * + * Input: + * psp + * + * Output: + * none + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_PARAMETER + */ +u32 dapl_psp_free(struct dat_sp *psp) +{ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + DAPL_SP_STATE save_state; + u32 status = DAT_SUCCESS; + + sp_ptr = (struct dapl_sp *)psp; + /* + * Verify handle + */ + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp); + + if (!sp_ptr) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); + goto bail; + } + + ia_ptr = sp_ptr->common.owner_ia; + /* + * Remove the connection listener if it has been established + * and there are no current connections in progress. + * If we defer removing the sp it becomes something of a zombie + * container until the last connection is disconnected, after + * which it will be cleaned up. + */ + spin_lock_irqsave(&sp_ptr->common.lock, sp_ptr->common.flags); + + sp_ptr->listening = FALSE; + + /* + * Release reference on EVD. If an error was encountered in a previous + * free the evd will be NULL + */ + if (sp_ptr->evd) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd)-> + evd_ref_count); + sp_ptr->evd = NULL; + } + + /* + * Release the base resource if there are no outstanding + * connections; else the last disconnect on this PSP will free it + * up. The PSP is used to contain CR records for each connection, + * which contain information necessary to disconnect. + */ + dapl_dbg_log(DAPL_DBG_TYPE_CM, + ">>> dapl_psp_free: state %d cr_list_count %d\n", + sp_ptr->state, sp_ptr->cr_list_count); + if ((sp_ptr->state == DAPL_SP_STATE_PSP_LISTENING || + sp_ptr->state == DAPL_SP_STATE_PSP_PENDING) && + sp_ptr->cr_list_count == 0) { + save_state = sp_ptr->state; + sp_ptr->state = DAPL_SP_STATE_FREE; + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + + status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); + if (status != DAT_SUCCESS) { + /* revert to entry state on error */ + sp_ptr->state = save_state; + goto bail; + } + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + } else { + /* + * The PSP is now in the pending state, where it will sit until + * the last connection terminates or the app uses the same + * ServiceID again, which will reactivate it. + */ + sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; + spin_unlock_irqrestore(&sp_ptr->common.lock, + sp_ptr->common.flags); + dapl_dbg_log(DAPL_DBG_TYPE_CM, + ">>> dapl_psp_free: PSP PENDING\n"); + } + +bail: + return status; +} + +u32 dapl_psp_query(struct dat_sp *psp, struct dat_psp_param *psp_param) +{ + struct dapl_sp *sp_ptr; + u32 status; + + if (!psp || ((struct dapl_sp *)psp)->listening != TRUE) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); + goto bail; + } + + if (NULL == psp_param) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto bail; + } + + sp_ptr = (struct dapl_sp *)psp; + + psp_param->ia = sp_ptr->ia; + psp_param->conn_qual = sp_ptr->conn_qual; + psp_param->evd = sp_ptr->evd; + psp_param->psp_flags = sp_ptr->psp_flags; + + status = DAT_SUCCESS; + +bail: + return status; +} Index: linux-kernel/dat-provider/dapl_sp_util.h =================================================================== --- linux-kernel/dat-provider/dapl_sp_util.h (revision 2585) +++ linux-kernel/dat-provider/dapl_sp_util.h (working copy) @@ -1,48 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_PSP_UTIL_H -#define DAPL_PSP_UTIL_H - -struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp); - -void dapl_sp_dealloc(struct dapl_sp *sp_ptr); - -void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr); - -struct dapl_cr *dapl_sp_search_cr(struct dapl_sp *sp_ptr, - struct dapl_cm_id *ib_cm_handle); - -void dapl_sp_remove_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr); - -void dapl_sp_remove_ep(struct dapl_ep *ep_ptr); - -#endif /* DAPL_PSP_UTIL_H */ Index: linux-kernel/dat-provider/dapl_sp.h =================================================================== --- linux-kernel/dat-provider/dapl_sp.h (revision 2585) +++ linux-kernel/dat-provider/dapl_sp.h (working copy) @@ -29,12 +29,8 @@ * $Id$ */ -#ifndef DAPL_PSP_UTIL_H -#define DAPL_PSP_UTIL_H - -struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp); - -void dapl_sp_dealloc(struct dapl_sp *sp_ptr); +#ifndef DAPL_SP_H +#define DAPL_SP_H void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr); @@ -45,4 +41,4 @@ void dapl_sp_remove_cr(struct dapl_sp *s void dapl_sp_remove_ep(struct dapl_ep *ep_ptr); -#endif /* DAPL_PSP_UTIL_H */ +#endif /* DAPL_SP_H */ Index: linux-kernel/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel/patches/alt_dat_provider_makefile (revision 2585) +++ linux-kernel/patches/alt_dat_provider_makefile (working copy) @@ -22,13 +22,10 @@ PROVIDER_MODULES := \ dapl_lmr \ dapl_mr_util \ dapl_provider \ - dapl_sp_util \ - dapl_psp \ + dapl_sp \ dapl_pz \ dapl_ring_buffer_util \ dapl_rmr \ - dapl_rsp \ - dapl_sp_util \ dapl_srq \ dapl_util From bjordan.ics at gmail.com Fri Jun 10 10:32:16 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Fri, 10 Jun 2005 13:32:16 -0400 Subject: [openib-general] IPoIB: Waiting for ib0 to become free In-Reply-To: <527jh21bbh.fsf@topspin.com> References: <1118411513.4476.20.camel@localhost.localdomain> <527jh21bbh.fsf@topspin.com> Message-ID: <78d18e2050610103212bf0a0d@mail.gmail.com> On 6/10/05, Roland Dreier wrote: > Hal> dev/core.c netdev_wait_allrefs says: * Any protocol or device > Hal> that holds a reference should register * for netdevice > Hal> notification, and cleanup and put back the * reference if > Hal> they receive an UNREGISTER event. > > Hal> Is it correct that IPoIB does not need to register for these > Hal> events ? If it is, then this must be something else which is > Hal> using the IPoIB driver causing the reference count to be > Hal> incremented but not handling these events. Troy, any idea on > Hal> how to recreate this ? > > IPoIB doesn't need to handle these events, since it is the one doing > the unregistering. > I've seen this problem before with several network drivers. Is there any possibility that any skb's have been leaked (lost by the driver, never freed), or that the driver is still holding onto any skb's (waiting in a send queue)? The refcount of the skb's dst is decremented when an skb is freed. I also sometimes see this problem with particular kernels with our own IPoIB implementation when IPoIB is the only network interface. In this instance, I'm suspicious that there may be someone else in the stack that is still holding a reference on an skb that our driver has freed, preventing the refcount from being decremented. -- Bill Jordan SilverStorm Technologies From rep.nop at aon.at Fri Jun 10 11:50:55 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Fri, 10 Jun 2005 20:50:55 +0200 Subject: [openib-general] [osm] segfault in libibumad Message-ID: <20050610185054.GF5412@aon.at> Hi, I have no ib drivers loaded, starting opensm fails with a segfault. I'm not sure if userspace is supposed to work without sysfs. What do you think? In osm_vendor_init(), i'd set int r = -1, n_cas = -1; and would say else\nif ((n_cas = umad_get_cas_names. Also, in umad_get_cas_names() i guess only freeing namelist if scandir did not return <0 may be better.. (gdb) run Starting program: /opt/infiniband/ib/bin/opensm ------------------------------------------------- OpenSM Rev:openib-1.0.0 Command Line Arguments: Log File: /var/log/osm.log ------------------------------------------------- warn: [5900] umad_init: can't read ABI version from /sys/class/infiniband_mad/abi_version (No such file or directory): is ib_umad module loaded? Program received signal SIGSEGV, Segmentation fault. 0x400b8735 in free () from /lib/tls/libc.so.6 (gdb) bt #0 0x400b8735 in free () from /lib/tls/libc.so.6 #1 0x4002235a in umad_get_cas_names (cas=0x80acc1c, max=32) at umad.c:513 #2 0x4001afe1 in osm_vendor_init (p_vend=0x80acb88, p_log=0x80a979c, timeout=100) at osm_vendor_ibumad.c:418 #3 0x4001b0ba in osm_vendor_new (p_log=0x80a979c, timeout=100) at osm_vendor_ibumad.c:452 #4 0x0805d2e4 in osm_opensm_init (p_osm=0x80a8520, p_opt=0xbfb5dc40) at osm_opensm.c:234 #5 0x0804ca19 in main (argc=1, argv=0xbfb5de14) at main.c:632 -- Bernhard From halr at voltaire.com Fri Jun 10 12:22:32 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 Jun 2005 15:22:32 -0400 Subject: [openib-general] [osm] segfault in libibumad In-Reply-To: <20050610185054.GF5412@aon.at> References: <20050610185054.GF5412@aon.at> Message-ID: <1118431352.4476.30.camel@localhost.localdomain> On Fri, 2005-06-10 at 14:50, Bernhard Fischer wrote: > Hi, > > I have no ib drivers loaded, starting opensm fails with a segfault. > > I'm not sure if userspace is supposed to work without sysfs. What do > you think? It shouldn't segv... > In osm_vendor_init(), i'd set int r = -1, n_cas = -1; and would say > else\nif ((n_cas = umad_get_cas_names. > Also, in umad_get_cas_names() i guess only freeing namelist if > scandir did not return <0 may be better.. Yes to both. > (gdb) run > Starting program: /opt/infiniband/ib/bin/opensm > ------------------------------------------------- > OpenSM Rev:openib-1.0.0 > Command Line Arguments: > Log File: /var/log/osm.log > ------------------------------------------------- > warn: [5900] umad_init: can't read ABI version from > /sys/class/infiniband_mad/abi_version (No such file or directory): is > ib_umad module loaded? > > Program received signal SIGSEGV, Segmentation fault. > 0x400b8735 in free () from /lib/tls/libc.so.6 > (gdb) bt > #0 0x400b8735 in free () from /lib/tls/libc.so.6 > #1 0x4002235a in umad_get_cas_names (cas=0x80acc1c, max=32) at > umad.c:513 > #2 0x4001afe1 in osm_vendor_init (p_vend=0x80acb88, p_log=0x80a979c, > timeout=100) at osm_vendor_ibumad.c:418 > #3 0x4001b0ba in osm_vendor_new (p_log=0x80a979c, timeout=100) > at osm_vendor_ibumad.c:452 > #4 0x0805d2e4 in osm_opensm_init (p_osm=0x80a8520, p_opt=0xbfb5dc40) > at osm_opensm.c:234 > #5 0x0804ca19 in main (argc=1, argv=0xbfb5de14) at main.c:632 Can you try this patch ? Thanks. -- Hal Index: libibumad/src/umad.c =================================================================== --- libibumad/src/umad.c (revision 2580) +++ libibumad/src/umad.c (working copy) @@ -511,7 +511,8 @@ DEBUG("return 1 ca"); j = 1; } - free(namelist); + if (n >= 0) + free(namelist); return j; } Index: osm/libvendor/osm_vendor_ibumad.c =================================================================== --- osm/libvendor/osm_vendor_ibumad.c (revision 2580) +++ osm/libvendor/osm_vendor_ibumad.c (working copy) @@ -405,7 +405,8 @@ "osm_vendor_init: umad_get_cas_names failed\n"); r = n_cas; goto Exit; - } + } else + r = n_cas = -1; p_vend->ca_count = n_cas; p_vend->mtbl.max = OSM_UMAD_MAX_PENDING; From rep.nop at aon.at Fri Jun 10 12:34:10 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Fri, 10 Jun 2005 21:34:10 +0200 Subject: [openib-general] [sdp] debug strings Message-ID: <20050610193410.GG5412@aon.at> Hi, Upon loading ib_sdp with debugging enabled, i see: INIT: : SDP module load. with attached patchlet, i see the (to me) more appealing string ib_sdp INIT: SDP module load. - remove duplicate ': ' and prefix messages from sdp with ib_sdp like ib_mthca does. - fix odd "INIT: : INIT: SDP module unload." debug message. thank you, -------------- next part -------------- diff -X excl -rduNp gen2.2551.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c gen2.2551/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c --- gen2.2551.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c 2005-05-17 07:36:11.000000000 +0000 +++ gen2.2551/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c 2005-06-10 19:07:49.000000000 +0000 @@ -79,7 +79,7 @@ MODULE_PARM_DESC(recv_post_max, "Set the receive completion queue size."); module_param(recv_buff_max, int, 0); -MODULE_PARM_DESC(recv_buff_max, +MODULE_PARM_DESC(recv_buff_max, "Set the maximum number of receives buffered."); module_param(send_post_max, int, 0); @@ -91,7 +91,7 @@ MODULE_PARM_DESC(send_buff_max, "Set the maximum number of sends buffered."); module_param(send_usig_max, int, 0); -MODULE_PARM_DESC(send_usig_max, +MODULE_PARM_DESC(send_usig_max, "Set the maximum consecutive unsignalled send events."); module_param(sdp_debug_level, int, 0); @@ -1571,7 +1571,7 @@ error_proc: */ static void __exit sdp_exit(void) { - sdp_dbg_init("INIT: SDP module unload."); + sdp_dbg_init("SDP module unload."); /* * unregister */ diff -X excl -rduNp gen2.2551.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_proto.h gen2.2551/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_proto.h --- gen2.2551.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_proto.h 2005-05-17 07:36:11.000000000 +0000 +++ gen2.2551/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_proto.h 2005-06-10 16:26:10.000000000 +0000 @@ -471,7 +471,7 @@ extern int sdp_debug_level; #define sdp_dbg_out(level, type, format, arg...) \ do { \ if (!(level > sdp_debug_level)) { \ - printk("<%d>%s: " format "\n", \ + printk("<%d>ib_sdp %s: " format "\n", \ level, type, ## arg); \ } \ } while (0) @@ -516,7 +516,7 @@ extern int sdp_debug_level; #define sdp_dbg_init(format, arg...) do { } while (0) #else #define sdp_dbg_init(format, arg...) \ - sdp_dbg_out(__SDP_DEBUG_INIT, "INIT: ", format, ## arg) + sdp_dbg_out(__SDP_DEBUG_INIT, "INIT", format, ## arg) #endif #if __SDP_DEBUG_LEVEL < __SDP_DEBUG_WARN From rep.nop at aon.at Fri Jun 10 12:46:14 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Fri, 10 Jun 2005 21:46:14 +0200 Subject: [openib-general] [osm] segfault in libibumad In-Reply-To: <1118431352.4476.30.camel@localhost.localdomain> References: <20050610185054.GF5412@aon.at> <1118431352.4476.30.camel@localhost.localdomain> Message-ID: <20050610194614.GH5412@aon.at> On Fri, Jun 10, 2005 at 03:22:32PM -0400, Hal Rosenstock wrote: >On Fri, 2005-06-10 at 14:50, Bernhard Fischer wrote: >> I'm not sure if userspace is supposed to work without sysfs. What do >> you think? > >It shouldn't segv... Yes, but should it work? I don't know if its possible and implemented (in the long run) to get to and query the respective API versions in order not to rely on sysfs. >> In osm_vendor_init(), i'd set int r = -1, n_cas = -1; and would say >> else\nif ((n_cas = umad_get_cas_names. >> Also, in umad_get_cas_names() i guess only freeing namelist if >> scandir did not return <0 may be better.. > >Yes to both. >Can you try this patch ? Thanks. >- free(namelist); >+ if (n >= 0) >+ free(namelist); yes, should do. >- } >+ } else >+ r = n_cas = -1; yes, but why not just initialize r and n_cas to -1? thank you, -- Bernhard From tduffy at sun.com Fri Jun 10 13:57:14 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 10 Jun 2005 13:57:14 -0700 Subject: [openib-general] [PATCH] kDAPL: convert the ep list to linux native Message-ID: <1118437034.29601.6.camel@duffman> This patch converts the ep list to the linux native linked list structure. Let me know what you think. Signed-off-by: Tom Duffy Index: linux-kernel-ll2/dat-provider/dapl_ia.c =================================================================== --- linux-kernel-ll2/dat-provider/dapl_ia.c (revision 2585) +++ linux-kernel-ll2/dat-provider/dapl_ia.c (working copy) @@ -63,7 +63,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat ia->async_error_evd = NULL; ia->cleanup_async_error_evd = FALSE; dapl_llist_init_entry(&ia->hca_ia_list_entry); - dapl_llist_init_head(&ia->ep_list_head); + INIT_LIST_HEAD(&ia->ep_list); dapl_llist_init_head(&ia->lmr_list_head); dapl_llist_init_head(&ia->rmr_list_head); dapl_llist_init_head(&ia->pz_list_head); @@ -104,7 +104,7 @@ bail: u32 dapl_ia_abrupt_close(struct dapl_ia *ia) { u32 dat_status = DAT_SUCCESS; - struct dapl_ep *ep, *next_ep; + struct dapl_ep *ep; struct dapl_lmr *lmr, *next_lmr; struct dapl_rmr *rmr, *next_rmr; struct dapl_pz *pz, *next_pz; @@ -151,12 +151,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia sp = next_sp; } - ep = (dapl_llist_is_empty(&ia->ep_list_head) - ? NULL : dapl_llist_peek_head(&ia->ep_list_head)); - while (ep != NULL) { - next_ep = dapl_llist_next_entry(&ia->ep_list_head, - &ep->common. - ia_list_entry); + list_for_each_entry(ep, &ia->ep_list, list) { /* * Issue a disconnect if the EP needs it */ @@ -181,7 +176,6 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_free(%p) returns %x\n", ep, dat_status); - ep = next_ep; } lmr = (dapl_llist_is_empty(&ia->lmr_list_head) @@ -332,7 +326,7 @@ u32 dapl_ia_graceful_close(struct dapl_i if (!dapl_llist_is_empty(&ia->rmr_list_head) || !dapl_llist_is_empty(&ia->rsp_list_head) || - !dapl_llist_is_empty(&ia->ep_list_head) || + !list_empty(&ia->ep_list) || !dapl_llist_is_empty(&ia->lmr_list_head) || !dapl_llist_is_empty(&ia->psp_list_head) || !dapl_llist_is_empty(&ia->pz_list_head)) { @@ -427,7 +421,7 @@ void dapl_ia_free(struct dapl_ia *ia) dapl_os_assert(ia->async_error_evd == NULL); dapl_os_assert(dapl_llist_is_empty(&ia->lmr_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia->rmr_list_head)); - dapl_os_assert(dapl_llist_is_empty(&ia->ep_list_head)); + dapl_os_assert(list_empty(&ia->ep_list)); dapl_os_assert(dapl_llist_is_empty(&ia->evd_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia->psp_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia->rsp_list_head)); @@ -444,19 +438,17 @@ void dapl_ia_free(struct dapl_ia *ia) void dapl_ia_link_ep(struct dapl_ia *ia, struct dapl_ep *ep) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_add_head(&ia->ep_list_head, - &ep->common.ia_list_entry, ep); + list_add(&ep->list, &ia->ep_list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } /* - * Remove an ep from the ia info structure + * Remove an ep from the ia structure */ void dapl_ia_unlink_ep(struct dapl_ia *ia, struct dapl_ep *ep) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_remove_entry(&ia->ep_list_head, - &ep->common.ia_list_entry); + list_del(&ep->list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } Index: linux-kernel-ll2/dat-provider/dapl.h =================================================================== --- linux-kernel-ll2/dat-provider/dapl.h (revision 2585) +++ linux-kernel-ll2/dat-provider/dapl.h (working copy) @@ -34,6 +34,8 @@ #ifndef DAPL_H #define DAPL_H +#include + #include #include "dapl_util.h" @@ -152,7 +154,7 @@ struct dapl_ia { boolean_t cleanup_async_error_evd; struct dapl_llist_entry hca_ia_list_entry; /* HCAs list of IAs */ - struct dapl_llist_entry *ep_list_head; /* EP queue */ + struct list_head ep_list; /* EP queue */ struct dapl_llist_entry *lmr_list_head; /* LMR queue */ struct dapl_llist_entry *rmr_list_head; /* RMR queue */ struct dapl_llist_entry *pz_list_head; /* PZ queue */ @@ -195,6 +197,7 @@ struct dapl_evd { struct dapl_ep { struct dat_ep ep; struct dapl_common common; + struct list_head list; /* What the DAT Consumer asked for */ struct dat_ep_param param; From tduffy at sun.com Fri Jun 10 13:59:49 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 10 Jun 2005 13:59:49 -0700 Subject: [openib-general] [osm] segfault in libibumad In-Reply-To: <20050610194614.GH5412@aon.at> References: <20050610185054.GF5412@aon.at> <1118431352.4476.30.camel@localhost.localdomain> <20050610194614.GH5412@aon.at> Message-ID: <1118437189.29601.8.camel@duffman> On Fri, 2005-06-10 at 21:46 +0200, Bernhard Fischer wrote: > Yes, but should it work? I don't know if its possible and implemented > (in the long run) to get to and query the respective API versions in > order not to rely on sysfs. Do you have a system without sysfs? Is there a reason why you would not? The behavior /could/ be that if it didn't find the sysfs entry, opensm would continue on its merry way and just assume the API was correct. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Fri Jun 10 14:05:29 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 Jun 2005 17:05:29 -0400 Subject: [openib-general] [osm] segfault in libibumad In-Reply-To: <20050610194614.GH5412@aon.at> References: <20050610185054.GF5412@aon.at> <1118431352.4476.30.camel@localhost.localdomain> <20050610194614.GH5412@aon.at> Message-ID: <1118437529.4476.47.camel@localhost.localdomain> On Fri, 2005-06-10 at 15:46, Bernhard Fischer wrote: > On Fri, Jun 10, 2005 at 03:22:32PM -0400, Hal Rosenstock wrote: > >On Fri, 2005-06-10 at 14:50, Bernhard Fischer wrote: > > >> I'm not sure if userspace is supposed to work without sysfs. What do > >> you think? > > > >It shouldn't segv... > > Yes, but should it work? I don't know if its possible and implemented > (in the long run) to get to and query the respective API versions in > order not to rely on sysfs. It is not designed to run without sysfs. sysfs is a requirement for OpenIB. > >> In osm_vendor_init(), i'd set int r = -1, n_cas = -1; and would say > >> else\nif ((n_cas = umad_get_cas_names. > >> Also, in umad_get_cas_names() i guess only freeing namelist if > >> scandir did not return <0 may be better.. > > > >Yes to both. > > >Can you try this patch ? Thanks. > > >- free(namelist); > >+ if (n >= 0) > >+ free(namelist); > yes, should do. > > >- } > >+ } else > >+ r = n_cas = -1; > yes, but why not just initialize r and n_cas to -1? OK. Did you try it ? -- Hal From iod00d at hp.com Fri Jun 10 14:59:52 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 10 Jun 2005 14:59:52 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050610003352.GE12434@esmail.cup.hp.com> References: <20050610003352.GE12434@esmail.cup.hp.com> Message-ID: <20050610215952.GK10802@esmail.cup.hp.com> On Thu, Jun 09, 2005 at 05:33:52PM -0700, Grant Grundler wrote: > I think adding such a calibration to get_cpu_mhz() > so we can warn if /proc/cpuinfo data doesn't agree with > gettimeofday() and get_cycles(). The appended patch prints the Mhz reported in cpuinfo and the Mhz calculated from get_cycles()/gettimeofday(). (Sorry - this is a complete diff including previous changes that haven't been committed yet.) Conclusion: Maybe add another parameter to rdma_bw() to switch the source of the CPU Mhz. rdma_bw should continue using cpuinfo to report BW by default. The problem is gettimeofday() will also measure any other activity that occurs while the test is running. So it is more likely to under report bandwidth. Theoretically rdma_bw can discard extreme get_cycle() readings caused by noise on the test machine. (We don't do that today). My r2600 says: /proc/cpuinfo reports 1500 Mhz Calculated speed is 1492 Mhz (130150841056 cycles / 871849 usec) This is derived from 199. MHz FSB speed * (15:2 ratio) == 1492.5 Mhz. The "Calculated speed" is really 1492.81 Mhz. That's < 0.5% error. I'm told only HP 1 and 2-way boxes exhibit this "feature" - other boxes calculate the actual ITC speed just like the patch does and report that in /proc/cpuinfo. I just checked my rx4640 (also ZX1 chipset) and it seems to report the proper speed: cpu MHz : 1299.762145 itc MHz : 1299.762145 While closer, rmda_bw still doesn't seem to agree: /proc/cpuinfo reports 1299.76 Mhz Calculated speed is 1295 Mhz (118700953436 cycles / 916265 usec) On this machine, I'm going to trust the firmware is more accurate than my user space test. hth, grant Index: rdma_bw.c =================================================================== --- rdma_bw.c (revision 2572) +++ rdma_bw.c (working copy) @@ -61,10 +61,6 @@ static int page_size; -struct report_options { - int cycles; /* report delta's in cycles, not microsec's */ -}; - struct pingpong_context { struct ibv_context *context; struct ibv_pd *pd; @@ -422,19 +418,14 @@ printf(" -t, --tx-depth= size of tx queue (default 100)\n"); printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); - printf(" -C, --report-cycles report times in cpu cycle units (default seconds)\n"); - printf(" -H, --report-histogram print out all results (default print summary only)\n"); - printf(" -U, --report-unsorted (implies -H) print out unsorted results (default sorted)\n"); } -static void print_report(struct report_options * options, - unsigned int iters, double size, int duplex, +static void print_report(unsigned int iters, int size, int duplex, cycles_t *tposted, cycles_t *tcompleted) { double cycles_to_units; - double tsize; /* Transferred size, in megabytes */ + unsigned long tsize; /* Transferred size, in megabytes */ int i, j; - const char* units; int opt_posted = 0, opt_completed = 0; cycles_t opt_delta; cycles_t t; @@ -453,19 +444,21 @@ } } - if (options->cycles) { - cycles_to_units = 1; - units = "cycles"; - } else { - cycles_to_units = get_cpu_mhz() * 1000000; - units = "sec"; - } + cycles_to_units = get_cpu_mhz() * 1000000; tsize = duplex ? 2 : 1; - tsize = tsize * size / 0x100000; + tsize = tsize * size / 1024; - printf("Bandwidth peak (#%d to #%d): %g MByte/%s\n", opt_posted, opt_completed, tsize * cycles_to_units / opt_delta, units); - printf("Bandwidth average: %g MByte/%s\n", tsize * iters * cycles_to_units / (tcompleted[iters - 1] - tposted[0]), units); + printf("Bandwidth peak (#%d to #%d): %g MB/sec\n", + opt_posted, opt_completed, + tsize * cycles_to_units / opt_delta / 1024); + printf("Bandwidth average: %g MB/sec\n", + tsize * iters * cycles_to_units / (tcompleted[iters - 1] - tposted[0]) / 1024); + + printf("Service Demand peak (#%d to #%d): %ld cycles/KB\n", + opt_posted, opt_completed, opt_delta/tsize); + printf("Service Demand Avg : %ld cycles/KB\n", + (tcompleted[iters - 1] - tposted[0])/(tsize * iters)); } @@ -478,16 +471,18 @@ struct pingpong_dest *rem_dest; char *ib_devname = NULL; char *servername = NULL; + struct timeval tod_start, tod_end; + cycles_t tsc_start, tsc_end; int port = 18515; int ib_port = 1; - int size = 1; + int size = 4 * 1024; int tx_depth = 100; int iters = 1000; int scnt, ccnt; int sockfd; int duplex = 0; struct ibv_qp *qp; - struct report_options report = {}; + unsigned long long tsc_total, tod_total; cycles_t *tposted; cycles_t *tcompleted; @@ -504,11 +499,10 @@ { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "bidirectional", .has_arg = 0, .val = 'b' }, - { .name = "report-cycles", .has_arg = 0, .val = 'C' }, { 0 } }; - c = getopt_long(argc, argv, "p:d:i:s:n:t:bC", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:s:n:t:b", long_options, NULL); if (c == -1) break; @@ -565,10 +559,6 @@ duplex = 1; break; - case 'C': - report.cycles = 1; - break; - default: usage(argv[0]); return 1; @@ -701,10 +691,15 @@ } /* Done with setup. Start the test. */ - + if (gettimeofday(&tod_start, NULL)) { + perror("gettimeofday"); + return 1; + } + tsc_start = get_cycles(); + while (scnt < iters || ccnt < iters) { - while (scnt < iters && scnt - ccnt < tx_depth) { + while ((scnt < iters) && (scnt - ccnt < tx_depth)) { struct ibv_send_wr *bad_wr; tposted[scnt] = get_cycles(); @@ -742,6 +737,12 @@ } } + if (gettimeofday(&tod_end, NULL)) { + perror("gettimeofday"); + return 1; + } + tsc_end = get_cycles(); + if (servername) { rem_dest = pp_client_exch_dest(sockfd, &my_dest); } else { @@ -751,8 +752,22 @@ write(sockfd, "done", sizeof "done"); close(sockfd); - print_report(&report, iters, size, duplex, tposted, tcompleted); + /* sum total cycles into tsc_start */ + tsc_total = tposted[0] - tsc_start; + for (ccnt = 0; ccnt < iters; ccnt++) + tsc_total += tcompleted[ccnt] - tposted[ccnt]; + tsc_total += tsc_end - tcompleted[ccnt-1]; + tod_total = (unsigned long long) (tod_end.tv_sec - tod_start.tv_sec) * 1000000 + + (tod_end.tv_usec - tod_start.tv_usec); + + printf("/proc/cpuinfo reports %5g Mhz\n", get_cpu_mhz()); + + printf("Calculated speed is %5Ld Mhz", tsc_total / tod_total / 100); + printf(" (%Ld cycles / %Ld usec)\n", tsc_total, tod_total); + + print_report(iters, size, duplex, tposted, tcompleted); + free(tposted); free(tcompleted); return 0; From rep.nop at aon.at Fri Jun 10 15:04:49 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Sat, 11 Jun 2005 00:04:49 +0200 Subject: [openib-general] [osm] segfault in libibumad In-Reply-To: <1118437189.29601.8.camel@duffman> References: <20050610185054.GF5412@aon.at> <1118431352.4476.30.camel@localhost.localdomain> <20050610194614.GH5412@aon.at> <1118437189.29601.8.camel@duffman> Message-ID: <20050610220449.GI5412@aon.at> On Fri, Jun 10, 2005 at 01:59:49PM -0700, Tom Duffy wrote: >On Fri, 2005-06-10 at 21:46 +0200, Bernhard Fischer wrote: >> Yes, but should it work? I don't know if its possible and implemented >> (in the long run) to get to and query the respective API versions in >> order not to rely on sysfs. > >Do you have a system without sysfs? Is there a reason why you would >not? For normal small systems, i do not see a requirements for sysfs. Artifical use case (i don't own HCAs nor euid==0 access to them): /me builds a well-connected firewall/router/whatever. I don't want to waste RAM/ROM for sysfs or sysfs support. I do, however, agree that normal nodes will have sysfs support enabled in the mid/short run. > >The behavior /could/ be that if it didn't find the sysfs entry, opensm >would continue on its merry way and just assume the API was correct. As Hal expressed, sysfs is a requirement for OpenIB; so be it. -- Bernhard From rep.nop at aon.at Fri Jun 10 15:59:33 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Sat, 11 Jun 2005 00:59:33 +0200 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050610215952.GK10802@esmail.cup.hp.com> References: <20050610003352.GE12434@esmail.cup.hp.com> <20050610215952.GK10802@esmail.cup.hp.com> Message-ID: <20050610225933.GJ5412@aon.at> On Fri, Jun 10, 2005 at 02:59:52PM -0700, Grant Grundler wrote: >On Thu, Jun 09, 2005 at 05:33:52PM -0700, Grant Grundler wrote: >(Sorry - this is a complete diff including previous >changes that haven't been committed yet.) np, as long as you use at least 4*page_size and reflect this in usage() :) >- int size = 1; >+ int size = 4 * 1024; cheers, -- Bernhard From jlentini at netapp.com Fri Jun 10 16:05:12 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 10 Jun 2005 19:05:12 -0400 (EDT) Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <1118424237.26791.4.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35C74@mtvex01.mtv.mtl.com> <1118424237.26791.4.camel@duffman> Message-ID: Committed in revision 2588 except for the item below. On Fri, 10 Jun 2005, Tom Duffy wrote: tduffy> Index: linux-kernel/dat-provider/dapl_sp.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp.h (revision 2585) tduffy> +++ linux-kernel/dat-provider/dapl_sp.h (working copy) tduffy> @@ -29,12 +29,8 @@ tduffy> * $Id$ tduffy> */ tduffy> tduffy> -#ifndef DAPL_PSP_UTIL_H tduffy> -#define DAPL_PSP_UTIL_H tduffy> - tduffy> -struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp); tduffy> - tduffy> -void dapl_sp_dealloc(struct dapl_sp *sp_ptr); I didn't remove this declaration from the header because dapl_sp_dealloc is being called by dapl_get_sp_ep tduffy> +#ifndef DAPL_SP_H tduffy> +#define DAPL_SP_H tduffy> tduffy> void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr); tduffy> tduffy> @@ -45,4 +41,4 @@ void dapl_sp_remove_cr(struct dapl_sp *s tduffy> tduffy> void dapl_sp_remove_ep(struct dapl_ep *ep_ptr); tduffy> tduffy> -#endif /* DAPL_PSP_UTIL_H */ tduffy> +#endif /* DAPL_SP_H */ tduffy> Index: linux-kernel/patches/alt_dat_provider_makefile From iod00d at hp.com Fri Jun 10 16:31:48 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 10 Jun 2005 16:31:48 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050610225933.GJ5412@aon.at> References: <20050610003352.GE12434@esmail.cup.hp.com> <20050610215952.GK10802@esmail.cup.hp.com> <20050610225933.GJ5412@aon.at> Message-ID: <20050610233148.GA12389@esmail.cup.hp.com> On Sat, Jun 11, 2005 at 12:59:33AM +0200, Bernhard Fischer wrote: > np, as long as you use at least 4*page_size and reflect this in > usage() :) > > >- int size = 1; > >+ int size = 4 * 1024; heh - I'll fix that. :^) thanks, grant From berny.f at aon.at Fri Jun 10 16:35:52 2005 From: berny.f at aon.at (Bernhard Fischer) Date: Sat, 11 Jun 2005 01:35:52 +0200 Subject: [openib-general] [mthca] debug strings In-Reply-To: <20050610193410.GG5412@aon.at> References: <20050610193410.GG5412@aon.at> Message-ID: <20050610233552.GK5412@aon.at> On Fri, Jun 10, 2005 at 09:34:10PM +0200, Bernhard Fischer wrote: >Hi, >- remove duplicate ': ' and prefix messages from sdp with ib_sdp > like ib_mthca does. hm. when loading mthca i see: ib_mthca: Mellanox InfiniBand HCA driver v0.06-pre (November 8, 2004) ib_mthca: Initializing Mellanox Technologies ..... ib_mthca 0000:04:00.0: FW version 000400060002, max commands 64 ... which is a bit inconsistent. No functional obj-code changes. - peruse DRV_NAME - trim ':' from PFX to be consistent with dev_{info,warn,err} output. - whitespace in struct mthca_profile: replace spaces with tabs to be consistent with other member of that struct wrt spacing style. - mthca_dev_lim() remove tab (was that ment to be a reminder?) - add apparently missing newlines to stings emitted by dev_err. - add note whom to contact if old FW is found. In mthca_map_eq_regs(), i take it that it is of no interrest to know who of "..dev->fw.arbel.eq_"arm_base|eq_set_ci_base") + "(0|4) did trigger? Memfree vs. not would be obvious from previous msgs. Same in mthca_tune_pci for cap messages (which don't fit into width=80 terminals) find_mgm(): 'if (0)\n\t\tmthca_dbg(dev, "Hash for..' should not rely on dead code elimination but HEAVY_DEBUG. Haven't looked for other occurances of that, yet. Will you? thank you, -- Bernhard -------------- next part -------------- diff -X excl -rduNp gen2.2551.oorig/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_dev.h gen2.2551/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_dev.h --- gen2.2551.oorig/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_dev.h 2005-05-27 11:06:10.000000000 +0200 +++ gen2.2551/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_dev.h 2005-06-11 01:27:31.000000000 +0200 @@ -45,7 +45,7 @@ #include "mthca_doorbell.h" #define DRV_NAME "ib_mthca" -#define PFX DRV_NAME ": " +#define PFX DRV_NAME " " #define DRV_VERSION "0.06-pre" #define DRV_RELDATE "November 8, 2004" diff -X excl -rduNp gen2.2551.oorig/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_main.c gen2.2551/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_main.c --- gen2.2551.oorig/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_main.c 2005-05-27 11:06:10.000000000 +0200 +++ gen2.2551/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_main.c 2005-06-10 23:14:43.000000000 +0200 @@ -69,7 +69,7 @@ MODULE_PARM_DESC(msi, "attempt to use MS #endif /* CONFIG_PCI_MSI */ static const char mthca_version[] __devinitdata = - "ib_mthca: Mellanox InfiniBand HCA driver v" + DRV_NAME " Mellanox InfiniBand HCA driver v" DRV_VERSION " (" DRV_RELDATE ")\n"; static struct mthca_profile default_profile = { @@ -154,11 +154,11 @@ static int __devinit mthca_dev_lim(struc return -ENODEV; } - mdev->limits.num_ports = dev_lim->num_ports; + mdev->limits.num_ports = dev_lim->num_ports; mdev->limits.vl_cap = dev_lim->max_vl; mdev->limits.mtu_cap = dev_lim->max_mtu; - mdev->limits.gid_table_len = dev_lim->max_gids; - mdev->limits.pkey_table_len = dev_lim->max_pkeys; + mdev->limits.gid_table_len = dev_lim->max_gids; + mdev->limits.pkey_table_len = dev_lim->max_pkeys; mdev->limits.local_ca_ack_delay = dev_lim->local_ca_ack_delay; mdev->limits.max_sg = dev_lim->max_sg; mdev->limits.reserved_qps = dev_lim->reserved_qps; @@ -188,7 +188,7 @@ static int __devinit mthca_dev_lim(struc if (dev_lim->flags & DEV_LIM_FLAG_BAD_QKEY_CNTR) mdev->device_cap_flags |= IB_DEVICE_BAD_QKEY_CNTR; - + if (dev_lim->flags & DEV_LIM_FLAG_RAW_MULTI) mdev->device_cap_flags |= IB_DEVICE_RAW_MULTI; @@ -927,13 +927,13 @@ static int __devinit mthca_init_one(stru */ if (!(pci_resource_flags(pdev, 0) & IORESOURCE_MEM) || pci_resource_len(pdev, 0) != 1 << 20) { - dev_err(&pdev->dev, "Missing DCS, aborting."); + dev_err(&pdev->dev, "Missing DCS, aborting.\n"); err = -ENODEV; goto err_disable_pdev; } if (!(pci_resource_flags(pdev, 2) & IORESOURCE_MEM) || pci_resource_len(pdev, 2) != 1 << 23) { - dev_err(&pdev->dev, "Missing UAR, aborting."); + dev_err(&pdev->dev, "Missing UAR, aborting.\n"); err = -ENODEV; goto err_disable_pdev; } @@ -1032,6 +1032,7 @@ static int __devinit mthca_init_one(stru (int) (mthca_hca_table[id->driver_data].latest_fw >> 16) & 0xffff, (int) (mthca_hca_table[id->driver_data].latest_fw & 0xffff)); mthca_warn(mdev, "If you have problems, try updating your HCA FW.\n"); + mthca_warn(mdev, "Contact your HW supplier for current FW.\n"); } err = mthca_setup_hca(mdev); @@ -1163,7 +1164,7 @@ static struct pci_device_id mthca_pci_ta MODULE_DEVICE_TABLE(pci, mthca_pci_table); static struct pci_driver mthca_driver = { - .name = "ib_mthca", + .name = DRV_NAME, .id_table = mthca_pci_table, .probe = mthca_init_one, .remove = __devexit_p(mthca_remove_one) From rep.nop at aon.at Fri Jun 10 16:52:51 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Sat, 11 Jun 2005 01:52:51 +0200 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <52psuv14md.fsf@topspin.com> References: <52psuv14md.fsf@topspin.com> Message-ID: <20050610235251.GL5412@aon.at> On Thu, Jun 09, 2005 at 04:40:10PM -0700, Roland Dreier wrote: > Shirley> I used cyclets_to_units() get Bandwidth peak (#0 to > Shirley> #972): 1852.72 MByte/sec > > Shirley> Which is far away from the 1X (2.5Gbit/s) throughtput. > >Actually it's not that far away -- the 2.5 Gbit/sec is after 8b/10b >coding, so the raw data rate is really 2 Gbit/sec. Getting > 92% link >utilization is pretty good considering packet header overhead and so >on. BTW I'm assuming "MByte/sec" is a typo for "Mbit/sec". I may be wrong and OT, but from what i got, nowadays, http://kerneltrap.org/node/340 ymmv, -- Bernhard From iod00d at hp.com Fri Jun 10 21:27:27 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 10 Jun 2005 21:27:27 -0700 Subject: [openib-general] cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050610215952.GK10802@esmail.cup.hp.com> References: <20050610003352.GE12434@esmail.cup.hp.com> <20050610215952.GK10802@esmail.cup.hp.com> Message-ID: <20050611042727.GB12389@esmail.cup.hp.com> On Fri, Jun 10, 2005 at 02:59:52PM -0700, Grant Grundler wrote: > My r2600 says: > /proc/cpuinfo reports 1500 Mhz > Calculated speed is 1492 Mhz (130150841056 cycles / 871849 usec) Sorry, I wasn't paying attention...the 1492 number is wrong and I finally figured out why. > This is derived from 199. MHz FSB speed * (15:2 ratio) == 1492.5 Mhz. > The "Calculated speed" is really 1492.81 Mhz. That's < 0.5% error. "" is probably 199.9 or so - very close to spec (200Mhz). The error is well below %0.1. > I just checked my rx4640 (also ZX1 chipset) > and it seems to report the proper speed: > cpu MHz : 1299.762145 > itc MHz : 1299.762145 In this case, the FSB is 199.96 Mhz. > On this machine, I'm going to trust the firmware is more accurate > than my user space test. Definitely. > + printf("Calculated speed is %5Ld Mhz", tsc_total / tod_total / 100); Sorry - I should have figured out why I needed the "/ 100". It directly corresponds to tx_depth (default 100). And it was a big, fat, hairy clue something wasn't right. I just didn't work out the math at the time though it bothered me a bit becuase it didn't make sense. The posted/completed get_cycles() measurements overlap. Ergo with big "-n 10000" I was getting a number that was 100x too big. My second clue was that smaller -n would have "bigger" errors. Patch below sums the get_cycle measurements correctly (I think) and results look like: /proc/cpuinfo reports 1500.00 Mhz Calculated speed is 1500.02 Mhz (642797211 cycles / 428525 usec) or with different parameters: /proc/cpuinfo reports 1500.00 Mhz Calculated speed is 1499.99 Mhz (129930481 cycles / 86621 usec) (129930481/86621 is 1499.98823...) Patch below adds "-g" (gettimeofday) and "-M=" parameters in case someone's firmware is less accurate than HP's. And to keep Bernhard happy, fixes usage() output to match 16KB default. :^) apologies, grant Index: rdma_bw.c =================================================================== --- rdma_bw.c (revision 2572) +++ rdma_bw.c (working copy) @@ -58,13 +58,10 @@ #include "get_clock.h" #define PINGPONG_RDMA_WRID 3 +#define MILLION 1000000 static int page_size; -struct report_options { - int cycles; /* report delta's in cycles, not microsec's */ -}; - struct pingpong_context { struct ibv_context *context; struct ibv_pd *pd; @@ -418,23 +415,20 @@ static void usage(const char *argv0) printf(" -p, --port= listen on/connect to port (default 18515)\n"); printf(" -d, --ib-dev= use IB device (default first device found)\n"); printf(" -i, --ib-port= use port of IB device (default 1)\n"); - printf(" -s, --size= size of message to exchange (default 1)\n"); + printf(" -s, --size= size of message to exchange (default 16KB)\n"); printf(" -t, --tx-depth= size of tx queue (default 100)\n"); printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -b, --bidirectional measure bidirectional bandwidth (default unidirectional)\n"); - printf(" -C, --report-cycles report times in cpu cycle units (default seconds)\n"); - printf(" -H, --report-histogram print out all results (default print summary only)\n"); - printf(" -U, --report-unsorted (implies -H) print out unsorted results (default sorted)\n"); + printf(" -g, --gettimeofday Use gettimeofday() to calibrate Mhz (default /proc/cpuinfo)\n"); + printf(" -M, --mhz= Use for Mhz\n"); } -static void print_report(struct report_options * options, - unsigned int iters, double size, int duplex, +static void print_report(double cycles_to_units, unsigned int iters, + int size, int duplex, cycles_t *tposted, cycles_t *tcompleted) { - double cycles_to_units; - double tsize; /* Transferred size, in megabytes */ + unsigned long tsize; /* Transferred size, in megabytes */ int i, j; - const char* units; int opt_posted = 0, opt_completed = 0; cycles_t opt_delta; cycles_t t; @@ -453,19 +447,20 @@ static void print_report(struct report_o } } - if (options->cycles) { - cycles_to_units = 1; - units = "cycles"; - } else { - cycles_to_units = get_cpu_mhz() * 1000000; - units = "sec"; - } tsize = duplex ? 2 : 1; - tsize = tsize * size / 0x100000; + tsize = tsize * size / 1024; /* convert to KB */ - printf("Bandwidth peak (#%d to #%d): %g MByte/%s\n", opt_posted, opt_completed, tsize * cycles_to_units / opt_delta, units); - printf("Bandwidth average: %g MByte/%s\n", tsize * iters * cycles_to_units / (tcompleted[iters - 1] - tposted[0]), units); + printf("Bandwidth peak (#%d to #%d): %'.2f MB/sec\n", + opt_posted, opt_completed, + tsize * cycles_to_units / opt_delta / 1024); + printf("Bandwidth average: %'.2f MB/sec\n", + tsize * iters * cycles_to_units / (tcompleted[iters - 1] - tposted[0]) / 1024); + + printf("Service Demand peak (#%d to #%d): %ld cycles/KB\n", + opt_posted, opt_completed, opt_delta/tsize); + printf("Service Demand Avg : %ld cycles/KB\n", + (tcompleted[iters - 1] - tposted[0])/(tsize * iters)); } @@ -478,16 +473,21 @@ int main(int argc, char *argv[]) struct pingpong_dest *rem_dest; char *ib_devname = NULL; char *servername = NULL; + struct timeval tod_start, tod_end; + cycles_t tsc_start, tsc_end; int port = 18515; int ib_port = 1; - int size = 1; + int size = 16 * 1024; int tx_depth = 100; int iters = 1000; int scnt, ccnt; int sockfd; int duplex = 0; + int use_gettod = 0; struct ibv_qp *qp; - struct report_options report = {}; + unsigned long long tsc_total, tod_total; + double cycles_to_units = get_cpu_mhz() * MILLION; + double mhz = 0; cycles_t *tposted; cycles_t *tcompleted; @@ -503,12 +503,13 @@ int main(int argc, char *argv[]) { .name = "size", .has_arg = 1, .val = 's' }, { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, + { .name = "mhz", .has_arg = 1, .val = 'M' }, { .name = "bidirectional", .has_arg = 0, .val = 'b' }, - { .name = "report-cycles", .has_arg = 0, .val = 'C' }, + { .name = "gettimeofday", .has_arg = 0, .val = 'g' }, { 0 } }; - c = getopt_long(argc, argv, "p:d:i:s:n:t:bC", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:s:n:t:M:bg", long_options, NULL); if (c == -1) break; @@ -565,8 +566,16 @@ int main(int argc, char *argv[]) duplex = 1; break; - case 'C': - report.cycles = 1; + case 'g': + use_gettod = 1; /* i.e. don't use /proc/cpuinfo */ + break; + + case 'M': + mhz = strtod(optarg, NULL); + if (mhz < 1) { + usage(argv[0]); + return 1; + } break; default: @@ -701,10 +710,15 @@ int main(int argc, char *argv[]) } /* Done with setup. Start the test. */ - + if (gettimeofday(&tod_start, NULL)) { + perror("gettimeofday"); + return 1; + } + tsc_start = get_cycles(); + while (scnt < iters || ccnt < iters) { - while (scnt < iters && scnt - ccnt < tx_depth) { + while ((scnt < iters) && (scnt - ccnt < tx_depth)) { struct ibv_send_wr *bad_wr; tposted[scnt] = get_cycles(); @@ -742,6 +756,12 @@ int main(int argc, char *argv[]) } } + if (gettimeofday(&tod_end, NULL)) { + perror("gettimeofday"); + return 1; + } + tsc_end = get_cycles(); + if (servername) { rem_dest = pp_client_exch_dest(sockfd, &my_dest); } else { @@ -751,7 +771,38 @@ int main(int argc, char *argv[]) write(sockfd, "done", sizeof "done"); close(sockfd); - print_report(&report, iters, size, duplex, tposted, tcompleted); + /* sum total cycles into tsc_start */ + tsc_total = tposted[0] - tsc_start; /* prelude */ + for (ccnt = 0; ccnt < iters-1; ccnt++) /* time in loop sending */ + tsc_total += tposted[ccnt+1] - tposted[ccnt]; + + /* time until last send completed */ + tsc_total += tcompleted[ccnt] - tposted[ccnt]; + tsc_total += tsc_end - tcompleted[ccnt]; /* epilogue */ + + tod_total = (unsigned long long) (tod_end.tv_sec - tod_start.tv_sec) * + MILLION + + (tod_end.tv_usec - tod_start.tv_usec); + + + printf("/proc/cpuinfo reports %'.2f Mhz\n", cycles_to_units/MILLION); + printf("Calculated speed is %'.2f Mhz", (double) tsc_total / tod_total); + printf(" (%Ld cycles / %Ld usec)\n", tsc_total, tod_total); + + if (mhz) { + printf("User specified %7g Mhz", mhz); + cycles_to_units = mhz * MILLION; + } else if (use_gettod) { + if (tod_total < 10000) + printf( "WARNING: Test was too short (%1.08g sec) " + " not using gettimeofday()\n" + " Use -b, more (-n) or bigger (-s) samples.\n" + , (double) tod_total / MILLION); + else + cycles_to_units = (double) tsc_total / tod_total * MILLION; + } + + print_report(cycles_to_units, iters, size, duplex, tposted, tcompleted); free(tposted); free(tcompleted); From hch at lst.de Sat Jun 11 02:20:40 2005 From: hch at lst.de (Christoph Hellwig) Date: Sat, 11 Jun 2005 11:20:40 +0200 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <469958e0050609135862ace6fc@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <469958e0050609135862ace6fc@mail.gmail.com> Message-ID: <20050611092040.GA18263@lst.de> On Thu, Jun 09, 2005 at 01:58:20PM -0700, Caitlin Bestler wrote: > The reason the method table is pointed at in each object is > to avoid having to run long chains of memory references to > get the method table. If you have the ep pointer you should > be able to get the method table directly, not have to fetch > the ia then the hca then the method table. Currently every dapl object embedds dapl_common which has a pointer to the dapl_ia. Now fixing that mess up a little every dat object can have a dat_ia pointer (without the utter _common braindamage) and there's one additional dereference, while saving one pointer per allocated object. From hch at lst.de Sat Jun 11 02:31:44 2005 From: hch at lst.de (Christoph Hellwig) Date: Sat, 11 Jun 2005 11:31:44 +0200 Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flags Message-ID: <20050611093144.GA18469@lst.de> in the linux kernel any provider must use the version it was compiled against, and must always be thread safe. Index: test/dapltest/test/dapl_test_util.c =================================================================== --- test/dapltest/test/dapl_test_util.c (revision 2588) +++ test/dapltest/test/dapl_test_util.c (working copy) @@ -76,12 +76,10 @@ struct sockaddr_in *ip_addr; DT_Tdep_PT_Printf (phead, "***** DAPL Characteristics *****\n"); - DT_Tdep_PT_Printf (phead, "Provider: %s Version %d.%d DAPL %d.%d\n", + DT_Tdep_PT_Printf (phead, "Provider: %s Version %d.%d\n", pt_ptr->provider_attr.provider_name, pt_ptr->provider_attr.provider_version_major, - pt_ptr->provider_attr.provider_version_minor, - pt_ptr->provider_attr.dat_version_major, - pt_ptr->provider_attr.dat_version_minor ); + pt_ptr->provider_attr.provider_version_minor); DT_Tdep_PT_Printf (phead, "Adapter: %s by %s Version %d.%d\n", pt_ptr->ia_attr.adapter_name, pt_ptr->ia_attr.vendor_name, Index: dat-provider/dapl_ia.c =================================================================== --- dat-provider/dapl_ia.c (revision 2588) +++ dat-provider/dapl_ia.c (working copy) @@ -881,15 +881,12 @@ DAT_NAME_MAX_LENGTH); provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; - provider_attr->dat_version_major = DAT_VERSION_MAJOR; - provider_attr->dat_version_minor = DAT_VERSION_MINOR; provider_attr->lmr_mem_types_supported = DAT_MEM_TYPE_VIRTUAL | DAT_MEM_TYPE_LMR; provider_attr->iov_ownership_on_return = DAT_IOV_CONSUMER; provider_attr->dat_qos_supported = DAT_QOS_BEST_EFFORT; provider_attr->completion_flags_supported = DAT_COMPLETION_DEFAULT_FLAG; - provider_attr->is_thread_safe = FALSE; provider_attr->max_private_data_size = IB_CM_REQ_PRIVATE_DATA_SIZE; provider_attr->supports_multipath = FALSE; provider_attr->ep_creator = DAT_PSP_CREATES_EP_NEVER; Index: dat-provider/dapl_provider.c =================================================================== --- dat-provider/dapl_provider.c (revision 2588) +++ dat-provider/dapl_provider.c (working copy) @@ -244,9 +244,6 @@ (size_t) (DAT_NAME_MAX_LENGTH - 2)); provider_info->ia_name[i] = 'a' + port - 1; /* IB ports are 1 based */ provider_info->ia_name[i+1] = '\0'; - provider_info->dat_version_major = DAT_VERSION_MAJOR; - provider_info->dat_version_minor = DAT_VERSION_MINOR; - provider_info->is_thread_safe = DAPL_THREADSAFE; } static void dapl_add_port(struct ib_device *device, u8 port) Index: dat/dictionary.c =================================================================== --- dat/dictionary.c (revision 2588) +++ dat/dictionary.c (working copy) @@ -212,10 +212,6 @@ BUG_ON(NULL == new_key); strncpy(new_key->ia_name, old_key->ia_name, DAT_NAME_MAX_LENGTH); - new_key->dat_version_major = old_key->dat_version_major; - new_key->dat_version_minor = old_key->dat_version_minor; - new_key->is_thread_safe = old_key->is_thread_safe; - return DAT_SUCCESS; } @@ -226,10 +222,7 @@ BUG_ON(NULL == b); if ((strlen(a->ia_name) == strlen(b->ia_name)) && - (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name))) - && (a->dat_version_major == b->dat_version_major) - && (a->dat_version_minor == b->dat_version_minor) - && (a->is_thread_safe == b->is_thread_safe)) + (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name)))) return TRUE; else return FALSE; Index: dat/api.c =================================================================== --- dat/api.c (revision 2588) +++ dat/api.c (working copy) @@ -349,9 +349,8 @@ } } -u32 dat_ia_openv(const char *name, int async_event_qlen, - struct dat_evd **async_event_handle, struct dat_ia **ia, - u32 dat_major, u32 dat_minor, boolean_t thread_safety) +u32 dat_ia_open(const char *name, int async_event_qlen, + struct dat_evd **async_event_handle, struct dat_ia **ia) { DAT_IA_OPEN_FUNC ia_open_func; struct dat_provider_info info; @@ -360,8 +359,7 @@ dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "%s(name:%s, async_qlen:%i, ver:%x.%x, thrd_safe:%x)\n", - __func__, name, async_event_qlen, dat_major, dat_minor, - thread_safety); + __func__, name, async_event_qlen); if (DAT_IS_BAD_POINTER(name)) return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); @@ -374,10 +372,6 @@ strncpy(info.ia_name, name, len); info.ia_name[len] = '\0'; - info.dat_version_major = dat_major; - info.dat_version_minor = dat_minor; - info.is_thread_safe = thread_safety; - status = dat_dr_provider_open(&info, &ia_open_func); if (status != DAT_SUCCESS) { dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, @@ -390,7 +384,7 @@ return ia_open_func(name, async_event_qlen, async_event_handle, ia); } -EXPORT_SYMBOL(dat_ia_openv); +EXPORT_SYMBOL(dat_ia_open); u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) { @@ -423,10 +417,6 @@ strncpy(info.ia_name, ia_name, len); info.ia_name[len] = '\0'; - info.dat_version_major = provider_attr.dat_version_major; - info.dat_version_minor = provider_attr.dat_version_minor; - info.is_thread_safe = provider_attr.is_thread_safe; - status = dat_dr_provider_close(&info); if (DAT_SUCCESS != status) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, @@ -472,10 +462,7 @@ { dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", - __func__, provider_info->ia_name, - provider_info->dat_version_major, - provider_info->dat_version_minor, - provider_info->is_thread_safe); + __func__, provider_info->ia_name); if (DAT_IS_BAD_POINTER(provider)) return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); Index: dat/Makefile =================================================================== --- dat/Makefile (revision 2588) +++ dat/Makefile (working copy) @@ -1,7 +1,6 @@ EXTRA_CFLAGS += \ - -Idrivers/dat \ - -Werror + -I$(obj) obj-$(CONFIG_DAT) += dat.o Index: dat/dat.h =================================================================== --- dat/dat.h (revision 2588) +++ dat/dat.h (working copy) @@ -33,10 +33,6 @@ #include #include -#define DAT_VERSION_MAJOR 1 -#define DAT_VERSION_MINOR 2 -#define DAT_THREADSAFE FALSE - /* * All return codes are actually a 3-way tuple: * @@ -894,9 +890,6 @@ struct dat_provider_info { char ia_name[DAT_NAME_MAX_LENGTH]; - u32 dat_version_major; - u32 dat_version_minor; - boolean_t is_thread_safe; }; struct dat_evd_param { @@ -1017,13 +1010,10 @@ char provider_name[DAT_NAME_MAX_LENGTH]; u32 provider_version_major; u32 provider_version_minor; - u32 dat_version_major; - u32 dat_version_minor; enum dat_mem_type lmr_mem_types_supported; enum dat_iov_ownership iov_ownership_on_return; enum dat_qos dat_qos_supported; enum dat_completion_flags completion_flags_supported; - boolean_t is_thread_safe; int max_private_data_size; boolean_t supports_multipath; enum dat_ep_creator_for_psp ep_creator; @@ -1361,16 +1351,10 @@ /* * DAT registry functions for consumers */ -extern u32 dat_ia_openv(const char *name, int async_event_qlen, +extern u32 dat_ia_open(const char *name, int async_event_qlen, struct dat_evd **async_event_handle, - struct dat_ia **ia, u32 dat_major, u32 dat_minor, - boolean_t thread_safety); + struct dat_ia **ia); -#define dat_ia_open(name, qlen, async_evd, ia) \ - dat_ia_openv((name), (qlen), (async_evd), (ia), \ - DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ - DAT_THREADSAFE) - extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); extern u32 dat_strerror(u32, const char **, const char **); From mst at mellanox.co.il Sun Jun 12 02:09:29 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 12 Jun 2005 12:09:29 +0300 Subject: [openib-general] Re: [PATCHv6] SDP: Allow SDP to compile on 2.6.12-rc6 In-Reply-To: <1118343092.32250.15.camel@duffman> References: <1118343092.32250.15.camel@duffman> Message-ID: <20050612090929.GF23303@mellanox.co.il> Quoting r. Tom Duffy : > Subject: [PATCHv6] SDP: Allow SDP to compile on 2.6.12-rc6 > > This updates the 2.6.12 SDP patch against the state change patch Libor > checked into r2577. > > ... > > Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c > =================================================================== > --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2577) > +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) > @@ -129,9 +129,9 @@ MODULE_PARM_DESC(sdp_debug_level, > */ > void sdp_inet_wake_send(struct sock *sk) > { > - struct sdp_opt *conn; > + struct sdp_opt *conn = sdp_sk(sk); > > - if (!sk || !(conn = SDP_GET_CONN(sk))) > + if (!sk) > return; > > if (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) && I didnt look at this in depth, but is it safe to call sdp_sk(sk) if sk may be NULL as the above seems to do? -- MST From Postmaster at mx.bigpond.com Sun Jun 12 03:15:27 2005 From: Postmaster at mx.bigpond.com (Mail Administrator) Date: Sun, 12 Jun 2005 10:15:27 +0000 Subject: [openib-general] Mail System Error - Returned Mail Message-ID: <20050612101527.OVXE7650.omta04sl.mx.bigpond.com@omta04sl> This Message was undeliverable due to the following reason: Each of the following recipients was rejected by a remote mail server. The reasons given by the server are included to help you determine why each recipient was rejected. Recipient: Reason: Recipient Rejected: Not accepting mail for this account : Account Inactive Please reply to if you feel this message to be in error. -------------- next part -------------- An embedded message was scrubbed... From: openib-general at openib.org Subject: Here is your documents. Date: Sun, 12 Jun 2005 18:14:38 +0800 Size: 49850 URL: From tduffy at sun.com Sun Jun 12 08:57:38 2005 From: tduffy at sun.com (Tom Duffy) Date: Sun, 12 Jun 2005 08:57:38 -0700 Subject: [openib-general] Re: [PATCHv6] SDP: Allow SDP to compile on 2.6.12-rc6 In-Reply-To: <20050612090929.GF23303@mellanox.co.il> References: <1118343092.32250.15.camel@duffman> <20050612090929.GF23303@mellanox.co.il> Message-ID: <1118591858.625.2.camel@duffman> On Sun, 2005-06-12 at 12:09 +0300, Michael S. Tsirkin wrote: > > Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c > > =================================================================== > > --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2577) > > +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) > > @@ -129,9 +129,9 @@ MODULE_PARM_DESC(sdp_debug_level, > > */ > > void sdp_inet_wake_send(struct sock *sk) > > { > > - struct sdp_opt *conn; > > + struct sdp_opt *conn = sdp_sk(sk); > > > > - if (!sk || !(conn = SDP_GET_CONN(sk))) > > + if (!sk) > > return; > > > > if (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) && > > I didnt look at this in depth, but is it safe to call sdp_sk(sk) if sk may > be NULL as the above seems to do? Yes, it should be safe as all sdp_sk() does is a cast. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Sun Jun 12 08:59:47 2005 From: tduffy at sun.com (Tom Duffy) Date: Sun, 12 Jun 2005 08:59:47 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <20050611092040.GA18263@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <469958e0050609135862ace6fc@mail.gmail.com> <20050611092040.GA18263@lst.de> Message-ID: <1118591987.625.6.camel@duffman> On Sat, 2005-06-11 at 11:20 +0200, Christoph Hellwig wrote: > Currently every dapl object embedds dapl_common which has a pointer > to the dapl_ia. Now fixing that mess up a little every dat object > can have a dat_ia pointer (without the utter _common braindamage) > and there's one additional dereference, while saving one pointer per > allocated object. The compiler should be smart enough to not make the access any more costly since they contain that struct and don't have a pointer to it, but I do agree that the _common structure is not ideal and sorta ugly. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Mon Jun 13 05:14:07 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 Jun 2005 08:14:07 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> Message-ID: <1118664517.4476.9585.camel@localhost.localdomain> On Wed, 2005-06-08 at 17:53, James Lentini wrote: > On Wed, 8 Jun 2005, Hal Rosenstock wrote: > > halr> On Wed, 2005-06-08 at 11:44, James Lentini wrote: > halr> > We interpreted the above to mean "give the connection protocol as > halr> > much time as it needs to establish a connection, but don't mask > halr> > errors (no path to the remove node, etc.)". For that reason we changed > halr> > the variable name to DAT_TIMEOUT_MAX. > halr> > halr> But if the REQ is lost, the timeout is really really long (longer than > halr> most will wait for an error). > > If a user doesn't want to wait DAT_TIMEOUT_MAX time, it can pass a > smaller amount of time to dat_ep_connect. Does this satisfy your > requirements? Is it the intended that the only way out is via user intervention (e.g. ctl-C) ? If one connection attempt (REQ) is made and it is lost, then there is no chance of it completing and the user needs to intervene. If that is the intended behavior, we are there. (This (lost REQ) can even occur when the timeout is non infinite too). An alternative (as Sean suggested) is to continually retry (at a periodicity below the supplied timeout) until the time period specified expires. That seems to be better (at least to me and Sean) in terms of handling the lost REQ case. As retries is not part of the API for connect, I would presume the implementor is free to what they want under the covers of dapl_ib_connect. -- Hal From libor at topspin.com Mon Jun 13 10:34:00 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 13 Jun 2005 10:34:00 -0700 Subject: [openib-general] [CM] possible problem with crossing DREQs. In-Reply-To: <1118408695.4477.10.camel@localhost.localdomain>; from halr@voltaire.com on Fri, Jun 10, 2005 at 09:04:55AM -0400 References: <1118408695.4477.10.camel@localhost.localdomain> Message-ID: <20050613103400.A4839@topspin.com> On Fri, Jun 10, 2005 at 09:04:55AM -0400, Hal Rosenstock wrote: > On Thu, 2005-06-09 at 15:47, Sean Hefty wrote: > > > I'm seeing an unusual problem when both halves of a connection > > >actively disconnect at the same time. Each connection peer issues > > >a DREQ at the same time, next each receive the DREQ and responds > > >with a DREP, and finally each connection gets a callback for the > > >transition to the idle state. However, at this point it appears > > >that each CM keeps retransmitting DREQ requests, which then seems > > >to interfere with new connection establishment. > > > > I think that I understand what's happening. Receiving the DREQ > > changed the state of the cm_id, but did not cancel the previous send. > > > > I'm actually out on vacation for a little over two weeks (and will > > be totally away from e-mail after Friday), but something > > like the patch below might fix the issue. (Note that I didn't test / > > compile this.) If it does work for you, feel free to commit it. > > This works for me. My test case is a little different. It is repeated > loopback kdapl quit tests but it does resolve the same problem. I am > comitting this change. Thanks. This change fixed my problem as well. I'm commiting the following patch which removes a SDP work around for the original problem. -Libor Index: infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- infiniband/ulp/sdp/sdp_rcvd.c (revision 2588) +++ infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -56,7 +56,6 @@ case SDP_CONN_ST_DIS_SENT_1: SDP_CONN_ST_SET(conn, SDP_CONN_ST_TIME_WAIT_1); band = POLL_HUP; -#if 0 /* crossing DREQs appear not to work... */ /* * Begin IB/CM disconnect */ @@ -72,7 +71,7 @@ if (result != -EPROTO) goto error; } -#endif + break; default: sdp_warn("Disconnect rcvd, unexpected state. <%d> <%04x>", From roland at topspin.com Mon Jun 13 10:34:17 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 13 Jun 2005 10:34:17 -0700 Subject: [openib-general] [mthca] debug strings In-Reply-To: <20050610233552.GK5412@aon.at> (Bernhard Fischer's message of "Sat, 11 Jun 2005 01:35:52 +0200") References: <20050610193410.GG5412@aon.at> <20050610233552.GK5412@aon.at> Message-ID: <52ekb6rwiu.fsf@topspin.com> Thanks, I applied most of these changes. Bernhard> In mthca_map_eq_regs(), i take it that it is of no Bernhard> interrest to know who of Bernhard> "..dev->fw.arbel.eq_"arm_base|eq_set_ci_base") + "(0|4) Bernhard> did trigger? Memfree vs. not would be obvious from Bernhard> previous msgs. Same in mthca_tune_pci for cap messages Bernhard> (which don't fit into width=80 terminals) I'm not sure I understood what you wrote here, but the messages in mthca_map_eq_regs() were indeed wrong -- I sloppily cut and pasted and didn't update the error messages. However I couldn't find anything wrong with the messages in mthca_tune_pci(). Bernhard> find_mgm(): 'if (0)\n\t\tmthca_dbg(dev, "Hash for..' Bernhard> should not rely on dead code elimination but Bernhard> HEAVY_DEBUG. Haven't looked for other occurances of Bernhard> that, yet. Will you? I'm sorry, I can't understand this paragraph. Could you rephrase it? Thanks, Roland From iod00d at hp.com Mon Jun 13 10:54:08 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 13 Jun 2005 10:54:08 -0700 Subject: [openib-general] [mthca] debug strings In-Reply-To: <52ekb6rwiu.fsf@topspin.com> References: <20050610193410.GG5412@aon.at> <20050610233552.GK5412@aon.at> <52ekb6rwiu.fsf@topspin.com> Message-ID: <20050613175408.GB20498@esmail.cup.hp.com> On Mon, Jun 13, 2005 at 10:34:17AM -0700, Roland Dreier wrote: > Bernhard> find_mgm(): 'if (0)\n\t\tmthca_dbg(dev, "Hash for..' > Bernhard> should not rely on dead code elimination but > Bernhard> HEAVY_DEBUG. Haven't looked for other occurances of > Bernhard> that, yet. Will you? > > I'm sorry, I can't understand this paragraph. Could you rephrase it? I think he meant to say this should be "#ifdef HEAVY_DEBUG" (or maybe CONFIG_HEAVY_DEBUG) instead of "if (0)". grant From libor at topspin.com Mon Jun 13 10:58:13 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 13 Jun 2005 10:58:13 -0700 Subject: [openib-general] [sdp] debug strings In-Reply-To: <20050610193410.GG5412@aon.at>; from rep.nop@aon.at on Fri, Jun 10, 2005 at 09:34:10PM +0200 References: <20050610193410.GG5412@aon.at> Message-ID: <20050613105813.B4839@topspin.com> On Fri, Jun 10, 2005 at 09:34:10PM +0200, Bernhard Fischer wrote: > Hi, > > Upon loading ib_sdp with debugging enabled, i see: > > - remove duplicate ': ' and prefix messages from sdp with ib_sdp > like ib_mthca does. > - fix odd "INIT: : INIT: SDP module unload." debug message. Thanks, applied and commited. -Libor From Thomas.Talpey at netapp.com Mon Jun 13 11:00:28 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Mon, 13 Jun 2005 14:00:28 -0400 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <20050611092040.GA18263@lst.de> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <469958e0050609135862ace6fc@mail.gmail.com> <20050611092040.GA18263@lst.de> Message-ID: <6.2.1.2.2.20050613135836.04c734c0@exnane01.nane.netapp.com> At 05:20 AM 6/11/2005, Christoph Hellwig wrote: >Currently every dapl object embedds dapl_common which has a pointer >to the dapl_ia. Now fixing that mess up a little every dat object >can have a dat_ia pointer (without the utter _common braindamage) >and there's one additional dereference, while saving one pointer per >allocated object. Not to defend the dapl_common stuff, but that extra pointer saves more than an indirection, it also avoids fetching another cacheline. Which, on the hotpath routines, can make a significant difference and is generally worth the extra single pointer storage. Tom. From tduffy at sun.com Mon Jun 13 12:17:21 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 13 Jun 2005 12:17:21 -0700 Subject: [openib-general] [PATCH] DAPL: update TODO Message-ID: <1118690241.1985.7.camel@duffman> Update the TODO so we don't lose track of needs. Signed-off-by: Tom Duffy Index: TODO =================================================================== --- TODO (revision 2588) +++ TODO (working copy) @@ -3,6 +3,8 @@ DAT *) change dat function to use errno return code as all linux kernel export functions +*) remove the dat_func() wrappers and have code call the functions from the + method table directly IB DAT PROVIDER @@ -23,3 +25,5 @@ IB DAT PROVIDER *) cache remote peer's address so that it does not need to be retrieved in dapl_ep_query. *) Replace dapl linked lists with Linux linked lists +*) don't check for NULL all over the place -- if somebody calls a function + wrong and breaks things, let them keep both halves From libor at topspin.com Mon Jun 13 12:37:47 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 13 Jun 2005 12:37:47 -0700 Subject: [openib-general] FW: [PATCH] ib_ucm: Change sys class name to match ib_uverbs and ib_mad In-Reply-To: <5D78D28F88822E4D8702BB9EEF1A4367140264@mercury.infiniconsys.com>; from bjordan@silverstorm.com on Fri, Jun 03, 2005 at 05:06:19PM -0400 References: <5D78D28F88822E4D8702BB9EEF1A4367140264@mercury.infiniconsys.com> Message-ID: <20050613123747.C4839@topspin.com> On Fri, Jun 03, 2005 at 05:06:19PM -0400, Jordan, Bill wrote: > > Patch to change sys class name of userspace cm to match > format of userpace verbs and userspace mad class names. Thanks, applied and committed. -Libor From jcarr at linuxmachines.com Mon Jun 13 14:12:28 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 13 Jun 2005 14:12:28 -0700 Subject: [openib-general] cm-id->context should be NULL for IB_CM_REQ_RECEIVED ? Message-ID: <42ADF6BC.204@linuxmachines.com> On the recieving end of a request, when the handler gets called and the cm-id gets passed to your handler (aka when the handler gets IB_CM_REQ_RECEIVED) it seems cm-id->context isn't null. Is there some useful data in there? If not, I think it makes more sense (and makes code easier to write to use the cm handler) if the context is set to NULL. Enjoy, Jeff PS: Might it also be good to up the DRV_VERSION to include the SVN rev so it's possible to tell which version of what was used via dmesg. (or is there a /sys/ or /proc/ interface for that info?) From jlentini at netapp.com Mon Jun 13 14:10:31 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 13 Jun 2005 17:10:31 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <1118437034.29601.6.camel@duffman> References: <1118437034.29601.6.camel@duffman> Message-ID: Tom. This looks good. I've checked it in as revision 2595. Eventually, we could move the list up into the dapl_common structure. The one oddity is that all the objects are put on the IA list except CRs which are placed on an SP. I think it would be worth reflecting this in the list's member name. Instead of calling it a ia_list_entry as it is now, a name like parent_list or owner_list would be more appropriate. james On Fri, 10 Jun 2005, Tom Duffy wrote: tduffy> This patch converts the ep list to the linux native linked list tduffy> structure. Let me know what you think. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-ll2/dat-provider/dapl_ia.c tduffy> =================================================================== tduffy> --- linux-kernel-ll2/dat-provider/dapl_ia.c (revision 2585) tduffy> +++ linux-kernel-ll2/dat-provider/dapl_ia.c (working copy) tduffy> @@ -63,7 +63,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat tduffy> ia->async_error_evd = NULL; tduffy> ia->cleanup_async_error_evd = FALSE; tduffy> dapl_llist_init_entry(&ia->hca_ia_list_entry); tduffy> - dapl_llist_init_head(&ia->ep_list_head); tduffy> + INIT_LIST_HEAD(&ia->ep_list); tduffy> dapl_llist_init_head(&ia->lmr_list_head); tduffy> dapl_llist_init_head(&ia->rmr_list_head); tduffy> dapl_llist_init_head(&ia->pz_list_head); tduffy> @@ -104,7 +104,7 @@ bail: tduffy> u32 dapl_ia_abrupt_close(struct dapl_ia *ia) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> - struct dapl_ep *ep, *next_ep; tduffy> + struct dapl_ep *ep; tduffy> struct dapl_lmr *lmr, *next_lmr; tduffy> struct dapl_rmr *rmr, *next_rmr; tduffy> struct dapl_pz *pz, *next_pz; tduffy> @@ -151,12 +151,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia tduffy> sp = next_sp; tduffy> } tduffy> tduffy> - ep = (dapl_llist_is_empty(&ia->ep_list_head) tduffy> - ? NULL : dapl_llist_peek_head(&ia->ep_list_head)); tduffy> - while (ep != NULL) { tduffy> - next_ep = dapl_llist_next_entry(&ia->ep_list_head, tduffy> - &ep->common. tduffy> - ia_list_entry); tduffy> + list_for_each_entry(ep, &ia->ep_list, list) { tduffy> /* tduffy> * Issue a disconnect if the EP needs it tduffy> */ tduffy> @@ -181,7 +176,6 @@ u32 dapl_ia_abrupt_close(struct dapl_ia tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): ep_free(%p) returns %x\n", tduffy> ep, dat_status); tduffy> - ep = next_ep; tduffy> } tduffy> tduffy> lmr = (dapl_llist_is_empty(&ia->lmr_list_head) tduffy> @@ -332,7 +326,7 @@ u32 dapl_ia_graceful_close(struct dapl_i tduffy> tduffy> if (!dapl_llist_is_empty(&ia->rmr_list_head) || tduffy> !dapl_llist_is_empty(&ia->rsp_list_head) || tduffy> - !dapl_llist_is_empty(&ia->ep_list_head) || tduffy> + !list_empty(&ia->ep_list) || tduffy> !dapl_llist_is_empty(&ia->lmr_list_head) || tduffy> !dapl_llist_is_empty(&ia->psp_list_head) || tduffy> !dapl_llist_is_empty(&ia->pz_list_head)) { tduffy> @@ -427,7 +421,7 @@ void dapl_ia_free(struct dapl_ia *ia) tduffy> dapl_os_assert(ia->async_error_evd == NULL); tduffy> dapl_os_assert(dapl_llist_is_empty(&ia->lmr_list_head)); tduffy> dapl_os_assert(dapl_llist_is_empty(&ia->rmr_list_head)); tduffy> - dapl_os_assert(dapl_llist_is_empty(&ia->ep_list_head)); tduffy> + dapl_os_assert(list_empty(&ia->ep_list)); tduffy> dapl_os_assert(dapl_llist_is_empty(&ia->evd_list_head)); tduffy> dapl_os_assert(dapl_llist_is_empty(&ia->psp_list_head)); tduffy> dapl_os_assert(dapl_llist_is_empty(&ia->rsp_list_head)); tduffy> @@ -444,19 +438,17 @@ void dapl_ia_free(struct dapl_ia *ia) tduffy> void dapl_ia_link_ep(struct dapl_ia *ia, struct dapl_ep *ep) tduffy> { tduffy> spin_lock_irqsave(&ia->common.lock, ia->common.flags); tduffy> - dapl_llist_add_head(&ia->ep_list_head, tduffy> - &ep->common.ia_list_entry, ep); tduffy> + list_add(&ep->list, &ia->ep_list); tduffy> spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); tduffy> } tduffy> tduffy> /* tduffy> - * Remove an ep from the ia info structure tduffy> + * Remove an ep from the ia structure tduffy> */ tduffy> void dapl_ia_unlink_ep(struct dapl_ia *ia, struct dapl_ep *ep) tduffy> { tduffy> spin_lock_irqsave(&ia->common.lock, ia->common.flags); tduffy> - dapl_llist_remove_entry(&ia->ep_list_head, tduffy> - &ep->common.ia_list_entry); tduffy> + list_del(&ep->list); tduffy> spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); tduffy> } tduffy> tduffy> Index: linux-kernel-ll2/dat-provider/dapl.h tduffy> =================================================================== tduffy> --- linux-kernel-ll2/dat-provider/dapl.h (revision 2585) tduffy> +++ linux-kernel-ll2/dat-provider/dapl.h (working copy) tduffy> @@ -34,6 +34,8 @@ tduffy> #ifndef DAPL_H tduffy> #define DAPL_H tduffy> tduffy> +#include tduffy> + tduffy> #include tduffy> tduffy> #include "dapl_util.h" tduffy> @@ -152,7 +154,7 @@ struct dapl_ia { tduffy> boolean_t cleanup_async_error_evd; tduffy> tduffy> struct dapl_llist_entry hca_ia_list_entry; /* HCAs list of IAs */ tduffy> - struct dapl_llist_entry *ep_list_head; /* EP queue */ tduffy> + struct list_head ep_list; /* EP queue */ tduffy> struct dapl_llist_entry *lmr_list_head; /* LMR queue */ tduffy> struct dapl_llist_entry *rmr_list_head; /* RMR queue */ tduffy> struct dapl_llist_entry *pz_list_head; /* PZ queue */ tduffy> @@ -195,6 +197,7 @@ struct dapl_evd { tduffy> struct dapl_ep { tduffy> struct dat_ep ep; tduffy> struct dapl_common common; tduffy> + struct list_head list; tduffy> /* What the DAT Consumer asked for */ tduffy> struct dat_ep_param param; tduffy> tduffy> From hch at lst.de Mon Jun 13 14:12:26 2005 From: hch at lst.de (Christoph Hellwig) Date: Mon, 13 Jun 2005 23:12:26 +0200 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: References: <1118437034.29601.6.camel@duffman> Message-ID: <20050613211226.GA31086@lst.de> On Mon, Jun 13, 2005 at 05:10:31PM -0400, James Lentini wrote: > > Tom. > > This looks good. I've checked it in as revision 2595. > > Eventually, we could move the list up into the dapl_common structure. > The one oddity is that all the objects are put on the IA list except > CRs which are placed on an SP. I think it would be worth reflecting > this in the list's member name. Instead of calling it a ia_list_entry > as it is now, a name like parent_list or owner_list would be more > appropriate. Umm, no. Please try to get rid of dapl_common instead of adding to it. From roland at topspin.com Mon Jun 13 14:12:42 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 13 Jun 2005 14:12:42 -0700 Subject: [openib-general] cm-id->context should be NULL for IB_CM_REQ_RECEIVED ? In-Reply-To: <42ADF6BC.204@linuxmachines.com> (Jeff Carr's message of "Mon, 13 Jun 2005 14:12:28 -0700") References: <42ADF6BC.204@linuxmachines.com> Message-ID: <52y89eot9x.fsf@topspin.com> Jeff> On the recieving end of a request, when the handler gets Jeff> called and the cm-id gets passed to your handler (aka when Jeff> the handler gets IB_CM_REQ_RECEIVED) it seems cm-id->context Jeff> isn't null. Is there some useful data in there? If not, I Jeff> think it makes more sense (and makes code easier to write to Jeff> use the cm handler) if the context is set to NULL. I think the context gets populated from the context of the corresponding cm_id that was passed to ib_cm_listen(). - R. From tduffy at sun.com Mon Jun 13 14:35:30 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 13 Jun 2005 14:35:30 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: References: <1118437034.29601.6.camel@duffman> Message-ID: <1118698530.9114.16.camel@duffman> On Mon, 2005-06-13 at 17:10 -0400, James Lentini wrote: > This looks good. I've checked it in as revision 2595. Thanks. > Eventually, we could move the list up into the dapl_common structure. Gross. I agree with Christoph. > The one oddity is that all the objects are put on the IA list except > CRs which are placed on an SP. I think it would be worth reflecting > this in the list's member name. Instead of calling it a ia_list_entry > as it is now, a name like parent_list or owner_list would be more > appropriate. Eventually, wouldn't we get rid of ia_list_entry? Cause each struct has its own list entry (name "list", which seems to be the convention), and the IA has the heads of each list. Your wording is a bit confusing cause you say, "all the objects are put on the IA list". Shouldn't that read, "each of the objects is put on its respective list owned by the IA" or some such? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jcarr at linuxmachines.com Mon Jun 13 14:51:57 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 13 Jun 2005 14:51:57 -0700 Subject: [openib-general] cm-id->context should be NULL for IB_CM_REQ_RECEIVED ? In-Reply-To: <52y89eot9x.fsf@topspin.com> References: <42ADF6BC.204@linuxmachines.com> <52y89eot9x.fsf@topspin.com> Message-ID: <42ADFFFD.4010406@linuxmachines.com> On 06/13/05 14:12, Roland Dreier wrote: > Jeff> On the recieving end of a request, when the handler gets > Jeff> called and the cm-id gets passed to your handler (aka when > Jeff> the handler gets IB_CM_REQ_RECEIVED) it seems cm-id->context > Jeff> isn't null. Is there some useful data in there? If not, I > Jeff> think it makes more sense (and makes code easier to write to > Jeff> use the cm handler) if the context is set to NULL. > > I think the context gets populated from the context of the > corresponding cm_id that was passed to ib_cm_listen(). OK, thanks. I checked and it does look like that is what is happening. I'll set the context for the cm_id I pass into ib_cm_listen to null. Assuming of course, that the cm_id context for that one doesn't matter. Jeff From roland at topspin.com Mon Jun 13 14:54:08 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 13 Jun 2005 14:54:08 -0700 Subject: [openib-general] cm-id->context should be NULL for IB_CM_REQ_RECEIVED ? In-Reply-To: <42ADFFFD.4010406@linuxmachines.com> (Jeff Carr's message of "Mon, 13 Jun 2005 14:51:57 -0700") References: <42ADF6BC.204@linuxmachines.com> <52y89eot9x.fsf@topspin.com> <42ADFFFD.4010406@linuxmachines.com> Message-ID: <521x75q5xb.fsf@topspin.com> Jeff> OK, thanks. I checked and it does look like that is what is Jeff> happening. I'll set the context for the cm_id I pass into Jeff> ib_cm_listen to null. Assuming of course, that the cm_id Jeff> context for that one doesn't matter. The context is just an opaque cookie that the CM will pass back to you well it calls you back. So if you don't need any context in your REQ received callback, it's fine to use a NULL listen context. - R. From jlentini at netapp.com Mon Jun 13 14:54:24 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 13 Jun 2005 17:54:24 -0400 (EDT) Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <1118591987.625.6.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <469958e0050609135862ace6fc@mail.gmail.com> <20050611092040.GA18263@lst.de> <1118591987.625.6.camel@duffman> Message-ID: On Sun, 12 Jun 2005, Tom Duffy wrote: > On Sat, 2005-06-11 at 11:20 +0200, Christoph Hellwig wrote: >> Currently every dapl object embedds dapl_common which has a pointer >> to the dapl_ia. Now fixing that mess up a little every dat object >> can have a dat_ia pointer (without the utter _common braindamage) >> and there's one additional dereference, while saving one pointer per >> allocated object. > > The compiler should be smart enough to not make the access any more > costly since they contain that struct and don't have a pointer to it, I believe the "one additional dereference" Christoph was referring to occurs if we implement his proposal and define dat objects as follows: struct dat_ia { struct dat_provider *provider; union dat_context context; } struct dat_foo { struct dat_ia *ia; union dat_context context; } > but I do agree that the _common structure is not ideal and sorta ugly. I like it on the basis that if gathers up the common structure members in one place, hence making maintenance of these fields easier in the future. Granted, there are only two of them, so losing it wouldn't be a big deal. > > -tduffy > From ftillier at silverstorm.com Mon Jun 13 14:56:04 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Mon, 13 Jun 2005 14:56:04 -0700 Subject: [openib-general] cm-id->context should be NULL forIB_CM_REQ_RECEIVED ? In-Reply-To: <42ADFFFD.4010406@linuxmachines.com> Message-ID: <000001c57062$b301c230$9c5aa8c0@infiniconsys.com> > From: Jeff Carr [mailto:jcarr at linuxmachines.com] > Sent: Monday, June 13, 2005 2:52 PM > > On 06/13/05 14:12, Roland Dreier wrote: > > Jeff> On the recieving end of a request, when the handler gets > > Jeff> called and the cm-id gets passed to your handler (aka when > > Jeff> the handler gets IB_CM_REQ_RECEIVED) it seems cm-id->context > > Jeff> isn't null. Is there some useful data in there? If not, I > > Jeff> think it makes more sense (and makes code easier to write to > > Jeff> use the cm handler) if the context is set to NULL. > > > > I think the context gets populated from the context of the > > corresponding cm_id that was passed to ib_cm_listen(). > > OK, thanks. I checked and it does look like that is what is happening. > I'll set the context for the cm_id I pass into ib_cm_listen to null. > Assuming of course, that the cm_id context for that one doesn't matter. The CM doesn't interpret the user's context ever. It is just there for the benefit of the user. If your app doesn't need the context in the REQ callback, then you can set it to NULL. Out of curiosity, if you don't set the listen's context, how do you figure out which listen received the callback? Does DAPL not care? - Fab From jlentini at netapp.com Mon Jun 13 15:03:56 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 13 Jun 2005 18:03:56 -0400 (EDT) Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <469958e0050609135862ace6fc@mail.gmail.com> <20050611092040.GA18263@lst.de> <1118591987.625.6.camel@duffman> Message-ID: There is precedent for including a pointer to the function pointer table in each top level object. The OpenIB stack does this; each IB object (ib_pd, ib_qp, ib_mr, etc.) contain a pointer to the ib_device structure. Of course I know that just because it was done before doesn't make it right. james On Mon, 13 Jun 2005, James Lentini wrote: > > > On Sun, 12 Jun 2005, Tom Duffy wrote: > >> On Sat, 2005-06-11 at 11:20 +0200, Christoph Hellwig wrote: >>> Currently every dapl object embedds dapl_common which has a pointer >>> to the dapl_ia. Now fixing that mess up a little every dat object >>> can have a dat_ia pointer (without the utter _common braindamage) >>> and there's one additional dereference, while saving one pointer per >>> allocated object. >> >> The compiler should be smart enough to not make the access any more >> costly since they contain that struct and don't have a pointer to it, > > I believe the "one additional dereference" Christoph was referring to occurs > if we implement his proposal and define dat objects as follows: > > struct dat_ia { > struct dat_provider *provider; > union dat_context context; > } > > struct dat_foo { > struct dat_ia *ia; > union dat_context context; > } > >> but I do agree that the _common structure is not ideal and sorta ugly. > > I like it on the basis that if gathers up the common structure members in one > place, hence making maintenance of these fields easier in the future. > Granted, there are only two of them, so losing it wouldn't be a big deal. > >> >> -tduffy >> > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From caitlin.bestler at gmail.com Mon Jun 13 15:06:19 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Mon, 13 Jun 2005 15:06:19 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <469958e0050609135862ace6fc@mail.gmail.com> <20050611092040.GA18263@lst.de> <1118591987.625.6.camel@duffman> Message-ID: <469958e00506131506137bd164@mail.gmail.com> On 6/13/05, James Lentini wrote: > > but I do agree that the _common structure is not ideal and sorta ugly. > > I like it on the basis that if gathers up the common structure members > in one place, hence making maintenance of these fields easier in the > future. Granted, there are only two of them, so losing it wouldn't be > a big deal. > What's important is that enables dapl objects to be handled polymorphically. Internally there are many times when a utility routine only needs to know that something is a dapl object, not which one it is. The alternative is to have some other mechanism to ensure that the shared fields are defined compatibly in all sub-types. This is doable, but declaring a common header is generally the simpler and more self-documenting method of implementing a parent type with multiple sub-types in plain C. The external requirements for polymorphic dapl handles are fewer: get/set consumer handle (which ismostly be for user mode programs anyway) and some methods apply to either a PSP or RSP. > > > > -tduffy > > > From jcarr at linuxmachines.com Mon Jun 13 15:13:32 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 13 Jun 2005 15:13:32 -0700 Subject: [openib-general] cm-id->context should be NULL for IB_CM_REQ_RECEIVED ? In-Reply-To: <521x75q5xb.fsf@topspin.com> References: <42ADF6BC.204@linuxmachines.com> <52y89eot9x.fsf@topspin.com> <42ADFFFD.4010406@linuxmachines.com> <521x75q5xb.fsf@topspin.com> Message-ID: <42AE050C.7060103@linuxmachines.com> On 06/13/05 14:54, Roland Dreier wrote: > Jeff> OK, thanks. I checked and it does look like that is what is > Jeff> happening. I'll set the context for the cm_id I pass into > Jeff> ib_cm_listen to null. Assuming of course, that the cm_id > Jeff> context for that one doesn't matter. > > The context is just an opaque cookie that the CM will pass back to you > well it calls you back. So if you don't need any context in your REQ > received callback, it's fine to use a NULL listen context. > > - R. > I see the error in my code now. Sorry about that... I wasn't passing in NULL to the cm_id I was listening on. User error, Jeff From jcarr at linuxmachines.com Mon Jun 13 15:20:51 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 13 Jun 2005 15:20:51 -0700 Subject: [openib-general] cm-id->context should be NULL forIB_CM_REQ_RECEIVED ? In-Reply-To: <000001c57062$b301c230$9c5aa8c0@infiniconsys.com> References: <000001c57062$b301c230$9c5aa8c0@infiniconsys.com> Message-ID: <42AE06C3.7030406@linuxmachines.com> On 06/13/05 14:56, Fab Tillier wrote: >>From: Jeff Carr [mailto:jcarr at linuxmachines.com] >>Sent: Monday, June 13, 2005 2:52 PM >> >>On 06/13/05 14:12, Roland Dreier wrote: >> >>> Jeff> On the recieving end of a request, when the handler gets >>> Jeff> called and the cm-id gets passed to your handler (aka when >>> Jeff> the handler gets IB_CM_REQ_RECEIVED) it seems cm-id->context >>> Jeff> isn't null. Is there some useful data in there? If not, I >>> Jeff> think it makes more sense (and makes code easier to write to >>> Jeff> use the cm handler) if the context is set to NULL. >>> >>>I think the context gets populated from the context of the >>>corresponding cm_id that was passed to ib_cm_listen(). >> >>OK, thanks. I checked and it does look like that is what is happening. >>I'll set the context for the cm_id I pass into ib_cm_listen to null. >>Assuming of course, that the cm_id context for that one doesn't matter. > > > The CM doesn't interpret the user's context ever. It is just there for the > benefit of the user. If your app doesn't need the context in the REQ callback, > then you can set it to NULL. > > Out of curiosity, if you don't set the listen's context, how do you figure out > which listen received the callback? Does DAPL not care? I only call ib_cm_listen once: ib_cm_listen(listen_id, LISTEN_NUM, 0); Is it necessary to call listen for each connection I wish to recieve? It seems like the cm handler function I register when I create the cm_id continues to handle req events until it is destroyed so I'm assuming this is all I need. In the case where I only have one number I'm listening on, then it doesn't matter to me what the context is; NULL makes the code simple. Jeff From tduffy at sun.com Mon Jun 13 15:26:07 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 13 Jun 2005 15:26:07 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper funct ion dat_ia_query() In-Reply-To: <469958e00506131506137bd164@mail.gmail.com> References: <91DB792C7985D411BEC300B40080D29CC35C67@mtvex01.mtv.mtl.com> <20050609094759.GA5947@lst.de> <1118330531.21255.6.camel@duffman> <469958e0050609135862ace6fc@mail.gmail.com> <20050611092040.GA18263@lst.de> <1118591987.625.6.camel@duffman> <469958e00506131506137bd164@mail.gmail.com> Message-ID: <1118701567.9114.28.camel@duffman> On Mon, 2005-06-13 at 15:06 -0700, Caitlin Bestler wrote: > What's important is that enables dapl objects to be handled polymorphically. > Internally there are many times when a utility routine only needs to know > that something is a dapl object, not which one it is. > > The alternative is to have some other mechanism to ensure that the > shared fields are defined compatibly in all sub-types. This is doable, > but declaring a common header is generally the simpler and more > self-documenting method of implementing a parent type with > multiple sub-types in plain C. We have polymorphism already: dapl inherits from dat. Are you saying this is so we can have some sorta multiple inheritance? I don't think this is really used. In fact, in dapl, the list is going away, the lock (and flags) should be in a per struct anyways. So, all we have left is a pointer to ia? Just add that to each struct. Not a big deal. > The external requirements for polymorphic dapl handles are fewer: > get/set consumer handle (which ismostly be for user mode > programs anyway) done in dat, now. > and some methods apply to either a PSP or > RSP. which is a common structure with just a flag to tell the difference. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From ftillier at silverstorm.com Mon Jun 13 15:28:42 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Mon, 13 Jun 2005 15:28:42 -0700 Subject: [openib-general] cm-id->context should be NULL forIB_CM_REQ_RECEIVED ? In-Reply-To: <42AE06C3.7030406@linuxmachines.com> Message-ID: <000101c57067$41fab920$9c5aa8c0@infiniconsys.com> > From: Jeff Carr [mailto:jcarr at linuxmachines.com] > Sent: Monday, June 13, 2005 3:21 PM > > I only call ib_cm_listen once: > > ib_cm_listen(listen_id, LISTEN_NUM, 0); > > Is it necessary to call listen for each connection I wish to recieve? No, the listen stays around and keeps indicating connection requests as long as your app keeps it active. > It seems like the cm handler function I register when I create the cm_id > continues to handle req events until it is destroyed so I'm assuming > this is all I need. > > In the case where I only have one number I'm listening on, then it > doesn't matter to me what the context is; NULL makes the code simple. Yes, that makes sense. Thanks! - Fab From jlentini at netapp.com Mon Jun 13 15:33:56 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 13 Jun 2005 18:33:56 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1118664517.4476.9585.camel@localhost.localdomain> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <1118664517.4476.9585.camel@localhost.localdomain> Message-ID: On Mon, 13 Jun 2005, Hal Rosenstock wrote: halr> On Wed, 2005-06-08 at 17:53, James Lentini wrote: halr> > On Wed, 8 Jun 2005, Hal Rosenstock wrote: halr> > halr> > halr> On Wed, 2005-06-08 at 11:44, James Lentini wrote: halr> > halr> > We interpreted the above to mean "give the connection protocol as halr> > halr> > much time as it needs to establish a connection, but don't mask halr> > halr> > errors (no path to the remove node, etc.)". For that reason we changed halr> > halr> > the variable name to DAT_TIMEOUT_MAX. halr> > halr> halr> > halr> But if the REQ is lost, the timeout is really really long (longer than halr> > halr> most will wait for an error). halr> > halr> > If a user doesn't want to wait DAT_TIMEOUT_MAX time, it can pass a halr> > smaller amount of time to dat_ep_connect. Does this satisfy your halr> > requirements? halr> halr> Is it the intended that the only way out is via user intervention (e.g. halr> ctl-C) ? If one connection attempt (REQ) is made and it is lost, then halr> there is no chance of it completing and the user needs to intervene. Why does the user need to intervene? Did I misunderstanding the CM API? When dapl_ep_connect() is called with a timeout value of DAT_TIMEOUT_MAX, DAPL passes ib_sen_cm_req the value 0x1F in the ib_cm_req_param structure's remote_cm_response_timeout value. My understanding was that this is the maximum timeout and that once it expires the CM will inform the user that the REQ timed out. halr> If that is the intended behavior, we are there. (This (lost REQ) halr> can even occur when the timeout is non infinite too). We didn't intend for the active side to wait forever if a REQ was lost. halr> halr> An alternative (as Sean suggested) is to continually retry (at a halr> periodicity below the supplied timeout) until the time period specified halr> expires. That seems to be better (at least to me and Sean) in terms of halr> handling the lost REQ case. As retries is not part of the API for halr> connect, I would presume the implementor is free to what they want under halr> the covers of dapl_ib_connect. You're correct. halr> halr> -- Hal halr> From jlentini at netapp.com Mon Jun 13 15:35:23 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 13 Jun 2005 18:35:23 -0400 (EDT) Subject: [openib-general] cm-id->context should be NULL forIB_CM_REQ_RECEIVED ? In-Reply-To: <42AE06C3.7030406@linuxmachines.com> References: <000001c57062$b301c230$9c5aa8c0@infiniconsys.com> <42AE06C3.7030406@linuxmachines.com> Message-ID: On Mon, 13 Jun 2005, Jeff Carr wrote: > On 06/13/05 14:56, Fab Tillier wrote: >>> From: Jeff Carr [mailto:jcarr at linuxmachines.com] >>> Sent: Monday, June 13, 2005 2:52 PM >>> >>> On 06/13/05 14:12, Roland Dreier wrote: >>> >>>> Jeff> On the recieving end of a request, when the handler gets >>>> Jeff> called and the cm-id gets passed to your handler (aka when >>>> Jeff> the handler gets IB_CM_REQ_RECEIVED) it seems cm-id->context >>>> Jeff> isn't null. Is there some useful data in there? If not, I >>>> Jeff> think it makes more sense (and makes code easier to write to >>>> Jeff> use the cm handler) if the context is set to NULL. >>>> >>>> I think the context gets populated from the context of the >>>> corresponding cm_id that was passed to ib_cm_listen(). >>> >>> OK, thanks. I checked and it does look like that is what is happening. >>> I'll set the context for the cm_id I pass into ib_cm_listen to null. >>> Assuming of course, that the cm_id context for that one doesn't matter. >> >> >> The CM doesn't interpret the user's context ever. It is just there for the >> benefit of the user. If your app doesn't need the context in the REQ >> callback, >> then you can set it to NULL. >> >> Out of curiosity, if you don't set the listen's context, how do you figure >> out >> which listen received the callback? Does DAPL not care? I don't believe Jeff is using DAPL. > > I only call ib_cm_listen once: > > ib_cm_listen(listen_id, LISTEN_NUM, 0); > > Is it necessary to call listen for each connection I wish to recieve? It > seems like the cm handler function I register when I create the cm_id > continues to handle req events until it is destroyed so I'm assuming this is > all I need. > > In the case where I only have one number I'm listening on, then it doesn't > matter to me what the context is; NULL makes the code simple. > > Jeff > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From jcarr at linuxmachines.com Mon Jun 13 15:57:48 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 13 Jun 2005 15:57:48 -0700 Subject: [openib-general] cm-id->context should be NULL forIB_CM_REQ_RECEIVED ? In-Reply-To: References: <000001c57062$b301c230$9c5aa8c0@infiniconsys.com> <42AE06C3.7030406@linuxmachines.com> Message-ID: <42AE0F6C.2090804@linuxmachines.com> On 06/13/05 15:35, James Lentini wrote: > I don't believe Jeff is using DAPL. That's true; I just need something simple. There is enough abstraction and layers already so I don't want to add another one. :) Jeff From jjengla at sandia.gov Mon Jun 13 16:45:14 2005 From: jjengla at sandia.gov (Josh England) Date: Mon, 13 Jun 2005 16:45:14 -0700 Subject: [openib-general] dapl build problems Message-ID: <1118706314.17676.28.camel@localhost> I get these problems trying to build dat with 2.6.11.11 (svn rev 2595): *** Warning: "dat_registry_add_provider" [drivers/infiniband/ulp/dat-provider/ib_dat_provider.ko] undefined! *** Warning: "dat_registry_remove_provider" [drivers/infiniband/ulp/dat-provider/ib_dat_provider.ko] undefined! -JE From jjengla at sandia.gov Mon Jun 13 17:18:17 2005 From: jjengla at sandia.gov (Josh England) Date: Mon, 13 Jun 2005 17:18:17 -0700 Subject: [openib-general] kdaptest wedges server In-Reply-To: References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> Message-ID: <1118708297.17672.32.camel@localhost> On Thu, 2005-06-09 at 14:35 -0400, James Lentini wrote: > Josh, > > Was there any information about what occurred? > > Loading the the DAT modules with > > modprobe dat dbg_mask=0xff > modprobe ib_dat_provider dbg_mask=0xffff > > will produce a lot of debug information that might help. > > james I can't get DAT (provider) to build with the latest rev. I'll try again later. -JE > On Thu, 9 Jun 2005, Itamar Rabenstein wrote: > > > No , I dont see the problem at my system . > > > > Itamar > > > >> -----Original Message----- > >> From: James Lentini [mailto:jlentini at netapp.com] > >> Sent: Thursday, June 09, 2005 6:03 PM > >> To: Itamar Rabenstein > >> Cc: Josh England; openib-general > >> Subject: RE: [openib-general] kdaptest wedges server > >> > >> > >> > >> Itamar, > >> > >> Were you able to get to the bottom of this? > >> > >> james > >> > >> On Thu, 2 Jun 2005, Itamar Rabenstein wrote: > >> > >> itamar>Hi Josh, > >> itamar> > >> itamar>Can you run it with debug option ( -d ) > >> itamar>and tell me what is the last debug print on the server side ? > >> itamar>when you say "kill" can you hit ^c or is it hanging > >> the station ? > >> itamar> > >> itamar>what is system configuration ? (arch , Os ,kernel) > >> itamar> > >> itamar> Itamar > >> itamar> > >> itamar> > >> itamar>> -----Original Message----- > >> itamar>> From: Josh England [mailto:jjengla at sandia.gov] > >> itamar>> Sent: Thursday, June 02, 2005 9:00 PM > >> itamar>> To: openib-general > >> itamar>> Subject: [openib-general] kdaptest wedges server > >> itamar>> > >> itamar>> > >> itamar>> Hi, > >> itamar>> > >> itamar>> The kdapltest program wedges consistently wedges a > >> node. I just tried > >> itamar>> it with Revision 2523, and it runs fine the first couple of > >> itamar>> times. The > >> itamar>> third time took a while, and the fourth time killed > >> the server. > >> itamar>> > >> itamar>> The server is running: > >> itamar>> kdapltest -T S -D mthca0a > >> itamar>> > >> itamar>> The client runs this several times in succession: > >> itamar>> kdapltest -T T -D mthca0a -s client SR 256 > >> server SR 256 > >> itamar>> > >> itamar>> -JE > >> itamar>> > >> itamar>> > >> itamar>> _______________________________________________ > >> itamar>> openib-general mailing list > >> itamar>> openib-general at openib.org > >> itamar>> http://openib.org/mailman/listinfo/openib-general > >> itamar>> > >> itamar>> To unsubscribe, please visit > >> itamar>> http://openib.org/mailman/listinfo/openib-general > >> itamar>> > >> itamar>_______________________________________________ > >> itamar>openib-general mailing list > >> itamar>openib-general at openib.org > >> itamar>http://openib.org/mailman/listinfo/openib-general > >> itamar> > >> itamar>To unsubscribe, please visit > >> http://openib.org/mailman/listinfo/openib-general > >> itamar> > >> > > > From Tom.Duffy at Sun.COM Mon Jun 13 17:18:06 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:06 -0700 Subject: [openib-general] [PATCH 0/9] kDAPL: Use Linux standard linked lists Message-ID: <11187082863011@sun.com> This series of patches removes the use of dapl's own linked list implementation in favor of using standard Linux linked lists. Please apply in order. Signed-off-by: Tom Duffy From Tom.Duffy at Sun.COM Mon Jun 13 17:18:07 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:07 -0700 Subject: [openib-general] [PATCH 2/9] kDAPL: convert the lmr list to linux native In-Reply-To: <11187082862310@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> Message-ID: <11187082871393@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll2/dat-provider/dapl.h linux-kernel-ll3/dat-provider/dapl.h --- linux-kernel-ll2/dat-provider/dapl.h 2005-06-13 14:34:58.534000000 -0700 +++ linux-kernel-ll3/dat-provider/dapl.h 2005-06-13 15:27:19.134000000 -0700 @@ -155,7 +155,7 @@ struct dapl_ia { struct dapl_llist_entry hca_ia_list_entry; /* HCAs list of IAs */ struct list_head ep_list; /* EP queue */ - struct dapl_llist_entry *lmr_list_head; /* LMR queue */ + struct list_head lmr_list; /* LMR queue */ struct dapl_llist_entry *rmr_list_head; /* RMR queue */ struct dapl_llist_entry *pz_list_head; /* PZ queue */ struct dapl_llist_entry *evd_list_head; /* EVD queue */ @@ -251,6 +251,7 @@ struct dapl_pz { struct dapl_lmr { struct dat_lmr lmr; struct dapl_common common; + struct list_head list; struct dat_lmr_param param; struct ib_mr *mr; atomic_t lmr_ref_count; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll2/dat-provider/dapl_ia.c linux-kernel-ll3/dat-provider/dapl_ia.c --- linux-kernel-ll2/dat-provider/dapl_ia.c 2005-06-13 14:40:19.903015000 -0700 +++ linux-kernel-ll3/dat-provider/dapl_ia.c 2005-06-13 15:30:24.780010000 -0700 @@ -64,7 +64,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat ia->cleanup_async_error_evd = FALSE; dapl_llist_init_entry(&ia->hca_ia_list_entry); INIT_LIST_HEAD(&ia->ep_list); - dapl_llist_init_head(&ia->lmr_list_head); + INIT_LIST_HEAD(&ia->lmr_list); dapl_llist_init_head(&ia->rmr_list_head); dapl_llist_init_head(&ia->pz_list_head); dapl_llist_init_head(&ia->evd_list_head); @@ -105,7 +105,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia { u32 dat_status = DAT_SUCCESS; struct dapl_ep *ep; - struct dapl_lmr *lmr, *next_lmr; + struct dapl_lmr *lmr; struct dapl_rmr *rmr, *next_rmr; struct dapl_pz *pz, *next_pz; struct dapl_evd *evd, *next_evd; @@ -178,18 +178,12 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ep, dat_status); } - lmr = (dapl_llist_is_empty(&ia->lmr_list_head) - ? NULL : dapl_llist_peek_head(&ia->lmr_list_head)); - while (lmr != NULL) { - next_lmr = dapl_llist_next_entry(&ia->lmr_list_head, - &lmr->common. - ia_list_entry); + list_for_each_entry(lmr, &ia->lmr_list, list) { dat_status = dapl_lmr_free((struct dat_lmr *)lmr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): lmr_free(%p) returns %x\n", lmr, dat_status); - lmr = next_lmr; } sp = (dapl_llist_is_empty(&ia->psp_list_head) @@ -321,7 +315,7 @@ u32 dapl_ia_graceful_close(struct dapl_i if (!dapl_llist_is_empty(&ia->rmr_list_head) || !dapl_llist_is_empty(&ia->rsp_list_head) || !list_empty(&ia->ep_list) || - !dapl_llist_is_empty(&ia->lmr_list_head) || + !list_empty(&ia->lmr_list) || !dapl_llist_is_empty(&ia->psp_list_head) || !dapl_llist_is_empty(&ia->pz_list_head)) { dat_status = @@ -413,7 +407,7 @@ bail: void dapl_ia_free(struct dapl_ia *ia) { dapl_os_assert(ia->async_error_evd == NULL); - dapl_os_assert(dapl_llist_is_empty(&ia->lmr_list_head)); + dapl_os_assert(list_empty(&ia->lmr_list)); dapl_os_assert(dapl_llist_is_empty(&ia->rmr_list_head)); dapl_os_assert(list_empty(&ia->ep_list)); dapl_os_assert(dapl_llist_is_empty(&ia->evd_list_head)); @@ -474,8 +468,7 @@ void dapl_ia_unlink_srq(struct dapl_ia * void dapl_ia_link_lmr(struct dapl_ia *ia, struct dapl_lmr *lmr) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_add_head(&ia->lmr_list_head, - &lmr->common.ia_list_entry, lmr); + list_add(&lmr->list, &ia->lmr_list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } @@ -485,8 +478,7 @@ void dapl_ia_link_lmr(struct dapl_ia *ia void dapl_ia_unlink_lmr(struct dapl_ia *ia, struct dapl_lmr *lmr) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_remove_entry(&ia->lmr_list_head, - &lmr->common.ia_list_entry); + list_del(&lmr->list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } From Tom.Duffy at Sun.COM Mon Jun 13 17:18:07 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:07 -0700 Subject: [openib-general] [PATCH 3/9] kDAPL: convert the rmr list to linux native In-Reply-To: <11187082871393@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> Message-ID: <11187082871245@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll3/dat-provider/dapl.h linux-kernel-ll4/dat-provider/dapl.h --- linux-kernel-ll3/dat-provider/dapl.h 2005-06-13 15:27:19.134000000 -0700 +++ linux-kernel-ll4/dat-provider/dapl.h 2005-06-13 15:35:21.410008000 -0700 @@ -156,7 +156,7 @@ struct dapl_ia { struct dapl_llist_entry hca_ia_list_entry; /* HCAs list of IAs */ struct list_head ep_list; /* EP queue */ struct list_head lmr_list; /* LMR queue */ - struct dapl_llist_entry *rmr_list_head; /* RMR queue */ + struct list_head rmr_list; /* RMR queue */ struct dapl_llist_entry *pz_list_head; /* PZ queue */ struct dapl_llist_entry *evd_list_head; /* EVD queue */ struct dapl_llist_entry *psp_list_head; /* PSP queue */ @@ -260,6 +260,7 @@ struct dapl_lmr { struct dapl_rmr { struct dat_rmr rmr; struct dapl_common common; + struct list_head list; struct dat_rmr_param param; struct dapl_ep *ep; struct dapl_pz *pz; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll3/dat-provider/dapl_ia.c linux-kernel-ll4/dat-provider/dapl_ia.c --- linux-kernel-ll3/dat-provider/dapl_ia.c 2005-06-13 15:30:24.780010000 -0700 +++ linux-kernel-ll4/dat-provider/dapl_ia.c 2005-06-13 15:38:50.547003000 -0700 @@ -65,7 +65,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat dapl_llist_init_entry(&ia->hca_ia_list_entry); INIT_LIST_HEAD(&ia->ep_list); INIT_LIST_HEAD(&ia->lmr_list); - dapl_llist_init_head(&ia->rmr_list_head); + INIT_LIST_HEAD(&ia->rmr_list); dapl_llist_init_head(&ia->pz_list_head); dapl_llist_init_head(&ia->evd_list_head); dapl_llist_init_head(&ia->rsp_list_head); @@ -106,7 +106,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia u32 dat_status = DAT_SUCCESS; struct dapl_ep *ep; struct dapl_lmr *lmr; - struct dapl_rmr *rmr, *next_rmr; + struct dapl_rmr *rmr; struct dapl_pz *pz, *next_pz; struct dapl_evd *evd, *next_evd; struct dapl_sp *sp, *next_sp; /* for PSP and RSP queues */ @@ -123,18 +123,12 @@ u32 dapl_ia_abrupt_close(struct dapl_ia * if we end up skipping an entry. */ - rmr = (dapl_llist_is_empty(&ia->rmr_list_head) - ? NULL : dapl_llist_peek_head(&ia->rmr_list_head)); - while (rmr != NULL) { - next_rmr = dapl_llist_next_entry(&ia->rmr_list_head, - &rmr->common. - ia_list_entry); + list_for_each_entry(rmr, &ia->rmr_list, list) { dat_status = dapl_rmr_free((struct dat_rmr *)rmr); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rmr_free(%p) returns %x\n", rmr, dat_status); - rmr = next_rmr; } sp = (dapl_llist_is_empty(&ia->rsp_list_head) @@ -312,7 +306,7 @@ u32 dapl_ia_graceful_close(struct dapl_i struct dapl_llist_entry *entry; struct dapl_hca *hca; - if (!dapl_llist_is_empty(&ia->rmr_list_head) || + if (!list_empty(&ia->rmr_list) || !dapl_llist_is_empty(&ia->rsp_list_head) || !list_empty(&ia->ep_list) || !list_empty(&ia->lmr_list) || @@ -408,7 +402,7 @@ void dapl_ia_free(struct dapl_ia *ia) { dapl_os_assert(ia->async_error_evd == NULL); dapl_os_assert(list_empty(&ia->lmr_list)); - dapl_os_assert(dapl_llist_is_empty(&ia->rmr_list_head)); + dapl_os_assert(list_empty(&ia->rmr_list)); dapl_os_assert(list_empty(&ia->ep_list)); dapl_os_assert(dapl_llist_is_empty(&ia->evd_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia->psp_list_head)); @@ -488,8 +482,7 @@ void dapl_ia_unlink_lmr(struct dapl_ia * void dapl_ia_link_rmr(struct dapl_ia *ia, struct dapl_rmr *rmr) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_add_head(&ia->rmr_list_head, - &rmr->common.ia_list_entry, rmr); + list_add(&rmr->list, &ia->rmr_list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } @@ -499,8 +492,7 @@ void dapl_ia_link_rmr(struct dapl_ia *ia void dapl_ia_unlink_rmr(struct dapl_ia *ia, struct dapl_rmr *rmr) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_remove_entry(&ia->rmr_list_head, - &rmr->common.ia_list_entry); + list_del(&rmr->list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } From Tom.Duffy at Sun.COM Mon Jun 13 17:18:06 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:06 -0700 Subject: [openib-general] [PATCH 1/9] kDAPL: convert the cr list to linux native In-Reply-To: <11187082863011@sun.com> References: <11187082863011@sun.com> Message-ID: <11187082862310@sun.com> Signed-off-by: Tom Duffy Index: linux-kernel-ll2/dat-provider/dapl_ia.c =================================================================== --- linux-kernel-ll2/dat-provider/dapl_ia.c (revision 2595) +++ linux-kernel-ll2/dat-provider/dapl_ia.c (working copy) @@ -110,7 +110,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia struct dapl_pz *pz, *next_pz; struct dapl_evd *evd, *next_evd; struct dapl_sp *sp, *next_sp; /* for PSP and RSP queues */ - struct dapl_cr *cr, *next_cr; + struct dapl_cr *cr; struct dapl_hca *hca; /* @@ -210,12 +210,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia ia_list_entry); /* Remove CR's from this PSP and clean them up */ - cr = dapl_llist_is_empty(&sp->cr_list_head) ? NULL : - dapl_llist_peek_head(&sp->cr_list_head); - while (cr != NULL) { - next_cr = dapl_llist_next_entry( - &sp->cr_list_head, - &cr->common.ia_list_entry); + list_for_each_entry(cr, &sp->cr_list, list) { /* Remove the CR from the queue & cleanup */ spin_lock_irqsave(&sp->common.lock, sp->common.flags); @@ -224,7 +219,6 @@ u32 dapl_ia_abrupt_close(struct dapl_ia sp->common.flags); dapl_cr_free(cr); - cr = next_cr; } dat_status = dapl_psp_free((struct dat_sp *)sp); Index: linux-kernel-ll2/dat-provider/dapl_sp.c =================================================================== --- linux-kernel-ll2/dat-provider/dapl_sp.c (revision 2595) +++ linux-kernel-ll2/dat-provider/dapl_sp.c (working copy) @@ -57,17 +57,14 @@ struct dapl_sp *dapl_sp_alloc(struct dap dapl_llist_init_entry(&sp->common.ia_list_entry); spin_lock_init(&sp->common.lock); - /* - * Initialize the Body (set to NULL above) - */ - dapl_llist_init_head(&sp->cr_list_head); + INIT_LIST_HEAD(&sp->cr_list); return sp; } void dapl_sp_dealloc(struct dapl_sp *sp) { - dapl_os_assert(dapl_llist_is_empty(&sp->cr_list_head)); + dapl_os_assert(list_empty(&sp->cr_list)); kfree(sp); } @@ -75,8 +72,7 @@ void dapl_sp_dealloc(struct dapl_sp *sp) void dapl_sp_link_cr(struct dapl_sp *sp, struct dapl_cr *cr) { spin_lock_irqsave(&sp->common.lock, sp->common.flags); - dapl_llist_add_tail(&sp->cr_list_head, - &cr->common.ia_list_entry, cr); + list_add_tail(&cr->list, &sp->cr_list); sp->cr_list_count++; spin_unlock_irqrestore(&sp->common.lock, sp->common.flags); } @@ -86,32 +82,23 @@ struct dapl_cr *dapl_sp_search_cr(struct { struct dapl_cr *cr; - if (dapl_llist_is_empty(&sp->cr_list_head)) - return NULL; - - cr = (struct dapl_cr *)dapl_llist_peek_head(&sp->cr_list_head); - - do { + list_for_each_entry(cr, &sp->cr_list, list) if (cr->cm_ctx == cm_ctx) return cr; - - cr = cr->common.ia_list_entry.flink->data; - } while ((void *)cr != (void *)sp->cr_list_head->data); - + return NULL; } void dapl_sp_remove_cr(struct dapl_sp *sp, struct dapl_cr *cr) { - if (dapl_llist_is_empty(&sp->cr_list_head)) { + if (list_empty(&sp->cr_list)) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "***dapl_sp_remove_cr: removing from empty queue! sp %p\n", sp); return; } - dapl_llist_remove_entry(&sp->cr_list_head, - &cr->common.ia_list_entry); + list_del(&cr->list); sp->cr_list_count--; } Index: linux-kernel-ll2/dat-provider/dapl.h =================================================================== --- linux-kernel-ll2/dat-provider/dapl.h (revision 2595) +++ linux-kernel-ll2/dat-provider/dapl.h (working copy) @@ -290,13 +290,14 @@ struct dapl_sp { /* maintenance fields */ boolean_t listening; /* PSP is registered & active */ struct ib_cm_id *cm_srvc_handle; /* Used by CM */ - struct dapl_llist_entry *cr_list_head; /* CR pending queue */ + struct list_head cr_list; /* CR pending queue */ int cr_list_count; /* count of CRs on queue */ }; struct dapl_cr { struct dat_cr cr; struct dapl_common common; + struct list_head list; /* for convenience the data is kept as a struct dat_cr_param. * however, the "local_endpoint" field is always NULL * so this wastes a pointer. This is probably ok to From Tom.Duffy at Sun.COM Mon Jun 13 17:18:08 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:08 -0700 Subject: [openib-general] [PATCH 4/9] kDAPL: convert the pz list to linux native In-Reply-To: <11187082871245@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> Message-ID: <1118708288376@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll4/dat-provider/dapl.h linux-kernel-ll5/dat-provider/dapl.h --- linux-kernel-ll4/dat-provider/dapl.h 2005-06-13 15:35:21.410008000 -0700 +++ linux-kernel-ll5/dat-provider/dapl.h 2005-06-13 15:46:07.858004000 -0700 @@ -157,7 +157,7 @@ struct dapl_ia { struct list_head ep_list; /* EP queue */ struct list_head lmr_list; /* LMR queue */ struct list_head rmr_list; /* RMR queue */ - struct dapl_llist_entry *pz_list_head; /* PZ queue */ + struct list_head pz_list; /* PZ queue */ struct dapl_llist_entry *evd_list_head; /* EVD queue */ struct dapl_llist_entry *psp_list_head; /* PSP queue */ struct dapl_llist_entry *rsp_list_head; /* RSP queue */ @@ -244,6 +244,7 @@ struct dapl_srq { struct dapl_pz { struct dat_pz pz; struct dapl_common common; + struct list_head list; struct ib_pd *pd; atomic_t pz_ref_count; }; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll4/dat-provider/dapl_ia.c linux-kernel-ll5/dat-provider/dapl_ia.c --- linux-kernel-ll4/dat-provider/dapl_ia.c 2005-06-13 15:38:50.547003000 -0700 +++ linux-kernel-ll5/dat-provider/dapl_ia.c 2005-06-13 15:48:31.170005000 -0700 @@ -66,7 +66,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat INIT_LIST_HEAD(&ia->ep_list); INIT_LIST_HEAD(&ia->lmr_list); INIT_LIST_HEAD(&ia->rmr_list); - dapl_llist_init_head(&ia->pz_list_head); + INIT_LIST_HEAD(&ia->pz_list); dapl_llist_init_head(&ia->evd_list_head); dapl_llist_init_head(&ia->rsp_list_head); dapl_llist_init_head(&ia->psp_list_head); @@ -107,7 +107,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia struct dapl_ep *ep; struct dapl_lmr *lmr; struct dapl_rmr *rmr; - struct dapl_pz *pz, *next_pz; + struct dapl_pz *pz; struct dapl_evd *evd, *next_evd; struct dapl_sp *sp, *next_sp; /* for PSP and RSP queues */ struct dapl_cr *cr; @@ -218,18 +218,12 @@ u32 dapl_ia_abrupt_close(struct dapl_ia sp = next_sp; } - pz = (dapl_llist_is_empty(&ia->pz_list_head) - ? NULL : dapl_llist_peek_head(&ia->pz_list_head)); - while (pz != NULL) { - next_pz = dapl_llist_next_entry(&ia->pz_list_head, - &pz->common. - ia_list_entry); + list_for_each_entry(pz, &ia->pz_list, list) { dat_status = dapl_pz_free((struct dat_pz *)pz); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): pz_free(%p) returns %x\n", pz, dat_status); - pz = next_pz; } /* @@ -311,7 +305,7 @@ u32 dapl_ia_graceful_close(struct dapl_i !list_empty(&ia->ep_list) || !list_empty(&ia->lmr_list) || !dapl_llist_is_empty(&ia->psp_list_head) || - !dapl_llist_is_empty(&ia->pz_list_head)) { + !list_empty(&ia->pz_list)) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_IA_IN_USE); goto bail; @@ -502,8 +496,7 @@ void dapl_ia_unlink_rmr(struct dapl_ia * void dapl_ia_link_pz(struct dapl_ia *ia, struct dapl_pz *pz) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_add_head(&ia->pz_list_head, - &pz->common.ia_list_entry, pz); + list_add(&pz->list, &ia->pz_list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } @@ -513,8 +506,7 @@ void dapl_ia_link_pz(struct dapl_ia *ia, void dapl_ia_unlink_pz(struct dapl_ia *ia, struct dapl_pz *pz) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_remove_entry(&ia->pz_list_head, - &pz->common.ia_list_entry); + list_del(&pz->list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } From Tom.Duffy at Sun.COM Mon Jun 13 17:18:08 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:08 -0700 Subject: [openib-general] [PATCH 5/9] kDAPL: convert the evd list to linux native In-Reply-To: <1118708288376@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> Message-ID: <1118708288856@sun.com> Please pay particular attention to the part where the asynchronous event is kept around. I think I got it right, but would like to make sure another pair of eyeballs looked it over. Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll5/dat-provider/dapl.h linux-kernel-ll6/dat-provider/dapl.h --- linux-kernel-ll5/dat-provider/dapl.h 2005-06-13 15:46:07.858004000 -0700 +++ linux-kernel-ll6/dat-provider/dapl.h 2005-06-13 16:11:11.259003000 -0700 @@ -158,7 +158,7 @@ struct dapl_ia { struct list_head lmr_list; /* LMR queue */ struct list_head rmr_list; /* RMR queue */ struct list_head pz_list; /* PZ queue */ - struct dapl_llist_entry *evd_list_head; /* EVD queue */ + struct list_head evd_list; /* EVD queue */ struct dapl_llist_entry *psp_list_head; /* PSP queue */ struct dapl_llist_entry *rsp_list_head; /* RSP queue */ struct dapl_llist_entry *srq_list_head; /* SRQ queue */ @@ -167,6 +167,7 @@ struct dapl_ia { struct dapl_evd { struct dat_evd evd; struct dapl_common common; + struct list_head list; DAPL_EVD_STATE evd_state; enum dat_evd_flags evd_flags; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll5/dat-provider/dapl_ia.c linux-kernel-ll6/dat-provider/dapl_ia.c --- linux-kernel-ll5/dat-provider/dapl_ia.c 2005-06-13 15:48:31.170005000 -0700 +++ linux-kernel-ll6/dat-provider/dapl_ia.c 2005-06-13 16:30:09.267002000 -0700 @@ -67,7 +67,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat INIT_LIST_HEAD(&ia->lmr_list); INIT_LIST_HEAD(&ia->rmr_list); INIT_LIST_HEAD(&ia->pz_list); - dapl_llist_init_head(&ia->evd_list_head); + INIT_LIST_HEAD(&ia->evd_list); dapl_llist_init_head(&ia->rsp_list_head); dapl_llist_init_head(&ia->psp_list_head); @@ -108,7 +108,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia struct dapl_lmr *lmr; struct dapl_rmr *rmr; struct dapl_pz *pz; - struct dapl_evd *evd, *next_evd; + struct dapl_evd *evd; struct dapl_sp *sp, *next_sp; /* for PSP and RSP queues */ struct dapl_cr *cr; struct dapl_hca *hca; @@ -232,12 +232,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia * HCA, to accept any async events that occur. So we cycle through * the list with dapl_llist_next_entry instead of dapl_llist_is_empty. */ - evd = (dapl_llist_is_empty(&ia->evd_list_head) - ? NULL : dapl_llist_peek_head(&ia->evd_list_head)); - while (evd != NULL) { - next_evd = dapl_llist_next_entry(&ia->evd_list_head, - &evd->common. - ia_list_entry); + list_for_each_entry(evd, &ia->evd_list, list) { if (evd != ia->async_error_evd) { /* it isn't the async EVD; delete it. */ dat_status = dapl_evd_free((struct dat_evd *)evd); @@ -246,7 +241,6 @@ u32 dapl_ia_abrupt_close(struct dapl_ia "ia_close(ABRUPT): evd_free(%p) returns %x\n", evd, dat_status); } - evd = next_evd; } hca = ia->hca; @@ -297,7 +291,6 @@ u32 dapl_ia_graceful_close(struct dapl_i u32 dat_status = DAT_SUCCESS; u32 cur_dat_status; struct dapl_evd *evd; - struct dapl_llist_entry *entry; struct dapl_hca *hca; if (!list_empty(&ia->rmr_list) || @@ -315,7 +308,7 @@ u32 dapl_ia_graceful_close(struct dapl_i /* (ie. it was not created by dapl_ia_open) */ /* then the evd list should be empty */ if (FALSE == ia->cleanup_async_error_evd) { - if (!dapl_llist_is_empty(&ia->evd_list_head)) { + if (!list_empty(&ia->evd_list)) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_IA_IN_USE); @@ -325,9 +318,9 @@ u32 dapl_ia_graceful_close(struct dapl_i /* else the async evd should be the only evd in */ /* the list. */ else { - evd = (dapl_llist_is_empty(&ia->evd_list_head) - ? NULL : dapl_llist_peek_head(&ia-> - evd_list_head)); + evd = (list_empty(&ia->evd_list) + ? NULL : list_entry(ia->evd_list.next, + struct dapl_evd, list)); if (evd != NULL && !(evd->evd_flags & DAT_EVD_ASYNC_FLAG)) { @@ -337,10 +330,8 @@ u32 dapl_ia_graceful_close(struct dapl_i goto bail; } - entry = ia->evd_list_head; - /* if the async evd is not the only element in the list */ - if (entry->blink != entry->flink) { + if (evd->list.next != &ia->evd_list) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_IA_IN_USE); @@ -398,7 +389,7 @@ void dapl_ia_free(struct dapl_ia *ia) dapl_os_assert(list_empty(&ia->lmr_list)); dapl_os_assert(list_empty(&ia->rmr_list)); dapl_os_assert(list_empty(&ia->ep_list)); - dapl_os_assert(dapl_llist_is_empty(&ia->evd_list_head)); + dapl_os_assert(list_empty(&ia->evd_list)); dapl_os_assert(dapl_llist_is_empty(&ia->psp_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia->rsp_list_head)); @@ -516,8 +507,7 @@ void dapl_ia_unlink_pz(struct dapl_ia *i void dapl_ia_link_evd(struct dapl_ia *ia, struct dapl_evd *evd) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_add_head(&ia->evd_list_head, - &evd->common.ia_list_entry, evd); + list_add(&evd->list, &ia->evd_list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } @@ -527,8 +517,7 @@ void dapl_ia_link_evd(struct dapl_ia *ia void dapl_ia_unlink_evd(struct dapl_ia *ia, struct dapl_evd *evd) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_remove_entry(&ia->evd_list_head, - &evd->common.ia_list_entry); + list_del(&evd->list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } From Tom.Duffy at Sun.COM Mon Jun 13 17:18:09 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:09 -0700 Subject: [openib-general] [PATCH 6/9] kDAPL: convert the psp and rsp lists to linux native In-Reply-To: <1118708288856@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> <1118708288856@sun.com> Message-ID: <11187082893179@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll6/dat-provider/dapl.h linux-kernel-ll7/dat-provider/dapl.h --- linux-kernel-ll6/dat-provider/dapl.h 2005-06-13 16:11:11.259003000 -0700 +++ linux-kernel-ll7/dat-provider/dapl.h 2005-06-13 16:36:22.902013000 -0700 @@ -159,8 +159,8 @@ struct dapl_ia { struct list_head rmr_list; /* RMR queue */ struct list_head pz_list; /* PZ queue */ struct list_head evd_list; /* EVD queue */ - struct dapl_llist_entry *psp_list_head; /* PSP queue */ - struct dapl_llist_entry *rsp_list_head; /* RSP queue */ + struct list_head psp_list; /* PSP queue */ + struct list_head rsp_list; /* RSP queue */ struct dapl_llist_entry *srq_list_head; /* SRQ queue */ }; @@ -282,6 +282,7 @@ typedef enum dapl_sp_state { struct dapl_sp { struct dat_sp sp; struct dapl_common common; + struct list_head list; DAPL_SP_STATE state; /* type and queue of the SP */ /* PSP/RSP PARAM fields */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll6/dat-provider/dapl_ia.c linux-kernel-ll7/dat-provider/dapl_ia.c --- linux-kernel-ll6/dat-provider/dapl_ia.c 2005-06-13 16:30:09.267002000 -0700 +++ linux-kernel-ll7/dat-provider/dapl_ia.c 2005-06-13 16:46:18.755001000 -0700 @@ -68,8 +68,8 @@ struct dapl_ia *dapl_ia_alloc(struct dat INIT_LIST_HEAD(&ia->rmr_list); INIT_LIST_HEAD(&ia->pz_list); INIT_LIST_HEAD(&ia->evd_list); - dapl_llist_init_head(&ia->rsp_list_head); - dapl_llist_init_head(&ia->psp_list_head); + INIT_LIST_HEAD(&ia->rsp_list); + INIT_LIST_HEAD(&ia->psp_list); dapl_hca_link_ia(hca, ia); @@ -109,7 +109,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia struct dapl_rmr *rmr; struct dapl_pz *pz; struct dapl_evd *evd; - struct dapl_sp *sp, *next_sp; /* for PSP and RSP queues */ + struct dapl_sp *sp; /* for PSP and RSP queues */ struct dapl_cr *cr; struct dapl_hca *hca; @@ -131,18 +131,12 @@ u32 dapl_ia_abrupt_close(struct dapl_ia rmr, dat_status); } - sp = (dapl_llist_is_empty(&ia->rsp_list_head) - ? NULL : dapl_llist_peek_head(&ia->rsp_list_head)); - while (sp != NULL) { - next_sp = dapl_llist_next_entry(&ia->rsp_list_head, - &sp->common. - ia_list_entry); + list_for_each_entry(sp, &ia->rsp_list, list) { dat_status = dapl_rsp_free((struct dat_sp *)sp); if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rsp_free(%p) returns %x\n", sp, dat_status); - sp = next_sp; } list_for_each_entry(ep, &ia->ep_list, list) { @@ -180,9 +174,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia lmr, dat_status); } - sp = (dapl_llist_is_empty(&ia->psp_list_head) - ? NULL : dapl_llist_peek_head(&ia->psp_list_head)); - while (sp != NULL) { + list_for_each_entry(sp, &ia->psp_list, list) { /* * Shut down the PSP so we get no further callbacks. There * should be no competing threads after this. @@ -193,10 +185,6 @@ u32 dapl_ia_abrupt_close(struct dapl_ia "ia_close(ABRUPT): psp cannot remove listener, returns %x\n", dat_status); - next_sp = dapl_llist_next_entry(&ia->psp_list_head, - &sp->common. - ia_list_entry); - /* Remove CR's from this PSP and clean them up */ list_for_each_entry(cr, &sp->cr_list, list) { /* Remove the CR from the queue & cleanup */ @@ -214,8 +202,6 @@ u32 dapl_ia_abrupt_close(struct dapl_ia dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): psp_free(%p) returns %x\n", sp, dat_status); - - sp = next_sp; } list_for_each_entry(pz, &ia->pz_list, list) { @@ -294,10 +280,10 @@ u32 dapl_ia_graceful_close(struct dapl_i struct dapl_hca *hca; if (!list_empty(&ia->rmr_list) || - !dapl_llist_is_empty(&ia->rsp_list_head) || + !list_empty(&ia->rsp_list) || !list_empty(&ia->ep_list) || !list_empty(&ia->lmr_list) || - !dapl_llist_is_empty(&ia->psp_list_head) || + !list_empty(&ia->psp_list) || !list_empty(&ia->pz_list)) { dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_IA_IN_USE); @@ -390,8 +376,8 @@ void dapl_ia_free(struct dapl_ia *ia) dapl_os_assert(list_empty(&ia->rmr_list)); dapl_os_assert(list_empty(&ia->ep_list)); dapl_os_assert(list_empty(&ia->evd_list)); - dapl_os_assert(dapl_llist_is_empty(&ia->psp_list_head)); - dapl_os_assert(dapl_llist_is_empty(&ia->rsp_list_head)); + dapl_os_assert(list_empty(&ia->psp_list)); + dapl_os_assert(list_empty(&ia->rsp_list)); dapl_hca_unlink_ia(ia->hca, ia); /* no need to destroy ia->common.lock */ @@ -527,8 +513,7 @@ void dapl_ia_unlink_evd(struct dapl_ia * void dapl_ia_link_psp(struct dapl_ia *ia, struct dapl_sp *sp) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_add_head(&ia->psp_list_head, - &sp->common.ia_list_entry, sp); + list_add(&sp->list, &ia->psp_list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } @@ -537,18 +522,8 @@ void dapl_ia_link_psp(struct dapl_ia *ia */ void dapl_ia_unlink_sp(struct dapl_ia *ia, struct dapl_sp *sp) { - struct dapl_llist_entry **list_head; - - if (sp->sp.type == DAT_SP_TYPE_PSP) { - list_head = &ia->psp_list_head; - } else { - dapl_os_assert(sp->sp.type == - DAT_SP_TYPE_RSP); - list_head = &ia->rsp_list_head; - } - spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_remove_entry(list_head, &sp->common.ia_list_entry); + list_del(&sp->list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } @@ -556,36 +531,30 @@ struct dapl_sp *dapl_ia_sp_search(struct boolean_t is_psp) { struct dapl_sp *sp; - struct dapl_llist_entry **list_head; + struct list_head *list_head; if (is_psp) - list_head = &ia->psp_list_head; + list_head = &ia->psp_list; else - list_head = &ia->rsp_list_head; + list_head = &ia->rsp_list; spin_lock_irqsave(&ia->common.lock, ia->common.flags); - sp = (dapl_llist_is_empty(list_head) ? NULL : - dapl_llist_peek_head(list_head)); - - while (sp != NULL) { - if (sp->conn_qual == conn_qual) - break; - - sp = dapl_llist_next_entry(list_head, - &sp->common.ia_list_entry); - } + list_for_each_entry(sp, list_head, list) + if (sp->conn_qual == conn_qual) { + spin_unlock_irqrestore(&ia->common.lock, + ia->common.flags); + return sp; + } spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); - - return sp; + return NULL; } void dapl_ia_link_rsp(struct dapl_ia *ia, struct dapl_sp *sp) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_add_head(&ia->rsp_list_head, - &sp->common.ia_list_entry, sp); + list_add(&sp->list, &ia->rsp_list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } From Tom.Duffy at Sun.COM Mon Jun 13 17:18:10 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:10 -0700 Subject: [openib-general] [PATCH 7/9] kDAPL: convert the srq list to linux native In-Reply-To: <11187082893179@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> <1118708288856@sun.com> <11187082893179@sun.com> Message-ID: <1118708290446@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll7/dat-provider/dapl.h linux-kernel-ll8/dat-provider/dapl.h --- linux-kernel-ll7/dat-provider/dapl.h 2005-06-13 16:36:22.902013000 -0700 +++ linux-kernel-ll8/dat-provider/dapl.h 2005-06-13 16:49:24.143015000 -0700 @@ -161,7 +161,7 @@ struct dapl_ia { struct list_head evd_list; /* EVD queue */ struct list_head psp_list; /* PSP queue */ struct list_head rsp_list; /* RSP queue */ - struct dapl_llist_entry *srq_list_head; /* SRQ queue */ + struct list_head srq_list; /* SRQ queue */ }; struct dapl_evd { @@ -236,6 +236,7 @@ struct dapl_ep { struct dapl_srq { struct dat_srq srq; struct dapl_common common; + struct list_head list; struct dat_srq_param param; atomic_t srq_ref_count; struct dapl_cookie_buffer recv_buffer; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll7/dat-provider/dapl_ia.c linux-kernel-ll8/dat-provider/dapl_ia.c --- linux-kernel-ll7/dat-provider/dapl_ia.c 2005-06-13 16:46:18.755001000 -0700 +++ linux-kernel-ll8/dat-provider/dapl_ia.c 2005-06-13 16:50:30.624000000 -0700 @@ -70,6 +70,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat INIT_LIST_HEAD(&ia->evd_list); INIT_LIST_HEAD(&ia->rsp_list); INIT_LIST_HEAD(&ia->psp_list); + INIT_LIST_HEAD(&ia->srq_list); dapl_hca_link_ia(hca, ia); @@ -411,8 +412,7 @@ void dapl_ia_unlink_ep(struct dapl_ia *i void dapl_ia_link_srq(struct dapl_ia *ia, struct dapl_srq *srq) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_add_head(&ia->srq_list_head, - &srq->common.ia_list_entry, srq); + list_add(&srq->list, &ia->srq_list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } @@ -422,8 +422,7 @@ void dapl_ia_link_srq(struct dapl_ia *ia void dapl_ia_unlink_srq(struct dapl_ia *ia, struct dapl_srq *srq) { spin_lock_irqsave(&ia->common.lock, ia->common.flags); - dapl_llist_remove_entry(&ia->srq_list_head, - &srq->common.ia_list_entry); + list_del(&srq->list); spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } From Tom.Duffy at Sun.COM Mon Jun 13 17:18:10 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:10 -0700 Subject: [openib-general] [PATCH 8/9] kDAPL: convert the hcas ia list to linux native In-Reply-To: <1118708290446@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> <1118708288856@sun.com> <11187082893179@sun.com> <1118708290446@sun.com> Message-ID: <11187082903037@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll8/dat-provider/dapl.h linux-kernel-ll9/dat-provider/dapl.h --- linux-kernel-ll8/dat-provider/dapl.h 2005-06-13 16:49:24.143015000 -0700 +++ linux-kernel-ll9/dat-provider/dapl.h 2005-06-13 16:53:00.300010000 -0700 @@ -123,7 +123,7 @@ struct dapl_cookie_buffer { struct dapl_hca { spinlock_t lock; unsigned long flags; - struct dapl_llist_entry *ia_list_head; /* list of all open IAs */ + struct list_head ia_list; /* list of all open IAs */ atomic_t handle_ref_count; /* open instance count */ char *name; /* provider name */ struct sockaddr_in6 hca_address; /* local address of HCA */ @@ -149,11 +149,11 @@ struct dapl_common { struct dapl_ia { struct dat_ia ia; struct dapl_common common; + struct list_head list; struct dapl_hca *hca; struct dapl_evd *async_error_evd; boolean_t cleanup_async_error_evd; - struct dapl_llist_entry hca_ia_list_entry; /* HCAs list of IAs */ struct list_head ep_list; /* EP queue */ struct list_head lmr_list; /* LMR queue */ struct list_head rmr_list; /* RMR queue */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll8/dat-provider/dapl_hca_util.c linux-kernel-ll9/dat-provider/dapl_hca_util.c --- linux-kernel-ll8/dat-provider/dapl_hca_util.c 2005-06-10 16:13:19.509014000 -0700 +++ linux-kernel-ll9/dat-provider/dapl_hca_util.c 2005-06-13 16:59:49.280012000 -0700 @@ -63,7 +63,7 @@ struct dapl_hca *dapl_hca_alloc(char *na dapl_hash_create(DAPL_HASH_TABLE_DEFAULT_CAPACITY, &hca->lmr_hash_table)) { spin_lock_init(&hca->lock); - dapl_llist_init_head(&hca->ia_list_head); + INIT_LIST_HEAD(&hca->ia_list); hca->name = dapl_os_strdup(name); hca->ib_hca_handle = device; @@ -122,8 +122,7 @@ void dapl_hca_free(struct dapl_hca *hca) void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia_ptr) { spin_lock_irqsave(&hca->lock, hca->flags); - dapl_llist_add_head(&hca->ia_list_head, - &ia_ptr->hca_ia_list_entry, ia_ptr); + list_add(&ia_ptr->list, &hca->ia_list); spin_unlock_irqrestore(&hca->lock, hca->flags); } @@ -143,17 +142,9 @@ void dapl_hca_link_ia(struct dapl_hca *h * none * */ -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia_ptr) +void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia) { spin_lock_irqsave(&hca->lock, hca->flags); - /* - * If an error occurred when we were opening the IA it - * will not be linked on the list; don't unlink an unlinked - * list! - */ - if (!dapl_llist_is_empty(&hca->ia_list_head)) { - dapl_llist_remove_entry(&hca->ia_list_head, - &ia_ptr->hca_ia_list_entry); - } + list_del(&ia->list); spin_unlock_irqrestore(&hca->lock, hca->flags); } diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll8/dat-provider/dapl_ia.c linux-kernel-ll9/dat-provider/dapl_ia.c --- linux-kernel-ll8/dat-provider/dapl_ia.c 2005-06-13 16:50:30.624000000 -0700 +++ linux-kernel-ll9/dat-provider/dapl_ia.c 2005-06-13 16:56:10.004000000 -0700 @@ -62,7 +62,6 @@ struct dapl_ia *dapl_ia_alloc(struct dat ia->hca = hca; ia->async_error_evd = NULL; ia->cleanup_async_error_evd = FALSE; - dapl_llist_init_entry(&ia->hca_ia_list_entry); INIT_LIST_HEAD(&ia->ep_list); INIT_LIST_HEAD(&ia->lmr_list); INIT_LIST_HEAD(&ia->rmr_list); From Tom.Duffy at Sun.COM Mon Jun 13 17:18:10 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 13 Jun 2005 17:18:10 -0700 Subject: [openib-general] [PATCH 9/9] kDAPL: remove all vestiges of dapls linked list implementation In-Reply-To: <11187082903037@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> <1118708288856@sun.com> <11187082893179@sun.com> <1118708290446@sun.com> <11187082903037@sun.com> Message-ID: <11187082902409@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_cno_util.c linux-kernel-ll10/dat-provider/dapl_cno_util.c --- linux-kernel-ll9/dat-provider/dapl_cno_util.c 2005-06-02 13:23:50.481015000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_cno_util.c 2005-06-13 17:02:22.166004000 -0700 @@ -54,7 +54,6 @@ struct dapl_cno *dapl_cno_alloc(struct d cno_ptr->common.owner_ia = ia_ptr; cno_ptr->common.user_context.as_64 = 0; cno_ptr->common.user_context.as_ptr = NULL; - dapl_llist_init_entry(&cno_ptr->common.ia_list_entry); spin_lock_init(&cno_ptr->common.lock); /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_cr.c linux-kernel-ll10/dat-provider/dapl_cr.c --- linux-kernel-ll9/dat-provider/dapl_cr.c 2005-06-10 16:13:19.912031000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_cr.c 2005-06-13 17:02:01.638003000 -0700 @@ -54,7 +54,6 @@ struct dapl_cr *dapl_cr_alloc(struct dap cr->cr.common.provider = ia->ia.common.provider; cr->common.owner_ia = ia; - dapl_llist_init_entry(&cr->common.ia_list_entry); spin_lock_init(&cr->common.lock); return cr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_ep.c linux-kernel-ll10/dat-provider/dapl_ep.c --- linux-kernel-ll9/dat-provider/dapl_ep.c 2005-06-10 16:13:19.878004000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_ep.c 2005-06-13 17:02:26.352003000 -0700 @@ -70,7 +70,6 @@ struct dapl_ep *dapl_ep_alloc(struct dap ep_ptr->ep.common.provider = ia_ptr->ia.common.provider; ep_ptr->common.owner_ia = ia_ptr; - dapl_llist_init_entry(&ep_ptr->common.ia_list_entry); spin_lock_init(&ep_ptr->common.lock); /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_ep_util.c linux-kernel-ll10/dat-provider/dapl_ep_util.c --- linux-kernel-ll9/dat-provider/dapl_ep_util.c 2005-06-03 09:38:29.715962000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_ep_util.c 2005-06-13 17:02:29.294000000 -0700 @@ -67,7 +67,6 @@ struct dapl_ep *dapl_ep_alloc(struct dap ep_ptr->common.owner_ia = ia_ptr; ep_ptr->common.user_context.as_64 = 0; ep_ptr->common.user_context.as_ptr = NULL; - dapl_llist_init_entry(&ep_ptr->common.ia_list_entry); spin_lock_init(&ep_ptr->common.lock); /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_evd.c linux-kernel-ll10/dat-provider/dapl_evd.c --- linux-kernel-ll9/dat-provider/dapl_evd.c 2005-06-10 16:16:31.918002000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_evd.c 2005-06-13 17:02:34.929003000 -0700 @@ -159,7 +159,6 @@ struct dapl_evd *dapl_evd_alloc(struct d evd->evd.common.provider = ia->ia.common.provider; evd->common.owner_ia = ia; - dapl_llist_init_entry(&evd->common.ia_list_entry); spin_lock_init(&evd->common.lock); /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_evd_util.c linux-kernel-ll10/dat-provider/dapl_evd_util.c --- linux-kernel-ll9/dat-provider/dapl_evd_util.c 2005-06-02 13:48:19.237003000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_evd_util.c 2005-06-13 17:02:37.774000000 -0700 @@ -135,7 +135,6 @@ struct dapl_evd *dapl_evd_alloc(struct d evd_ptr->common.owner_ia = ia_ptr; evd_ptr->common.user_context.as_64 = 0; evd_ptr->common.user_context.as_ptr = NULL; - dapl_llist_init_entry(&evd_ptr->common.ia_list_entry); spin_lock_init(&evd_ptr->common.lock); /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl.h linux-kernel-ll10/dat-provider/dapl.h --- linux-kernel-ll9/dat-provider/dapl.h 2005-06-13 16:53:00.300010000 -0700 +++ linux-kernel-ll10/dat-provider/dapl.h 2005-06-13 17:01:22.766003000 -0700 @@ -92,13 +92,6 @@ typedef void *DAPL_HASH_DATA; * * *********************************************************************/ -struct dapl_llist_entry { - struct dapl_llist_entry *flink; - struct dapl_llist_entry *blink; - void *data; - struct dapl_llist_entry **list_head; /* for consistency checking */ -}; - struct dapl_ring_buffer { void **base; /* base of element array */ int lim; /* mask, number of entries - 1 */ @@ -141,7 +134,6 @@ struct dapl_hca { /* DAPL Objects always have the following items */ struct dapl_common { struct dapl_ia *owner_ia; - struct dapl_llist_entry ia_list_entry; spinlock_t lock; unsigned long flags; /* saved lock flag values */ }; @@ -614,39 +606,4 @@ extern u32 dapl_srq_resize(struct dat_sr extern u32 dapl_srq_set_lw(struct dat_srq *srq, int low_watermark); -/* - * DAPL internal utility function prototypes - */ - -extern void dapl_llist_init_head(struct dapl_llist_entry **head); - -extern void dapl_llist_init_entry(struct dapl_llist_entry *entry); - -extern boolean_t dapl_llist_is_empty(struct dapl_llist_entry **head); - -extern void dapl_llist_add_head(struct dapl_llist_entry **head, - struct dapl_llist_entry *entry, void *data); - -extern void dapl_llist_add_tail(struct dapl_llist_entry **head, - struct dapl_llist_entry *entry, void *data); - -extern void dapl_llist_add_entry(struct dapl_llist_entry **head, - struct dapl_llist_entry *entry, - struct dapl_llist_entry *new_entry, - void *data); - -extern void *dapl_llist_remove_head(struct dapl_llist_entry **head); - -extern void *dapl_llist_remove_tail(struct dapl_llist_entry **head); - -extern void *dapl_llist_remove_entry(struct dapl_llist_entry **head, - struct dapl_llist_entry *entry); - -extern void *dapl_llist_peek_head(struct dapl_llist_entry **head); - -extern void *dapl_llist_next_entry(struct dapl_llist_entry **head, - struct dapl_llist_entry *cur_ent); - -extern void dapl_llist_debug_print_list(struct dapl_llist_entry **head); - #endif diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_ia.c linux-kernel-ll10/dat-provider/dapl_ia.c --- linux-kernel-ll9/dat-provider/dapl_ia.c 2005-06-13 16:56:10.004000000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_ia.c 2005-06-13 17:02:40.414002000 -0700 @@ -53,7 +53,6 @@ struct dapl_ia *dapl_ia_alloc(struct dat ia->ia.common.provider = provider; ia->common.owner_ia = ia; - dapl_llist_init_entry(&ia->common.ia_list_entry); spin_lock_init(&ia->common.lock); /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_llist.c linux-kernel-ll10/dat-provider/dapl_llist.c --- linux-kernel-ll9/dat-provider/dapl_llist.c 2005-05-20 22:44:36.215051000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_llist.c 1969-12-31 16:00:00.000000000 -0800 @@ -1,321 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * A link list head points to the first member of a linked list, but - * is itself not a member of the list. - * - * +---------------------------------------------+ - * | entry entry entry | - * HEAD -> | +-------+ +-------+ +-------+ | - * +--> | flink | --> | flink | --> | flink | >--+ - * | data | | data | | data | - * +--< | blink | <-- | blink | <-- | blink | <--| - * | +-------+ +-------+ +-------+ | - * | | - * +---------------------------------------------+ - * - * Note: Each of the remove functions takes an assertion failure if - * an element cannot be removed from the list. - * - * $Id: dapl_llist.c 2433 2005-05-21 04:11:03Z jlentini $ - */ - -#include "dapl.h" - -/* - * dapl_llist_init_head() - * - * Purpose: initialize a linked list head - */ -void dapl_llist_init_head(struct dapl_llist_entry **head) -{ - *head = NULL; -} - -/* - * dapl_llist_init_entry() - * - * Purpose: initialize a linked list entry - */ -void dapl_llist_init_entry(struct dapl_llist_entry *entry) -{ - entry->blink = NULL; - entry->flink = NULL; - entry->data = NULL; - entry->list_head = NULL; -} - -/* - * dapl_llist_is_empty() - * - * Purpose: returns TRUE if the linked list is empty - */ -boolean_t dapl_llist_is_empty(struct dapl_llist_entry **head) -{ - return (*head == NULL); -} - -/* - * dapl_llist_add_head() - * - * Purpose: Add an entry to the head of a linked list - */ -void -dapl_llist_add_head(struct dapl_llist_entry **head, - struct dapl_llist_entry *entry, void *data) -{ - struct dapl_llist_entry *first; - - /* deal with empty list */ - if (dapl_llist_is_empty(head)) { - entry->flink = entry; - entry->blink = entry; - } else { - first = *head; - entry->flink = first; - entry->blink = first->blink; - first->blink->flink = entry; - first->blink = entry; - } - - *head = entry; - entry->data = data; - entry->list_head = head; -} - -/* - * dapl_llist_add_tail() - * - * Purpose: Add an entry to the tail of a linked list - */ -void -dapl_llist_add_tail(struct dapl_llist_entry **head, - struct dapl_llist_entry *entry, void *data) -{ - struct dapl_llist_entry *last; - - /* deal with empty list */ - if (dapl_llist_is_empty(head)) { - *head = entry; - entry->flink = entry; - entry->blink = entry; - } else { - last = (*head)->blink; - entry->flink = last->flink; - entry->blink = last; - last->flink->blink = entry; - last->flink = entry; - } - entry->data = data; - entry->list_head = head; -} - -/* - * dapl_llist_add_entry() - * - * Purpose: Add an entry before an element in the list. Does - * not verify the list or the validity of the entries passed in. - */ -void -dapl_llist_add_entry(struct dapl_llist_entry **head, - struct dapl_llist_entry *entry, - struct dapl_llist_entry *new_entry, void *data) -{ - struct dapl_llist_entry *first; - struct dapl_llist_entry *prev; - - /* get the head pointer */ - first = *head; - - /* link the entry into the queue */ - prev = entry->blink; - entry->blink = new_entry; - prev->flink = new_entry; - - new_entry->flink = entry; - new_entry->blink = prev; - - new_entry->data = data; - new_entry->list_head = head; - - /* If the old entry was the head of the list, adjust */ - if (first == entry) { - (*head) = new_entry; - } - -} - -/* - * dapl_llist_remove_head() - * - * Purpose: Remove the first entry of a linked list - */ -void *dapl_llist_remove_head(struct dapl_llist_entry **head) -{ - struct dapl_llist_entry *first; - - dapl_os_assert(!dapl_llist_is_empty(head)); - first = *head; - *head = first->flink; - - first->flink->blink = first->blink; - first->blink->flink = first->flink; - - if (first->flink == first) { - *head = NULL; - } - /* clean up the links for good measure */ - first->flink = NULL; - first->blink = NULL; - first->list_head = NULL; - return first->data; -} - -/* - * dapl_llist_remove_tail() - * - * Purpose: Remove the last entry of a linked list - */ -void *dapl_llist_remove_tail(struct dapl_llist_entry **head) -{ - struct dapl_llist_entry *last; - - dapl_os_assert(!dapl_llist_is_empty(head)); - last = (*head)->blink; - - last->blink->flink = last->flink; - last->flink->blink = last->blink; - - if (last->flink == last) { - *head = NULL; - } - /* clean up the links for good measure */ - last->flink = NULL; - last->blink = NULL; - last->list_head = NULL; - - return last->data; -} - -/* - * dapl_llist_remove_entry() - * - * Purpose: Remove the specified entry from a linked list - */ -void *dapl_llist_remove_entry(struct dapl_llist_entry **head, - struct dapl_llist_entry *entry) -{ - struct dapl_llist_entry *first; - - dapl_os_assert(!dapl_llist_is_empty(head)); - first = *head; - - /* if it's the first entry, pull it off */ - if (first == entry) { - (*head) = first->flink; - /* if it was the only entry, kill the list */ - if (first->flink == first) { - (*head) = NULL; - } - } - - dapl_os_assert(entry->list_head == head); - entry->list_head = NULL; - - entry->flink->blink = entry->blink; - entry->blink->flink = entry->flink; - entry->flink = NULL; - entry->blink = NULL; - - return entry->data; -} - -/* - * dapl_llist_peek_head - */ - -void *dapl_llist_peek_head(struct dapl_llist_entry **head) -{ - struct dapl_llist_entry *first; - - dapl_os_assert(!dapl_llist_is_empty(head)); - first = *head; - return first->data; -} - -/* - * dapl_llist_next_entry - * - * Obtain the next entry in the list, return NULL when we get to the - * head - */ - -void *dapl_llist_next_entry(struct dapl_llist_entry **head, - struct dapl_llist_entry *cur_ent) -{ - struct dapl_llist_entry *next; - - dapl_os_assert(!dapl_llist_is_empty(head)); - if (cur_ent == NULL) { - next = *head; - } else { - next = cur_ent->flink; - if (next == *head) { - return NULL; - } - } - return next->data; -} - -#ifdef DAPL_DBG -/* - * dapl_llist_debug_print_list() - * - * Purpose: Prints the linked list for debugging - */ -void dapl_llist_debug_print_list(struct dapl_llist_entry **head) -{ - struct dapl_llist_entry *entry; - struct dapl_llist_entry *first; - first = *head; - if (!first) { - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "EMPTY_LIST\n"); - return; - } - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "HEAD %p\n", *head); - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "Entry %p %p %p %p\n", - first, first->flink, first->blink, first->data); - entry = first->flink; - while (entry != first) { - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "Entry %p %p %p %p\n", - entry, entry->flink, entry->blink, entry->data); - entry = entry->flink; - } -} - -#endif /* DAPL_DBG */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_lmr.c linux-kernel-ll10/dat-provider/dapl_lmr.c --- linux-kernel-ll9/dat-provider/dapl_lmr.c 2005-06-10 16:13:19.666059000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_lmr.c 2005-06-13 17:02:49.115001000 -0700 @@ -51,7 +51,6 @@ static struct dapl_lmr *dapl_lmr_alloc(s lmr->lmr.common.provider = ia->ia.common.provider; lmr->common.owner_ia = ia; - dapl_llist_init_entry(&lmr->common.ia_list_entry); dapl_ia_link_lmr(ia, lmr); spin_lock_init(&lmr->common.lock); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_pz.c linux-kernel-ll10/dat-provider/dapl_pz.c --- linux-kernel-ll9/dat-provider/dapl_pz.c 2005-06-10 16:13:19.546018000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_pz.c 2005-06-13 17:02:54.055003000 -0700 @@ -46,7 +46,6 @@ static struct dapl_pz *dapl_pz_alloc(str pz->pz.common.provider = ia->ia.common.provider; pz->common.owner_ia = ia; - dapl_llist_init_entry(&pz->common.ia_list_entry); dapl_ia_link_pz(ia, pz); spin_lock_init(&pz->common.lock); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_rmr.c linux-kernel-ll10/dat-provider/dapl_rmr.c --- linux-kernel-ll9/dat-provider/dapl_rmr.c 2005-06-10 16:13:19.832021000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_rmr.c 2005-06-13 17:02:57.553009000 -0700 @@ -63,7 +63,6 @@ static struct dapl_rmr *dapl_rmr_alloc(s rmr->rmr.common.provider = pz->pz.common.provider; rmr->common.owner_ia = pz->common.owner_ia; - dapl_llist_init_entry(&rmr->common.ia_list_entry); dapl_ia_link_rmr(rmr->common.owner_ia, rmr); spin_lock_init(&rmr->common.lock); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_sp.c linux-kernel-ll10/dat-provider/dapl_sp.c --- linux-kernel-ll9/dat-provider/dapl_sp.c 2005-06-13 14:52:58.454000000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_sp.c 2005-06-13 17:03:00.514002000 -0700 @@ -54,7 +54,6 @@ struct dapl_sp *dapl_sp_alloc(struct dap else sp->sp.type = DAT_SP_TYPE_RSP; sp->common.owner_ia = ia; - dapl_llist_init_entry(&sp->common.ia_list_entry); spin_lock_init(&sp->common.lock); INIT_LIST_HEAD(&sp->cr_list); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll9/dat-provider/dapl_srq.c linux-kernel-ll10/dat-provider/dapl_srq.c --- linux-kernel-ll9/dat-provider/dapl_srq.c 2005-06-10 10:49:26.886965000 -0700 +++ linux-kernel-ll10/dat-provider/dapl_srq.c 2005-06-13 17:03:05.631006000 -0700 @@ -90,8 +90,6 @@ static struct dapl_srq *dapl_srq_alloc(s srq_ptr->srq.common.provider = ia_ptr->ia.common.provider; srq_ptr->common.owner_ia = ia_ptr; atomic_set(&srq_ptr->srq_ref_count, 0); - - dapl_llist_init_entry(&srq_ptr->common.ia_list_entry); spin_lock_init(&srq_ptr->common.lock); /* Index: linux-kernel-ll10/dat-provider/Makefile =================================================================== --- linux-kernel-ll10/dat-provider/Makefile (revision 2595) +++ linux-kernel-ll10/dat-provider/Makefile (working copy) @@ -24,7 +24,6 @@ PROVIDER_MODULES := \ dapl_hash \ dapl_hca_util \ dapl_ia \ - dapl_llist \ dapl_lmr \ dapl_mr_util \ dapl_provider \ From tduffy at sun.com Mon Jun 13 17:25:44 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 13 Jun 2005 17:25:44 -0700 Subject: [openib-general] dapl build problems In-Reply-To: <1118706314.17676.28.camel@localhost> References: <1118706314.17676.28.camel@localhost> Message-ID: <1118708744.24800.1.camel@duffman> On Mon, 2005-06-13 at 16:45 -0700, Josh England wrote: > I get these problems trying to build dat with 2.6.11.11 (svn rev 2595): > > *** Warning: > "dat_registry_add_provider" [drivers/infiniband/ulp/dat-provider/ib_dat_provider.ko] undefined! > *** Warning: > "dat_registry_remove_provider" [drivers/infiniband/ulp/dat-provider/ib_dat_provider.ko] undefined! That is fine. These are provided by dat.ko. Make sure to load that module before ib_dat_provider.ko. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From iod00d at hp.com Mon Jun 13 20:51:10 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 13 Jun 2005 20:51:10 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <20050613211226.GA31086@lst.de> References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> Message-ID: <20050614035110.GE20498@esmail.cup.hp.com> On Mon, Jun 13, 2005 at 11:12:26PM +0200, Christoph Hellwig wrote: > Umm, no. Please try to get rid of dapl_common instead of adding to it. Why? The only reason I've seen so far is "it's brain damaged". I happen to agree with Tom Duffy that it doesn't cost anything extra. thanks, grant From iod00d at hp.com Mon Jun 13 21:35:32 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 13 Jun 2005 21:35:32 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <20050614035110.GE20498@esmail.cup.hp.com> References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> <20050614035110.GE20498@esmail.cup.hp.com> Message-ID: <20050614043532.GH20498@esmail.cup.hp.com> On Mon, Jun 13, 2005 at 08:51:10PM -0700, Grant Grundler wrote: > On Mon, Jun 13, 2005 at 11:12:26PM +0200, Christoph Hellwig wrote: > > Umm, no. Please try to get rid of dapl_common instead of adding to it. > > Why? > The only reason I've seen so far is "it's brain damaged". > I happen to agree with Tom Duffy that it doesn't cost > anything extra. nevermind...I just saw Tom's clarification on what's left (one pointer) dapl_common. grant From itamar at mellanox.co.il Tue Jun 14 05:49:13 2005 From: itamar at mellanox.co.il (Itamar) Date: Tue, 14 Jun 2005 15:49:13 +0300 Subject: [openib-general] [patch][kdapl] integrate dapl_mr_util function into dapl_rmr.c Message-ID: <20050614124913.GA7940@mellanox.co.il> Integrate dapl_mr_util functions into dapl_rmr.c Delete dapl_mr_util.[h|c] files Delete define DAPL_ATS in Makefile (not in use) Signed-off-by: Itamar Rabenstein Index: Makefile =================================================================== --- Makefile (revision 2595) +++ Makefile (working copy) @@ -9,7 +9,6 @@ endif EXTRA_CFLAGS += \ - -DDAPL_ATS \ -Idrivers/infiniband/include \ -Idrivers/dat @@ -26,7 +25,6 @@ dapl_ia \ dapl_llist \ dapl_lmr \ - dapl_mr_util \ dapl_provider \ dapl_pz \ dapl_ring_buffer_util \ Index: dapl_rmr.c =================================================================== --- dapl_rmr.c (revision 2595) +++ dapl_rmr.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_ep.h" #include "dapl_ia.h" -#include "dapl_mr_util.h" #include "dapl_hash.h" #include "dapl_cookie.h" #include "dapl_openib_util.h" @@ -92,6 +91,62 @@ kfree(rmr); } +/* + * dapl_mr_get_address + * + * Returns the memory address associated with the given memory descriptor + * + * Input: + * desc memory descriptor + * type type of memory represented by desc + * + * Output: + * None + * + */ +u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, enum dat_mem_type type) +{ + struct dapl_lmr *lmr; + + switch (type) { + case DAT_MEM_TYPE_VIRTUAL: + return (u64) (unsigned long) desc.for_va; + case DAT_MEM_TYPE_LMR: + lmr = (struct dapl_lmr *)desc.for_lmr; + + /* Since this function is recoursive we cannot inline it */ + return dapl_mr_get_address(lmr->param.region_desc, + lmr->param.mem_type); + case DAT_MEM_TYPE_PHYSICAL: + return desc.for_pa; + default: + /* + * The following kDAPL memory types have not been implemented: + * DAT_MEM_TYPE_PLATFORM + * DAT_MEM_TYPE_IA + * DAT_MEM_TYPE_BYPASS + */ + dapl_os_assert(0); + return 0; + } +} + +/* + * dapl_mr_bounds_check + * + * Returns true if region B is contained within region A + * and false otherwise + * + */ +static inline boolean_t dapl_mr_bounds_check(u64 addr_a, u64 length_a, + u64 addr_b, u64 length_b) +{ + if ((addr_a <= addr_b) && (addr_b + length_b) <= (addr_a + length_a)) + return TRUE; + else + return FALSE; +} + static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, Index: dapl_mr_util.c =================================================================== --- dapl_mr_util.c (revision 2595) +++ dapl_mr_util.c (working copy) @@ -1,79 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -#include "dapl_mr_util.h" - -/* - * $Id$ - */ - -/********************************************************************* - * * - * Function Definitions * - * * - *********************************************************************/ - -/* - * dapl_mr_get_address - * - * Returns the memory address associated with the given memory descriptor - * - * Input: - * desc memory descriptor - * type type of memory represented by desc - * - * Output: - * None - * - */ - -u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, enum dat_mem_type type) -{ - struct dapl_lmr *lmr; - - switch (type) { - case DAT_MEM_TYPE_VIRTUAL: - return (u64) (unsigned long) desc.for_va; - case DAT_MEM_TYPE_LMR: - lmr = (struct dapl_lmr *)desc.for_lmr; - - /* Since this function is recoursive we cannot inline it */ - return dapl_mr_get_address(lmr->param.region_desc, - lmr->param.mem_type); - case DAT_MEM_TYPE_PHYSICAL: - return desc.for_pa; - default: - /* - * The following kDAPL memory types have not been implemented: - * DAT_MEM_TYPE_PLATFORM - * DAT_MEM_TYPE_IA - * DAT_MEM_TYPE_BYPASS - */ - dapl_os_assert(0); - return 0; - } -} Index: dapl_mr_util.h =================================================================== --- dapl_mr_util.h (revision 2595) +++ dapl_mr_util.h (working copy) @@ -1,58 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_MR_UTIL_H -#define DAPL_MR_UTIL_H - -#include "dapl.h" -#include "dapl_hash.h" - -extern u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, - enum dat_mem_type type); - -/* - * dapl_mr_bounds_check - * - * Returns true if region B is contained within region A - * and false otherwise - * - */ - -static inline boolean_t dapl_mr_bounds_check(u64 addr_a, u64 length_a, - u64 addr_b, u64 length_b) -{ - if ((addr_a <= addr_b) && (addr_b + length_b) <= (addr_a + length_a)) - return TRUE; - else - return FALSE; -} - -#endif /* DAPL_MR_UTIL_H */ Index: dapl_provider.c =================================================================== --- dapl_provider.c (revision 2595) +++ dapl_provider.c (working copy) @@ -36,7 +36,6 @@ #include "dapl.h" #include "dapl_hca_util.h" #include "dapl_provider.h" -#include "dapl_mr_util.h" #include "dapl_util.h" #include "dapl_openib_util.h" -- Itamar From halr at voltaire.com Tue Jun 14 05:41:43 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 08:41:43 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <1118664517.4476.9585.camel@localhost.localdomain> Message-ID: <1118752898.4506.530.camel@localhost.localdomain> On Mon, 2005-06-13 at 18:33, James Lentini wrote: > On Mon, 13 Jun 2005, Hal Rosenstock wrote: > > halr> On Wed, 2005-06-08 at 17:53, James Lentini wrote: > halr> > On Wed, 8 Jun 2005, Hal Rosenstock wrote: > halr> > > halr> > halr> On Wed, 2005-06-08 at 11:44, James Lentini wrote: > halr> > halr> > We interpreted the above to mean "give the connection protocol as > halr> > halr> > much time as it needs to establish a connection, but don't mask > halr> > halr> > errors (no path to the remove node, etc.)". For that reason we changed > halr> > halr> > the variable name to DAT_TIMEOUT_MAX. > halr> > halr> > halr> > halr> But if the REQ is lost, the timeout is really really long (longer than > halr> > halr> most will wait for an error). > halr> > > halr> > If a user doesn't want to wait DAT_TIMEOUT_MAX time, it can pass a > halr> > smaller amount of time to dat_ep_connect. Does this satisfy your > halr> > requirements? > halr> > halr> Is it the intended that the only way out is via user intervention (e.g. > halr> ctl-C) ? If one connection attempt (REQ) is made and it is lost, then > halr> there is no chance of it completing and the user needs to intervene. > > Why does the user need to intervene? Did I misunderstanding the CM > API? > > When dapl_ep_connect() is called with a timeout value of > DAT_TIMEOUT_MAX, DAPL passes ib_sen_cm_req the value 0x1F in the > ib_cm_req_param structure's remote_cm_response_timeout value. My > understanding was that this is the maximum timeout and that once it > expires the CM will inform the user that the REQ timed out. Yes but it is a long time (4.096 * 2 ^ 31 usec ~ 8796 sec ~ 146.60 min (if my calcs are correct)). This is longer than (most) users would wait. They would usually hit ctl-C before this timeout is reached. > halr> If that is the intended behavior, we are there. (This (lost REQ) > halr> can even occur when the timeout is non infinite too). > > We didn't intend for the active side to wait forever if a REQ was > lost. The active side has no way of knowing that the REQ was lost (other than timeout/retry) and when the timeout is long, this is effectively the case. > halr> An alternative (as Sean suggested) is to continually retry (at a > halr> periodicity below the supplied timeout) until the time period specified > halr> expires. That seems to be better (at least to me and Sean) in terms of > halr> handling the lost REQ case. As retries is not part of the API for > halr> connect, I would presume the implementor is free to what they want under > halr> the covers of dapl_ib_connect. > > You're correct. The current implementation is: 1. address resolution phase for some amount of time followed by: 2. dapl_ib_connect timeout * 5 (since there are 4 retries) A better algorithm would be to divide down the timeout by some number of retries (which would vary based on the timeout requested) and have the number of retries vary based on the total timeout requested. -- Hal From Thomas.Talpey at netapp.com Tue Jun 14 06:36:26 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Tue, 14 Jun 2005 09:36:26 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1118752898.4506.530.camel@localhost.localdomain> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <1118664517.4476.9585.camel@localhost.localdomain> <1118752898.4506.530.camel@localhost.localdomain> Message-ID: <6.2.1.2.2.20050614093043.04f3c390@exnane01.nane.netapp.com> At 08:41 AM 6/14/2005, Hal Rosenstock wrote: >The current implementation is: >1. address resolution phase for some amount of time >followed by: >2. dapl_ib_connect timeout * 5 (since there are 4 retries) > >A better algorithm would be to divide down the timeout by some number of >retries (which would vary based on the timeout requested) and have the >number of retries vary based on the total timeout requested. Why is address resolution exempt from the timeout? If the caller wants a timeout, it should be independent of low-level link resolution. Socket connect()s don't care about ARP, for example. I don't like the idea of retry counts because there is no deterministic length of time that they will take. Exponential backoff could drive even a few retries to many minutes. Of course, if an IB provider can guarantee that N retries will be performed in M seconds, then okay, but not in general. Tom. From halr at voltaire.com Tue Jun 14 06:49:15 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 09:49:15 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <6.2.1.2.2.20050614093043.04f3c390@exnane01.nane.netapp.com> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <1118664517.4476.9585.camel@localhost.localdomain> <1118752898.4506.530.camel@localhost.localdomain> <6.2.1.2.2.20050614093043.04f3c390@exnane01.nane.netapp.com> Message-ID: <1118756955.4506.559.camel@localhost.localdomain> On Tue, 2005-06-14 at 09:36, Talpey, Thomas wrote: > At 08:41 AM 6/14/2005, Hal Rosenstock wrote: > >The current implementation is: > >1. address resolution phase for some amount of time > >followed by: > >2. dapl_ib_connect timeout * 5 (since there are 4 retries) > > > >A better algorithm would be to divide down the timeout by some number of > >retries (which would vary based on the timeout requested) and have the > >number of retries vary based on the total timeout requested. > > Why is address resolution exempt from the timeout? If the caller > wants a timeout, it should be independent of low-level link resolution. > Socket connect()s don't care about ARP, for example. I was just stating the way the algorithm is right now. The address resolution phase can be included in the calculation but this complicates things a little. I thought it was previously said that the timeout can be approximate. Also, the CM timeouts are approximate and not precise either. > I don't like the idea of retry counts because there is no deterministic > length of time that they will take. Are you proposing that the number of retries be set to 0 then (regardless of the timeout requested) ? > Exponential backoff could drive > even a few retries to many minutes. Of course, if an IB provider > can guarantee that N retries will be performed in M seconds, then > okay, but not in general. The CM is not using exponential backoff. -- Hal From Thomas.Talpey at netapp.com Tue Jun 14 07:00:16 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Tue, 14 Jun 2005 10:00:16 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1118756955.4506.559.camel@localhost.localdomain> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <1118664517.4476.9585.camel@localhost.localdomain> <1118752898.4506.530.camel@localhost.localdomain> <6.2.1.2.2.20050614093043.04f3c390@exnane01.nane.netapp.com> <1118756955.4506.559.camel@localhost.localdomain> Message-ID: <6.2.1.2.2.20050614095816.04ce1bc0@exnane01.nane.netapp.com> At 09:49 AM 6/14/2005, Hal Rosenstock wrote: >Are you proposing that the number of retries be set to 0 then >(regardless of the timeout requested) ? All I am suggesting is that the number of retries is not something the consumer can or should be specifying. Whatever the appropriate number is, is something for the transport to choose. It's an internal detail. >The CM is not using exponential backoff. Okay, though I would suggest it should. In any case, iWARP (TCP) does, and that's important to bear in mind. Tom. From halr at voltaire.com Tue Jun 14 07:01:32 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 10:01:32 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <6.2.1.2.2.20050614095816.04ce1bc0@exnane01.nane.netapp.com> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <1118664517.4476.9585.camel@localhost.localdomain> <1118752898.4506.530.camel@localhost.localdomain> <6.2.1.2.2.20050614093043.04f3c390@exnane01.nane.netapp.com> <1118756955.4506.559.camel@localhost.localdomain> <6.2.1.2.2.20050614095816.04ce1bc0@exnane01.nane.netapp.com> Message-ID: <1118757692.4506.565.camel@localhost.localdomain> On Tue, 2005-06-14 at 10:00, Talpey, Thomas wrote: > At 09:49 AM 6/14/2005, Hal Rosenstock wrote: > >Are you proposing that the number of retries be set to 0 then > >(regardless of the timeout requested) ? > > All I am suggesting is that the number of retries is not something the > consumer can or should be specifying. Whatever the appropriate > number is, is something for the transport to choose. It's an > internal detail. Yes, I was proposing that that this be calculated internally based on the requested timeout. Sorry if that was not clear. > >The CM is not using exponential backoff. > > Okay, though I would suggest it should. In any case, iWARP (TCP) > does, and that's important to bear in mind. It could easily be made to do this. What do others think about this ? -- Hal From jlentini at netapp.com Tue Jun 14 07:58:47 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 10:58:47 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flags In-Reply-To: <20050611093144.GA18469@lst.de> References: <20050611093144.GA18469@lst.de> Message-ID: I think this is a good change. If we trim the dat_provider_info structure down to just an IA name, then we should be able to remove it all together. We don't have to do that on this pass though. I have a question about the change you've proposed for the makefile: On Sat, 11 Jun 2005, Christoph Hellwig wrote: > Index: dat/Makefile > =================================================================== > --- dat/Makefile (revision 2588) > +++ dat/Makefile (working copy) > @@ -1,7 +1,6 @@ > > EXTRA_CFLAGS += \ > - -Idrivers/dat \ > - -Werror > + -I$(obj) Is $(obj) the directory containing the object files (drivers/dat)? Is the -Werror flag already being setup by kbuild? > > obj-$(CONFIG_DAT) += dat.o > From tduffy at sun.com Tue Jun 14 08:02:22 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 08:02:22 -0700 Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118708297.17672.32.camel@localhost> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> Message-ID: <1118761342.12164.0.camel@duffman> On Mon, 2005-06-13 at 17:18 -0700, Josh England wrote: > I can't get DAT (provider) to build with the latest rev. I'll try again > later. Besides the symbol warnings, what errors are you seeing? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Tue Jun 14 08:24:42 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 11:24:42 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <1118698530.9114.16.camel@duffman> References: <1118437034.29601.6.camel@duffman> <1118698530.9114.16.camel@duffman> Message-ID: On Mon, 13 Jun 2005, Tom Duffy wrote: > On Mon, 2005-06-13 at 17:10 -0400, James Lentini wrote: >> This looks good. I've checked it in as revision 2595. > > Thanks. > >> Eventually, we could move the list up into the dapl_common structure. > > Gross. I agree with Christoph. You find the dapl_common structure gross from a style point of view, right? I agree that some of the fields currently in dapl_common are not actually necessary for every DAPL object (e.g. a dapl_ia does not need an owner_ia pointer). I see these as implementation errors, but I still see value in using a common structure in principal. Is there a performance problem that I am missing? >> The one oddity is that all the objects are put on the IA list except >> CRs which are placed on an SP. I think it would be worth reflecting >> this in the list's member name. Instead of calling it a ia_list_entry >> as it is now, a name like parent_list or owner_list would be more >> appropriate. > > Eventually, wouldn't we get rid of ia_list_entry? Cause each struct has > its own list entry (name "list", which seems to be the convention), and > the IA has the heads of each list. That would be the other alternative. Right now, dapl_common has an ia_list_entry member. For each DAPL object, this list entry that gets linked on a list head stored in the parent IA. There are two exceptions to this rule: - a CR's ia_list_entry is linked onto its parent SP's cr list. - an IA's ia_list_entry is unused. It has a separate hca_ia_list_entry that it links onto its parent HCA's ia list Obviously this is confusing; when I looked at it yesterday I didn't realize that an IA's ia_list_entry was unused. My suggestion yesterday was to replace dapl_common's ia_list_entry member with a list_head (and remove the list_head's in the individual dapl object). The above option is moot if we remove dapl_common all together. > Your wording is a bit confusing cause you say, "all the objects are put > on the IA list". Shouldn't that read, "each of the objects is put on > its respective list owned by the IA" or some such? Your right, my wording was poor. You description is correct. > > -tduffy > From jlentini at netapp.com Tue Jun 14 08:29:49 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 11:29:49 -0400 (EDT) Subject: [openib-general] Re: [PATCH] DAPL: update TODO In-Reply-To: <1118690241.1985.7.camel@duffman> References: <1118690241.1985.7.camel@duffman> Message-ID: Committed in revision 2596. On Mon, 13 Jun 2005, Tom Duffy wrote: tduffy> Update the TODO so we don't lose track of needs. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: TODO tduffy> =================================================================== tduffy> --- TODO (revision 2588) tduffy> +++ TODO (working copy) tduffy> @@ -3,6 +3,8 @@ tduffy> DAT tduffy> *) change dat function to use errno return code as all linux kernel export tduffy> functions tduffy> +*) remove the dat_func() wrappers and have code call the functions from the tduffy> + method table directly tduffy> tduffy> IB DAT PROVIDER tduffy> tduffy> @@ -23,3 +25,5 @@ IB DAT PROVIDER tduffy> *) cache remote peer's address so that it does not need to be retrieved in tduffy> dapl_ep_query. tduffy> *) Replace dapl linked lists with Linux linked lists tduffy> +*) don't check for NULL all over the place -- if somebody calls a function tduffy> + wrong and breaks things, let them keep both halves tduffy> From halr at voltaire.com Tue Jun 14 08:28:01 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 11:28:01 -0400 Subject: [openib-general] Some more ucm questions Message-ID: <1118762880.4506.620.camel@localhost.localdomain> Hi Libor, In the ucm kernel module, in the routine ib_ucm_event, after the user: label, if various errors occur relating to copying back to userspace, all that is currently done is that the mutex is released and the result returned. Doesn't there need to be more cleanup here depending on which error occured ? (Won't the event be left at the beginning of the list and the ucm would never get past this in terms of any other events ?) Also, should there be a declaration ib_cm_event_get_timed be in the user space include file ? Thanks. -- Hal From caitlin.bestler at gmail.com Tue Jun 14 08:30:59 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 14 Jun 2005 08:30:59 -0700 Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flags In-Reply-To: References: <20050611093144.GA18469@lst.de> Message-ID: <469958e0050614083016bf7425@mail.gmail.com> If you eliminate the version info how would someone test DAT 1.3 code? Is support for 1.3 just supposed to magically appear in the mainline at some future date without intermediate testing? On 6/14/05, James Lentini wrote: > > I think this is a good change. If we trim the dat_provider_info > structure down to just an IA name, then we should be able to remove it > all together. We don't have to do that on this pass though. > > I have a question about the change you've proposed for the makefile: > > On Sat, 11 Jun 2005, Christoph Hellwig wrote: > > > Index: dat/Makefile > > =================================================================== > > --- dat/Makefile (revision 2588) > > +++ dat/Makefile (working copy) > > @@ -1,7 +1,6 @@ > > > > EXTRA_CFLAGS += \ > > - -Idrivers/dat \ > > - -Werror > > + -I$(obj) > > Is $(obj) the directory containing the object files (drivers/dat)? > > Is the -Werror flag already being setup by kbuild? > > > > > obj-$(CONFIG_DAT) += dat.o > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From jlentini at netapp.com Tue Jun 14 08:32:39 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 11:32:39 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <20050613211226.GA31086@lst.de> References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> Message-ID: On Mon, 13 Jun 2005, Christoph Hellwig wrote: > On Mon, Jun 13, 2005 at 05:10:31PM -0400, James Lentini wrote: >> >> Tom. >> >> This looks good. I've checked it in as revision 2595. >> >> Eventually, we could move the list up into the dapl_common structure. >> The one oddity is that all the objects are put on the IA list except >> CRs which are placed on an SP. I think it would be worth reflecting >> this in the list's member name. Instead of calling it a ia_list_entry >> as it is now, a name like parent_list or owner_list would be more >> appropriate. > > Umm, no. Please try to get rid of dapl_common instead of adding to it. Can you elaborate on this? Is this a style or performance issue? From jlentini at netapp.com Tue Jun 14 08:45:38 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 11:45:38 -0400 (EDT) Subject: [openib-general] dapl build problems In-Reply-To: <1118708744.24800.1.camel@duffman> References: <1118706314.17676.28.camel@localhost> <1118708744.24800.1.camel@duffman> Message-ID: On Mon, 13 Jun 2005, Tom Duffy wrote: > On Mon, 2005-06-13 at 16:45 -0700, Josh England wrote: >> I get these problems trying to build dat with 2.6.11.11 (svn rev 2595): >> >> *** Warning: >> "dat_registry_add_provider" [drivers/infiniband/ulp/dat-provider/ib_dat_provider.ko] undefined! >> *** Warning: >> "dat_registry_remove_provider" [drivers/infiniband/ulp/dat-provider/ib_dat_provider.ko] undefined! > > That is fine. These are provided by dat.ko. Make sure to load that > module before ib_dat_provider.ko. Josh, Were you attempting to load, not build, the module? If that is the case, you can load dat.ko with a "modprobe dat.ko". james From jlentini at netapp.com Tue Jun 14 08:46:41 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 11:46:41 -0400 (EDT) Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118708297.17672.32.camel@localhost> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> Message-ID: On Mon, 13 Jun 2005, Josh England wrote: > On Thu, 2005-06-09 at 14:35 -0400, James Lentini wrote: >> Josh, >> >> Was there any information about what occurred? >> >> Loading the the DAT modules with >> >> modprobe dat dbg_mask=0xff >> modprobe ib_dat_provider dbg_mask=0xffff >> >> will produce a lot of debug information that might help. >> >> james > > I can't get DAT (provider) to build with the latest rev. I'll try again > later. Let me know if you need any help building. > > -JE > >> On Thu, 9 Jun 2005, Itamar Rabenstein wrote: >> >>> No , I dont see the problem at my system . >>> >>> Itamar >>> >>>> -----Original Message----- >>>> From: James Lentini [mailto:jlentini at netapp.com] >>>> Sent: Thursday, June 09, 2005 6:03 PM >>>> To: Itamar Rabenstein >>>> Cc: Josh England; openib-general >>>> Subject: RE: [openib-general] kdaptest wedges server >>>> >>>> >>>> >>>> Itamar, >>>> >>>> Were you able to get to the bottom of this? >>>> >>>> james >>>> >>>> On Thu, 2 Jun 2005, Itamar Rabenstein wrote: >>>> >>>> itamar>Hi Josh, >>>> itamar> >>>> itamar>Can you run it with debug option ( -d ) >>>> itamar>and tell me what is the last debug print on the server side ? >>>> itamar>when you say "kill" can you hit ^c or is it hanging >>>> the station ? >>>> itamar> >>>> itamar>what is system configuration ? (arch , Os ,kernel) >>>> itamar> >>>> itamar> Itamar >>>> itamar> >>>> itamar> >>>> itamar>> -----Original Message----- >>>> itamar>> From: Josh England [mailto:jjengla at sandia.gov] >>>> itamar>> Sent: Thursday, June 02, 2005 9:00 PM >>>> itamar>> To: openib-general >>>> itamar>> Subject: [openib-general] kdaptest wedges server >>>> itamar>> >>>> itamar>> >>>> itamar>> Hi, >>>> itamar>> >>>> itamar>> The kdapltest program wedges consistently wedges a >>>> node. I just tried >>>> itamar>> it with Revision 2523, and it runs fine the first couple of >>>> itamar>> times. The >>>> itamar>> third time took a while, and the fourth time killed >>>> the server. >>>> itamar>> >>>> itamar>> The server is running: >>>> itamar>> kdapltest -T S -D mthca0a >>>> itamar>> >>>> itamar>> The client runs this several times in succession: >>>> itamar>> kdapltest -T T -D mthca0a -s client SR 256 >>>> server SR 256 >>>> itamar>> >>>> itamar>> -JE >>>> itamar>> >>>> itamar>> >>>> itamar>> _______________________________________________ >>>> itamar>> openib-general mailing list >>>> itamar>> openib-general at openib.org >>>> itamar>> http://openib.org/mailman/listinfo/openib-general >>>> itamar>> >>>> itamar>> To unsubscribe, please visit >>>> itamar>> http://openib.org/mailman/listinfo/openib-general >>>> itamar>> >>>> itamar>_______________________________________________ >>>> itamar>openib-general mailing list >>>> itamar>openib-general at openib.org >>>> itamar>http://openib.org/mailman/listinfo/openib-general >>>> itamar> >>>> itamar>To unsubscribe, please visit >>>> http://openib.org/mailman/listinfo/openib-general >>>> itamar> >>>> >>> >> > From jlentini at netapp.com Tue Jun 14 09:08:51 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 12:08:51 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1118752898.4506.530.camel@localhost.localdomain> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <1118664517.4476.9585.camel@localhost.localdomain> <1118752898.4506.530.camel@localhost.localdomain> Message-ID: On Tue, 14 Jun 2005, Hal Rosenstock wrote: > On Mon, 2005-06-13 at 18:33, James Lentini wrote: >> On Mon, 13 Jun 2005, Hal Rosenstock wrote: >> >> halr> On Wed, 2005-06-08 at 17:53, James Lentini wrote: >> halr> > On Wed, 8 Jun 2005, Hal Rosenstock wrote: >> halr> > >> halr> > halr> On Wed, 2005-06-08 at 11:44, James Lentini wrote: >> halr> > halr> > We interpreted the above to mean "give the connection protocol as >> halr> > halr> > much time as it needs to establish a connection, but don't mask >> halr> > halr> > errors (no path to the remove node, etc.)". For that reason we changed >> halr> > halr> > the variable name to DAT_TIMEOUT_MAX. >> halr> > halr> >> halr> > halr> But if the REQ is lost, the timeout is really really long (longer than >> halr> > halr> most will wait for an error). >> halr> > >> halr> > If a user doesn't want to wait DAT_TIMEOUT_MAX time, it can pass a >> halr> > smaller amount of time to dat_ep_connect. Does this satisfy your >> halr> > requirements? >> halr> >> halr> Is it the intended that the only way out is via user intervention (e.g. >> halr> ctl-C) ? If one connection attempt (REQ) is made and it is lost, then >> halr> there is no chance of it completing and the user needs to intervene. >> >> Why does the user need to intervene? Did I misunderstanding the CM >> API? >> >> When dapl_ep_connect() is called with a timeout value of >> DAT_TIMEOUT_MAX, DAPL passes ib_sen_cm_req the value 0x1F in the >> ib_cm_req_param structure's remote_cm_response_timeout value. My >> understanding was that this is the maximum timeout and that once it >> expires the CM will inform the user that the REQ timed out. > > Yes but it is a long time (4.096 * 2 ^ 31 usec ~ 8796 sec ~ 146.60 min > (if my calcs are correct)). This is longer than (most) users would wait. > They would usually hit ctl-C before this timeout is reached. Understood. As long as it is not infinite we've made a step in the right direction. I like your ideas below on how to improve this further. >> halr> If that is the intended behavior, we are there. (This (lost REQ) >> halr> can even occur when the timeout is non infinite too). >> >> We didn't intend for the active side to wait forever if a REQ was >> lost. > > The active side has no way of knowing that the REQ was lost (other than > timeout/retry) and when the timeout is long, this is effectively the > case. This behavior is ok. The DAT consumer should choose timeout value that makes sense, it doesn't need to use DAT_TIMEOUT_MAX (and probably shouldn't in most cases). We should update our dapltest program to use a smaller value (like 1 min). >> halr> An alternative (as Sean suggested) is to continually retry (at a >> halr> periodicity below the supplied timeout) until the time period specified >> halr> expires. That seems to be better (at least to me and Sean) in terms of >> halr> handling the lost REQ case. As retries is not part of the API for >> halr> connect, I would presume the implementor is free to what they want under >> halr> the covers of dapl_ib_connect. >> >> You're correct. > > The current implementation is: > 1. address resolution phase for some amount of time > followed by: > 2. dapl_ib_connect timeout * 5 (since there are 4 retries) Sounds like I need to understand the difference between the ib_cm_req_param's retry_count and max_cm_retries fields. We set the former to 0 and the later to 4. > A better algorithm would be to divide down the timeout by some number of > retries (which would vary based on the timeout requested) and have the > number of retries vary based on the total timeout requested. I agree that would be better. As you point out, we should also account for the address resolution time. I know that no one is working on this. Are you interested? > > -- Hal > From libor at topspin.com Tue Jun 14 10:00:43 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 14 Jun 2005 10:00:43 -0700 Subject: [openib-general] Re: Some more ucm questions In-Reply-To: <1118762880.4506.620.camel@localhost.localdomain>; from halr@voltaire.com on Tue, Jun 14, 2005 at 11:28:01AM -0400 References: <1118762880.4506.620.camel@localhost.localdomain> Message-ID: <20050614100043.A5582@topspin.com> On Tue, Jun 14, 2005 at 11:28:01AM -0400, Hal Rosenstock wrote: > Hi Libor, > > In the ucm kernel module, in the routine ib_ucm_event, after the user: > label, if various errors occur relating to copying back to userspace, > all that is currently done is that the mutex is released and the result > returned. Doesn't there need to be more cleanup here depending on which > error occured ? (Won't the event be left at the beginning of the list > and the ucm would never get past this in terms of any other events ?) The thinking was that the problem in all those ib_ucm_event error cases is not with the event itself, but how the user is using the interface. If the user does not correct the problem, the next event will have the exact same error, so deleting the current event would not solve the issue. This also gives the user a chance to recover without loosing anything. Since there is a wrapper library for the user interface, and the user is not expected to use the file descriptor directly, I would not expect these errors in normal operation, but only during debug when changes are made to the interface library itself. > Also, should there be a declaration ib_cm_event_get_timed be in the user > space include file ? You are correct, I'll commit the following. -Libor Index: userspace/libibcm/include/infiniband/cm.h =================================================================== --- userspace/libibcm/include/infiniband/cm.h (revision 2576) +++ userspace/libibcm/include/infiniband/cm.h (working copy) @@ -244,6 +244,17 @@ * to a user's existing communication identifier. */ int ib_cm_event_get(struct ib_cm_event **event); +/* + * ib_cm_event_get_timed - Retrieves the next pending communications event, + * if no event is pending wait up to a certain timeout for an event. + * @timeout_ms: Maximum time in milliseconds to wait for an event. + * @event: Allocated information about the next communication event. + * Event should be freed using ib_cm_event_put() + * + * If timeout expires without an event, the error -ETIMEDOUT will be + * returned + */ +int ib_cm_event_get_timed(int timeout_ms, struct ib_cm_event **event); /** * ib_cm_event_put - Free a communications event. From hch at lst.de Tue Jun 14 10:08:46 2005 From: hch at lst.de (Christoph Hellwig) Date: Tue, 14 Jun 2005 19:08:46 +0200 Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flags In-Reply-To: References: <20050611093144.GA18469@lst.de> Message-ID: <20050614170846.GA15215@lst.de> On Tue, Jun 14, 2005 at 10:58:47AM -0400, James Lentini wrote: > > I think this is a good change. If we trim the dat_provider_info > structure down to just an IA name, then we should be able to remove it > all together. We don't have to do that on this pass though. > > I have a question about the change you've proposed for the makefile: > > On Sat, 11 Jun 2005, Christoph Hellwig wrote: > > >Index: dat/Makefile > >=================================================================== > >--- dat/Makefile (revision 2588) > >+++ dat/Makefile (working copy) > >@@ -1,7 +1,6 @@ > > > >EXTRA_CFLAGS += \ > >- -Idrivers/dat \ > >- -Werror > >+ -I$(obj) > > Is $(obj) the directory containing the object files (drivers/dat)? > > Is the -Werror flag already being setup by kbuild? Sorry, this junk shouldn't have been in the patch. Just discard the Makefile changes completely. From hch at lst.de Tue Jun 14 10:10:49 2005 From: hch at lst.de (Christoph Hellwig) Date: Tue, 14 Jun 2005 19:10:49 +0200 Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flags In-Reply-To: <469958e0050614083016bf7425@mail.gmail.com> References: <20050611093144.GA18469@lst.de> <469958e0050614083016bf7425@mail.gmail.com> Message-ID: <20050614171049.GB15215@lst.de> On Tue, Jun 14, 2005 at 08:30:59AM -0700, Caitlin Bestler wrote: > If you eliminate the version info how would someone test DAT 1.3 code? > Is support for 1.3 just supposed to magically appear in the mainline at > some future date without intermediate testing? There won't be any DAPL 1.3 in the kernel at all, neither DAPL 1.2 nor 1.1. The kernel will have a steadily evolving rdma layer, and patches will usually be tested before they go into mainline. From hch at lst.de Tue Jun 14 10:11:22 2005 From: hch at lst.de (Christoph Hellwig) Date: Tue, 14 Jun 2005 19:11:22 +0200 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> Message-ID: <20050614171122.GC15215@lst.de> On Tue, Jun 14, 2005 at 11:32:39AM -0400, James Lentini wrote: > >Umm, no. Please try to get rid of dapl_common instead of adding to it. > > Can you elaborate on this? Is this a style or performance issue? Purely a style issue. From jlentini at netapp.com Tue Jun 14 10:12:51 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 13:12:51 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flag In-Reply-To: <469958e0050614083016bf7425@mail.gmail.com> References: <20050611093144.GA18469@lst.de> <469958e0050614083016bf7425@mail.gmail.com> Message-ID: Can we use the patch/kdat.h header to work around this? That is were we have been putting symbols for compatibility between the collaborative and Linux kDAPL headers. On Tue, 14 Jun 2005, Caitlin Bestler wrote: > If you eliminate the version info how would someone test DAT 1.3 code? > Is support for 1.3 just supposed to magically appear in the mainline at > some future date without intermediate testing? > > On 6/14/05, James Lentini wrote: >> >> I think this is a good change. If we trim the dat_provider_info >> structure down to just an IA name, then we should be able to remove it >> all together. We don't have to do that on this pass though. >> >> I have a question about the change you've proposed for the makefile: >> >> On Sat, 11 Jun 2005, Christoph Hellwig wrote: >> >>> Index: dat/Makefile >>> =================================================================== >>> --- dat/Makefile (revision 2588) >>> +++ dat/Makefile (working copy) >>> @@ -1,7 +1,6 @@ >>> >>> EXTRA_CFLAGS += \ >>> - -Idrivers/dat \ >>> - -Werror >>> + -I$(obj) >> >> Is $(obj) the directory containing the object files (drivers/dat)? >> >> Is the -Werror flag already being setup by kbuild? >> >>> >>> obj-$(CONFIG_DAT) += dat.o >>> >> _______________________________________________ >> openib-general mailing list >> openib-general at openib.org >> http://openib.org/mailman/listinfo/openib-general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> > From halr at voltaire.com Tue Jun 14 10:27:47 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 13:27:47 -0400 Subject: [openib-general] kdapltest server on x86_64 Message-ID: <1118770067.4748.8.camel@localhost.localdomain> Hi, With the latest changes (can't tell when this broke), I get the following error on x86_64 when I bring up the server: DT_cs_Server: dat_psp_create error: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_IA Server_Cmd.debug: 1 Server_Cmd.dapl_name: mthca0a DT_cs_Server: IA mthca0a opened DT_cs_Server: PZ created DT_cs_Server: EP created DT_cs_Server: dat_psp_create error: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_IA DT_cs_Server: Waiting for clients to all go away... DT_cs_Server: Cleaning up ... DT_cs_Server: dat_ep_disconnect fails: DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED DT_cs_Server: IA mthca0a closed DT_cs_Server (mthca0a): Exiting. TEST INSTANCE 0 TEST return code = 1 This works on x86. The client can be run on either x86 or x86_64. With debug on the dat and ib_dat_provider modules, the following is seen: DAT: dat_ia_openv(name:mthca0a, async_qlen:8, ver:1.2, thrd_safe:0) kDAPL: dapl_ia_open (mthca0a, 8, ffff810018aa7dc0, ffff810018aa7d90) kDAPL: dapl_ia_open () returns 0x0 kDAPL: dapl_pz_create(ffff81001a714480, ffff810018aa7d98) kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 20, ffff810018aa7da0) kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, A0, ffff810018aa7da8) kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 10, ffff810018aa7db8) kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 40, ffff810018aa7db0) kDAPL: dapl_ep_create (ffff81001a714480, ffff81001f098780, ffff81001776b0c0, ffff81001a714980, ffff81001a714d80, 0000000000000000, ffff810018aa7dd0) kDAPL: created qp_hndl = ffff81001ba61c00, qpn = 10406 DAT: dat_strerror(major:ffff81003aa0de38, minor:ffff81003aa0de40) called kDAPL: dapl_ep_disconnect(ffff81001796cc00, 0) kDAPL: dapl_ep_disconnect () returns 0x8007002e DAT: dat_strerror(major:ffff81003aa0de38, minor:ffff81003aa0de40) called kDAPL: dapl_ep_free (ffff81001796cc00) kDAPL: dapl_ep_disconnect(ffff81001796cc00, 0) kDAPL: dapl_ep_disconnect () returns 0x8007002e kDAPL: dapl_ep_free: state 0, ep ffff81001796cc00 qp_state 1 qp 1ba61c00 kDAPL: dapl_evd_free (ffff81001a714d80) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001a714b80) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001a714980) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001776b0c0) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_pz_free(ffff81001f098780) DAT: dat_ia_close(ia:1a714480, flags:1) kDAPL: dapl_ia_query (ffff81001a714480, 0000000000000000, 0000000000000000, ffff81003aa0dc28) kDAPL: dapl_ia_query () returns 0x0 kDAPL: dapl_ia_close (ffff81001a714480, 1) kDAPL: dapl_evd_free (ffff81001a714880) kDAPL: dapl_evd_free () returns 0x0 -- Hal From halr at voltaire.com Tue Jun 14 10:36:53 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 13:36:53 -0400 Subject: [openib-general] kdapl locking problem Message-ID: <1118770612.4748.16.camel@localhost.localdomain> Hi, When running in loopback mode (client and server on same machine (x86)): kdapltest -T T -s -D mthca0a -d -t 2 -w 8 -i 20 client SR server SR I see the following locking problem: Jun 14 13:30:08 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_ep.c:1111: spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c44b1c18) already locked by drivers/infiniband/ulp/dat-provider/dapl_evd.c/756 Jun 14 13:30:08 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_evd.c:797: spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c44b1c18) not locked -- Hal From tduffy at sun.com Tue Jun 14 10:53:36 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 10:53:36 -0700 Subject: [openib-general] Re: [openib-commits] r2597 - gen2/trunk/src/linux-kernel In-Reply-To: <20050614162002.986832283D9@openib.ca.sandia.gov> References: <20050614162002.986832283D9@openib.ca.sandia.gov> Message-ID: <1118771616.22484.7.camel@duffman> On Tue, 2005-06-14 at 09:20 -0700, roland at openib.org wrote: > Removed: > gen2/trunk/src/linux-kernel/TODO > Log: > Remove obsolete TODO file -- all the tasks listed there are either already > done or no longer apply. > > In the future let's keep the TODO in the wiki. I have updated the tiki site adding pages for most of the ulps including their TODO lists. Please add, change, etc. at your hearts desire. https://openib.org/tiki/tiki-index.php -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Tue Jun 14 10:56:59 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 14 Jun 2005 10:56:59 -0700 Subject: [openib-general] Re: [openib-commits] r2597 - gen2/trunk/src/linux-kernel In-Reply-To: <1118771616.22484.7.camel@duffman> (Tom Duffy's message of "Tue, 14 Jun 2005 10:53:36 -0700") References: <20050614162002.986832283D9@openib.ca.sandia.gov> <1118771616.22484.7.camel@duffman> Message-ID: <52zmtskej8.fsf@topspin.com> Tom> I have updated the tiki site adding pages for most of the Tom> ulps including their TODO lists. Cool, thanks a lot! - R. From tduffy at sun.com Tue Jun 14 10:56:43 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 10:56:43 -0700 Subject: [openib-general] [PATCH] kDAPL: update TODO Message-ID: <1118771803.22484.9.camel@duffman> Signed-off-by: Tom Duffy Index: linux-kernel/TODO =================================================================== --- linux-kernel/TODO (revision 2597) +++ linux-kernel/TODO (working copy) @@ -1,31 +1,3 @@ -*) Test static compilation of DAT and IB DAT PROVIDER +Please refer to the TODO at: -DAT -*) change dat function to use errno return code as all linux kernel export - functions -*) remove the dat_func() wrappers and have code call the functions from the - method table directly -*) move consumer to using a register/callback interface for discovering - providers (this will better support hot plug). - -IB DAT PROVIDER - -*) add ATS support (set,get) to support interoperability with old versions - of kdapl (IBDG ,Netapp filer) -*) integrate gen2 code: bring openib funcs into the common code by not - using dapl_openib_* functions -*) replace all typedef's -*) integrate linux abstraction into code (don't call dapl_os* functions ) -*) add missing functionality e.g (dat_evd_modify_upcall ...) -*) fix know bugs (connect race , disconnect race , disconnect graceful ...) -*) no need to check in dapl things that mthca also check and return error - code we can save the code and return the error from mthca. -*) register completion callback handlers when creating all cq's w/ ib_create_cq -*) fix flow control of dapl_cr_callback (DAT_CONNECTION_EVENT_BROKEN calls - dapl_ib_disconnect_clean and then tries to send a REJ on a non-existent - connection) -*) cache remote peer's address so that it does not need to be retrieved in - dapl_ep_query. -*) Replace dapl linked lists with Linux linked lists -*) don't check for NULL all over the place -- if somebody calls a function - wrong and breaks things, let them keep both halves +https://openib.org/tiki/tiki-index.php?page=kDAPLTodoList From tduffy at sun.com Tue Jun 14 10:59:38 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 10:59:38 -0700 Subject: [openib-general] [PATCH] SDP: update TODO Message-ID: <1118771978.22484.10.camel@duffman> Signed-off-by: Tom Duffy Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/TODO =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/TODO (revision 2597) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/TODO (working copy) @@ -1,58 +1,3 @@ -SDP +Please refer to the TODO at: - - Cosmetic cleanup as needed. - - - Reduce connection state machine complexity. Specifically there exists - redundant tracking of connection state which should be merged for - improved readability and maintainability. - - - Add full connection cleanup once Connection Manager IDLE is in place. - - - Reduce code bloat. - - - Enable FMRs once mthca support is provided. - - - Provide real support for SO_{RCV,SND}BUF - - internal posted buffers - - QP WQ size - - CQ size - - - Create slow start algorithm for posted receive buffers to improve - memory consumption in large connection count applications, especially - for connections that are predominately sending data. - - - sdp_link needs locking protection. - - - sdp_link request cancellation support. - - - SDP RDMA support for blocking socket operations. - - - IPv6 addressing - - - Alternate Path Migration (APM) support. - - - Keepalive (zero byte RDMA write) - - - Socket duplication (i.e. suspend) - -Possible Issues - - - SDP address resolution requires looking at the IPoIB device's private - data structure inorder to determine interface's SGID, Port, and PKey. - - - Memory locking for AIO requires a call to do_mlock() which is not a - kernel exported function, the method for calling the function is not - standard. - -Other - - - Kernel level transparent socket switch for dynamic TCP/SDP selection. - (Possible SDP port mapper protocol from RDMAC) - - - Extended Sockets API (ES-API). Initial Linux AIO based implemetation? - - - New SOCK_DGRAM protocol? - - - NFS over SDP. - - - QP striping? \ No newline at end of file +https://openib.org/tiki/tiki-index.php?page=SDPTodoList From roland at topspin.com Tue Jun 14 11:26:32 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 14 Jun 2005 11:26:32 -0700 Subject: [openib-general] UC support committed to svn Message-ID: <52mzpskd5z.fsf@topspin.com> I just committed UC support to userspace/libmthca and the kernel mthca driver (see mthca diff below). If I hack pingpong.c, this works for me but could use some testing in a real app. Also, I would appreciate it if someone could check the attribute for modify QP that I added to the table in mthca. - R. Index: infiniband/hw/mthca/mthca_qp.c =================================================================== --- infiniband/hw/mthca/mthca_qp.c (revision 2588) +++ infiniband/hw/mthca/mthca_qp.c (working copy) @@ -359,6 +359,9 @@ static const struct { [UD] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_QKEY), + [UC] = (IB_QP_PKEY_INDEX | + IB_QP_PORT | + IB_QP_ACCESS_FLAGS), [RC] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_ACCESS_FLAGS), @@ -380,6 +383,9 @@ static const struct { [UD] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_QKEY), + [UC] = (IB_QP_PKEY_INDEX | + IB_QP_PORT | + IB_QP_ACCESS_FLAGS), [RC] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_ACCESS_FLAGS), @@ -390,6 +396,11 @@ static const struct { [IB_QPS_RTR] = { .trans = MTHCA_TRANS_INIT2RTR, .req_param = { + [UC] = (IB_QP_AV | + IB_QP_PATH_MTU | + IB_QP_DEST_QPN | + IB_QP_RQ_PSN | + IB_QP_MAX_DEST_RD_ATOMIC), [RC] = (IB_QP_AV | IB_QP_PATH_MTU | IB_QP_DEST_QPN | @@ -400,6 +411,9 @@ static const struct { .opt_param = { [UD] = (IB_QP_PKEY_INDEX | IB_QP_QKEY), + [UC] = (IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PKEY_INDEX), [RC] = (IB_QP_ALT_PATH | IB_QP_ACCESS_FLAGS | IB_QP_PKEY_INDEX), @@ -415,6 +429,8 @@ static const struct { .trans = MTHCA_TRANS_RTR2RTS, .req_param = { [UD] = IB_QP_SQ_PSN, + [UC] = (IB_QP_SQ_PSN | + IB_QP_MAX_QP_RD_ATOMIC), [RC] = (IB_QP_TIMEOUT | IB_QP_RETRY_CNT | IB_QP_RNR_RETRY | @@ -425,6 +441,11 @@ static const struct { .opt_param = { [UD] = (IB_QP_CUR_STATE | IB_QP_QKEY), + [UC] = (IB_QP_CUR_STATE | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PKEY_INDEX | + IB_QP_PATH_MIG_STATE), [RC] = (IB_QP_CUR_STATE | IB_QP_ALT_PATH | IB_QP_ACCESS_FLAGS | @@ -444,6 +465,9 @@ static const struct { .opt_param = { [UD] = (IB_QP_CUR_STATE | IB_QP_QKEY), + [UC] = (IB_QP_ACCESS_FLAGS | + IB_QP_ALT_PATH | + IB_QP_PATH_MIG_STATE), [RC] = (IB_QP_ACCESS_FLAGS | IB_QP_ALT_PATH | IB_QP_PATH_MIG_STATE | @@ -464,6 +488,10 @@ static const struct { .opt_param = { [UD] = (IB_QP_CUR_STATE | IB_QP_QKEY), + [UC] = (IB_QP_CUR_STATE | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PATH_MIG_STATE), [RC] = (IB_QP_CUR_STATE | IB_QP_ALT_PATH | IB_QP_ACCESS_FLAGS | @@ -478,6 +506,15 @@ static const struct { .opt_param = { [UD] = (IB_QP_PKEY_INDEX | IB_QP_QKEY), + [UC] = (IB_QP_AV | + IB_QP_RNR_RETRY | + IB_QP_MAX_QP_RD_ATOMIC | + IB_QP_MAX_DEST_RD_ATOMIC | + IB_QP_CUR_STATE | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PKEY_INDEX | + IB_QP_PATH_MIG_STATE), [RC] = (IB_QP_AV | IB_QP_TIMEOUT | IB_QP_RETRY_CNT | @@ -503,6 +540,7 @@ static const struct { .opt_param = { [UD] = (IB_QP_CUR_STATE | IB_QP_QKEY), + [UC] = (IB_QP_CUR_STATE), [RC] = (IB_QP_CUR_STATE | IB_QP_MIN_RNR_TIMER), [MLX] = (IB_QP_CUR_STATE | @@ -1601,6 +1639,26 @@ int mthca_tavor_post_send(struct ib_qp * break; + case UC: + switch (wr->opcode) { + case IB_WR_RDMA_WRITE: + case IB_WR_RDMA_WRITE_WITH_IMM: + ((struct mthca_raddr_seg *) wqe)->raddr = + cpu_to_be64(wr->wr.rdma.remote_addr); + ((struct mthca_raddr_seg *) wqe)->rkey = + cpu_to_be32(wr->wr.rdma.rkey); + ((struct mthca_raddr_seg *) wqe)->reserved = 0; + wqe += sizeof (struct mthca_raddr_seg); + size += sizeof (struct mthca_raddr_seg) / 16; + break; + + default: + /* No extra segments required for sends */ + break; + } + + break; + case UD: ((struct mthca_tavor_ud_seg *) wqe)->lkey = cpu_to_be32(to_mah(wr->wr.ud.ah)->key); @@ -1886,9 +1944,29 @@ int mthca_arbel_post_send(struct ib_qp * sizeof (struct mthca_atomic_seg); break; + case IB_WR_RDMA_READ: + case IB_WR_RDMA_WRITE: + case IB_WR_RDMA_WRITE_WITH_IMM: + ((struct mthca_raddr_seg *) wqe)->raddr = + cpu_to_be64(wr->wr.rdma.remote_addr); + ((struct mthca_raddr_seg *) wqe)->rkey = + cpu_to_be32(wr->wr.rdma.rkey); + ((struct mthca_raddr_seg *) wqe)->reserved = 0; + wqe += sizeof (struct mthca_raddr_seg); + size += sizeof (struct mthca_raddr_seg) / 16; + break; + + default: + /* No extra segments required for sends */ + break; + } + + break; + + case UC: + switch (wr->opcode) { case IB_WR_RDMA_WRITE: case IB_WR_RDMA_WRITE_WITH_IMM: - case IB_WR_RDMA_READ: ((struct mthca_raddr_seg *) wqe)->raddr = cpu_to_be64(wr->wr.rdma.remote_addr); ((struct mthca_raddr_seg *) wqe)->rkey = From bjordan.ics at gmail.com Tue Jun 14 11:28:22 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Tue, 14 Jun 2005 14:28:22 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1118187249.4477.278.camel@localhost.localdomain> <1118245813.4477.92.camel@localhost.localdomain> <1118664517.4476.9585.camel@localhost.localdomain> <1118752898.4506.530.camel@localhost.localdomain> Message-ID: <78d18e205061411286b570b30@mail.gmail.com> On 6/14/05, James Lentini wrote: > > Sounds like I need to understand the difference between the > ib_cm_req_param's retry_count and max_cm_retries fields. We set the > former to 0 and the later to 4. The retry_count is the number of retries you want to configure for data on your connection once your connection is established. The max_cm_retries field is how many times you want the cm to retry establishing the connection. -- Bill Jordan SilverStorm Technologies From tduffy at sun.com Tue Jun 14 11:38:57 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 11:38:57 -0700 Subject: [openib-general] http://www.openib.org/doc.html update Message-ID: <1118774337.22484.16.camel@duffman> Can the folks at Sandia please update the http://www.openib.org/doc.html page to point to the wiki site? Also, we should move the FAQ's to WIKI. I can do this if you want. Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mlleinin at hpcn.ca.sandia.gov Tue Jun 14 11:53:41 2005 From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger) Date: Tue, 14 Jun 2005 11:53:41 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <1118774337.22484.16.camel@duffman> References: <1118774337.22484.16.camel@duffman> Message-ID: <1118775221.24424.21.camel@localhost> On Tue, 2005-06-14 at 11:38 -0700, Tom Duffy wrote: > Can the folks at Sandia please update the http://www.openib.org/doc.html > page to point to the wiki site? > > Also, we should move the FAQ's to WIKI. I can do this if you want. > I'll add a link to doc.html pointing to the wiki. Once the FAQ's are moved to the wiki I can just have doc.html point directly to the wiki. - Matt From jlentini at netapp.com Tue Jun 14 11:57:57 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 14:57:57 -0400 (EDT) Subject: [openib-general] mthca_tavor_post_send() question Message-ID: In mthca_tavor_post_send() at line 1524 of hw/mthca/mthca_qp.c, an int variable, "ind", is used to store the unsigned "next_ind" value. Why does "ind" not need to represent all of the possible "next_ind" values? Should the type of "ind" also be unsigned? james From roland at topspin.com Tue Jun 14 12:05:22 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 14 Jun 2005 12:05:22 -0700 Subject: [openib-general] Re: mthca_tavor_post_send() question In-Reply-To: (James Lentini's message of "Tue, 14 Jun 2005 14:57:57 -0400 (EDT)") References: Message-ID: <528y1ckbd9.fsf@topspin.com> James> In mthca_tavor_post_send() at line 1524 of James> hw/mthca/mthca_qp.c, an int variable, "ind", is used to James> store the unsigned "next_ind" value. Why does "ind" not James> need to represent all of the possible "next_ind" values? James> Should the type of "ind" also be unsigned? I don't think it makes much practical difference, since the largest value they'll store is 65535. It might be cleaner to change them to match, although I don't know which one should change. - R. From jlentini at netapp.com Tue Jun 14 12:16:23 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 15:16:23 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flags In-Reply-To: <20050611093144.GA18469@lst.de> References: <20050611093144.GA18469@lst.de> Message-ID: Christoph, I'm all ready to submit this minus the Makefile change. Can you re-send with a "Signed-off-by" line? (no need to make any changes) On Sat, 11 Jun 2005, Christoph Hellwig wrote: > in the linux kernel any provider must use the version it was compiled > against, and must always be thread safe. > > > Index: test/dapltest/test/dapl_test_util.c > =================================================================== > --- test/dapltest/test/dapl_test_util.c (revision 2588) > +++ test/dapltest/test/dapl_test_util.c (working copy) > @@ -76,12 +76,10 @@ > struct sockaddr_in *ip_addr; > > DT_Tdep_PT_Printf (phead, "***** DAPL Characteristics *****\n"); > - DT_Tdep_PT_Printf (phead, "Provider: %s Version %d.%d DAPL %d.%d\n", > + DT_Tdep_PT_Printf (phead, "Provider: %s Version %d.%d\n", > pt_ptr->provider_attr.provider_name, > pt_ptr->provider_attr.provider_version_major, > - pt_ptr->provider_attr.provider_version_minor, > - pt_ptr->provider_attr.dat_version_major, > - pt_ptr->provider_attr.dat_version_minor ); > + pt_ptr->provider_attr.provider_version_minor); > DT_Tdep_PT_Printf (phead, "Adapter: %s by %s Version %d.%d\n", > pt_ptr->ia_attr.adapter_name, > pt_ptr->ia_attr.vendor_name, > Index: dat-provider/dapl_ia.c > =================================================================== > --- dat-provider/dapl_ia.c (revision 2588) > +++ dat-provider/dapl_ia.c (working copy) > @@ -881,15 +881,12 @@ > DAT_NAME_MAX_LENGTH); > provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; > provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; > - provider_attr->dat_version_major = DAT_VERSION_MAJOR; > - provider_attr->dat_version_minor = DAT_VERSION_MINOR; > provider_attr->lmr_mem_types_supported = > DAT_MEM_TYPE_VIRTUAL | DAT_MEM_TYPE_LMR; > provider_attr->iov_ownership_on_return = DAT_IOV_CONSUMER; > provider_attr->dat_qos_supported = DAT_QOS_BEST_EFFORT; > provider_attr->completion_flags_supported = > DAT_COMPLETION_DEFAULT_FLAG; > - provider_attr->is_thread_safe = FALSE; > provider_attr->max_private_data_size = IB_CM_REQ_PRIVATE_DATA_SIZE; > provider_attr->supports_multipath = FALSE; > provider_attr->ep_creator = DAT_PSP_CREATES_EP_NEVER; > Index: dat-provider/dapl_provider.c > =================================================================== > --- dat-provider/dapl_provider.c (revision 2588) > +++ dat-provider/dapl_provider.c (working copy) > @@ -244,9 +244,6 @@ > (size_t) (DAT_NAME_MAX_LENGTH - 2)); > provider_info->ia_name[i] = 'a' + port - 1; /* IB ports are 1 based */ > provider_info->ia_name[i+1] = '\0'; > - provider_info->dat_version_major = DAT_VERSION_MAJOR; > - provider_info->dat_version_minor = DAT_VERSION_MINOR; > - provider_info->is_thread_safe = DAPL_THREADSAFE; > } > > static void dapl_add_port(struct ib_device *device, u8 port) > Index: dat/dictionary.c > =================================================================== > --- dat/dictionary.c (revision 2588) > +++ dat/dictionary.c (working copy) > @@ -212,10 +212,6 @@ > BUG_ON(NULL == new_key); > > strncpy(new_key->ia_name, old_key->ia_name, DAT_NAME_MAX_LENGTH); > - new_key->dat_version_major = old_key->dat_version_major; > - new_key->dat_version_minor = old_key->dat_version_minor; > - new_key->is_thread_safe = old_key->is_thread_safe; > - > return DAT_SUCCESS; > } > > @@ -226,10 +222,7 @@ > BUG_ON(NULL == b); > > if ((strlen(a->ia_name) == strlen(b->ia_name)) && > - (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name))) > - && (a->dat_version_major == b->dat_version_major) > - && (a->dat_version_minor == b->dat_version_minor) > - && (a->is_thread_safe == b->is_thread_safe)) > + (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name)))) > return TRUE; > else > return FALSE; > Index: dat/api.c > =================================================================== > --- dat/api.c (revision 2588) > +++ dat/api.c (working copy) > @@ -349,9 +349,8 @@ > } > } > > -u32 dat_ia_openv(const char *name, int async_event_qlen, > - struct dat_evd **async_event_handle, struct dat_ia **ia, > - u32 dat_major, u32 dat_minor, boolean_t thread_safety) > +u32 dat_ia_open(const char *name, int async_event_qlen, > + struct dat_evd **async_event_handle, struct dat_ia **ia) > { > DAT_IA_OPEN_FUNC ia_open_func; > struct dat_provider_info info; > @@ -360,8 +359,7 @@ > > dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > "%s(name:%s, async_qlen:%i, ver:%x.%x, thrd_safe:%x)\n", > - __func__, name, async_event_qlen, dat_major, dat_minor, > - thread_safety); > + __func__, name, async_event_qlen); > > if (DAT_IS_BAD_POINTER(name)) > return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); > @@ -374,10 +372,6 @@ > strncpy(info.ia_name, name, len); > info.ia_name[len] = '\0'; > > - info.dat_version_major = dat_major; > - info.dat_version_minor = dat_minor; > - info.is_thread_safe = thread_safety; > - > status = dat_dr_provider_open(&info, &ia_open_func); > if (status != DAT_SUCCESS) { > dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > @@ -390,7 +384,7 @@ > return ia_open_func(name, async_event_qlen, async_event_handle, ia); > } > > -EXPORT_SYMBOL(dat_ia_openv); > +EXPORT_SYMBOL(dat_ia_open); > > u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) > { > @@ -423,10 +417,6 @@ > strncpy(info.ia_name, ia_name, len); > info.ia_name[len] = '\0'; > > - info.dat_version_major = provider_attr.dat_version_major; > - info.dat_version_minor = provider_attr.dat_version_minor; > - info.is_thread_safe = provider_attr.is_thread_safe; > - > status = dat_dr_provider_close(&info); > if (DAT_SUCCESS != status) > dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > @@ -472,10 +462,7 @@ > { > dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, > "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", > - __func__, provider_info->ia_name, > - provider_info->dat_version_major, > - provider_info->dat_version_minor, > - provider_info->is_thread_safe); > + __func__, provider_info->ia_name); > > if (DAT_IS_BAD_POINTER(provider)) > return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); > Index: dat/Makefile > =================================================================== > --- dat/Makefile (revision 2588) > +++ dat/Makefile (working copy) > @@ -1,7 +1,6 @@ > > EXTRA_CFLAGS += \ > - -Idrivers/dat \ > - -Werror > + -I$(obj) > > obj-$(CONFIG_DAT) += dat.o > > Index: dat/dat.h > =================================================================== > --- dat/dat.h (revision 2588) > +++ dat/dat.h (working copy) > @@ -33,10 +33,6 @@ > #include > #include > > -#define DAT_VERSION_MAJOR 1 > -#define DAT_VERSION_MINOR 2 > -#define DAT_THREADSAFE FALSE > - > /* > * All return codes are actually a 3-way tuple: > * > @@ -894,9 +890,6 @@ > > struct dat_provider_info { > char ia_name[DAT_NAME_MAX_LENGTH]; > - u32 dat_version_major; > - u32 dat_version_minor; > - boolean_t is_thread_safe; > }; > > struct dat_evd_param { > @@ -1017,13 +1010,10 @@ > char provider_name[DAT_NAME_MAX_LENGTH]; > u32 provider_version_major; > u32 provider_version_minor; > - u32 dat_version_major; > - u32 dat_version_minor; > enum dat_mem_type lmr_mem_types_supported; > enum dat_iov_ownership iov_ownership_on_return; > enum dat_qos dat_qos_supported; > enum dat_completion_flags completion_flags_supported; > - boolean_t is_thread_safe; > int max_private_data_size; > boolean_t supports_multipath; > enum dat_ep_creator_for_psp ep_creator; > @@ -1361,16 +1351,10 @@ > /* > * DAT registry functions for consumers > */ > -extern u32 dat_ia_openv(const char *name, int async_event_qlen, > +extern u32 dat_ia_open(const char *name, int async_event_qlen, > struct dat_evd **async_event_handle, > - struct dat_ia **ia, u32 dat_major, u32 dat_minor, > - boolean_t thread_safety); > + struct dat_ia **ia); > > -#define dat_ia_open(name, qlen, async_evd, ia) \ > - dat_ia_openv((name), (qlen), (async_evd), (ia), \ > - DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ > - DAT_THREADSAFE) > - > extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); > > extern u32 dat_strerror(u32, const char **, const char **); > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From jlentini at netapp.com Tue Jun 14 12:29:50 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 15:29:50 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: update TODO In-Reply-To: <1118771803.22484.9.camel@duffman> References: <1118771803.22484.9.camel@duffman> Message-ID: Thanks Tom. Committed in revision 2600. On Tue, 14 Jun 2005, Tom Duffy wrote: tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/TODO tduffy> =================================================================== tduffy> --- linux-kernel/TODO (revision 2597) tduffy> +++ linux-kernel/TODO (working copy) tduffy> @@ -1,31 +1,3 @@ tduffy> -*) Test static compilation of DAT and IB DAT PROVIDER tduffy> +Please refer to the TODO at: tduffy> tduffy> -DAT tduffy> -*) change dat function to use errno return code as all linux kernel export tduffy> - functions tduffy> -*) remove the dat_func() wrappers and have code call the functions from the tduffy> - method table directly tduffy> -*) move consumer to using a register/callback interface for discovering tduffy> - providers (this will better support hot plug). tduffy> - tduffy> -IB DAT PROVIDER tduffy> - tduffy> -*) add ATS support (set,get) to support interoperability with old versions tduffy> - of kdapl (IBDG ,Netapp filer) tduffy> -*) integrate gen2 code: bring openib funcs into the common code by not tduffy> - using dapl_openib_* functions tduffy> -*) replace all typedef's tduffy> -*) integrate linux abstraction into code (don't call dapl_os* functions ) tduffy> -*) add missing functionality e.g (dat_evd_modify_upcall ...) tduffy> -*) fix know bugs (connect race , disconnect race , disconnect graceful ...) tduffy> -*) no need to check in dapl things that mthca also check and return error tduffy> - code we can save the code and return the error from mthca. tduffy> -*) register completion callback handlers when creating all cq's w/ ib_create_cq tduffy> -*) fix flow control of dapl_cr_callback (DAT_CONNECTION_EVENT_BROKEN calls tduffy> - dapl_ib_disconnect_clean and then tries to send a REJ on a non-existent tduffy> - connection) tduffy> -*) cache remote peer's address so that it does not need to be retrieved in tduffy> - dapl_ep_query. tduffy> -*) Replace dapl linked lists with Linux linked lists tduffy> -*) don't check for NULL all over the place -- if somebody calls a function tduffy> - wrong and breaks things, let them keep both halves tduffy> +https://openib.org/tiki/tiki-index.php?page=kDAPLTodoList tduffy> From jjengla at sandia.gov Tue Jun 14 12:37:28 2005 From: jjengla at sandia.gov (Josh England) Date: Tue, 14 Jun 2005 12:37:28 -0700 Subject: [openib-general] dapl build problems In-Reply-To: References: <1118706314.17676.28.camel@localhost> <1118708744.24800.1.camel@duffman> Message-ID: <1118777848.17674.58.camel@localhost> On Tue, 2005-06-14 at 11:45 -0400, James Lentini wrote: > > On Mon, 13 Jun 2005, Tom Duffy wrote: > > > On Mon, 2005-06-13 at 16:45 -0700, Josh England wrote: > >> I get these problems trying to build dat with 2.6.11.11 (svn rev 2595): > >> > >> *** Warning: > >> "dat_registry_add_provider" [drivers/infiniband/ulp/dat-provider/ib_dat_provider.ko] undefined! > >> *** Warning: > >> "dat_registry_remove_provider" [drivers/infiniband/ulp/dat-provider/ib_dat_provider.ko] undefined! > > > > That is fine. These are provided by dat.ko. Make sure to load that > > module before ib_dat_provider.ko. > > Josh, > > Were you attempting to load, not build, the module? If that is the > case, you can load dat.ko with a "modprobe dat.ko". No, I was just looking for everything under -> Device Drivers -> InfiniBand support and missed -> Device Drivers -> Direct Access Transport (DAT) support Its builds now. -JE > james > From jjengla at sandia.gov Tue Jun 14 12:40:50 2005 From: jjengla at sandia.gov (Josh England) Date: Tue, 14 Jun 2005 12:40:50 -0700 Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118761342.12164.0.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> <1118761342.12164.0.camel@duffman> Message-ID: <1118778050.17668.62.camel@localhost> On Tue, 2005-06-14 at 08:02 -0700, Tom Duffy wrote: > On Mon, 2005-06-13 at 17:18 -0700, Josh England wrote: > > I can't get DAT (provider) to build with the latest rev. I'll try again > > later. > > Besides the symbol warnings, what errors are you seeing? New problem: svn build 2599. dat and ib_dat_provider have debug on. kdapltest -T S -D mthca0a gives me this: DT_cs_Server: dat_psp_create error: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_IA DT_cs_Server: dat_ep_disconnect fails: DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED DT_cs_Server (mthca0a): Exiting. TEST INSTANCE 0 TEST return code = 1 dmesg shows: DAT: dat_ia_openv(name:mthca0a, async_qlen:8, ver:1.2, thrd_safe:0) kDAPL: dapl_ia_open (mthca0a, 8, ffff81007fc3be40, ffff81007fc3be10) kDAPL: dapl_ia_open () returns 0x0 kDAPL: dapl_pz_create(ffff81007de39480, ffff81007fc3be18) kDAPL: dapl_evd_kcreate(ffff81007de39480, 8, 1, ffff81007d087e08, 20, ffff81007fc3be20) kDAPL: dapl_evd_kcreate(ffff81007de39480, 8, 1, ffff81007d087e08, A0, ffff81007fc3be28) kDAPL: dapl_evd_kcreate(ffff81007de39480, 8, 1, ffff81007d087e08, 10, ffff81007fc3be38) kDAPL: dapl_evd_kcreate(ffff81007de39480, 8, 1, ffff81007d087e08, 40, ffff81007fc3be30) kDAPL: dapl_ep_create (ffff81007de39480, ffff81007e520680, ffff81007e547a80, ffff8100032ab5c0, ffff81007de39180, 0000000000000000, ffff81007fc3be50) kDAPL: created qp_hndl = ffff81007d8fbe00, qpn = 20406 DAT: dat_strerror(major:ffff81007d087e38, minor:ffff81007d087e40) called kDAPL: dapl_ep_disconnect(ffff81007f296c00, 0) kDAPL: dapl_ep_disconnect () returns 0x8007002e DAT: dat_strerror(major:ffff81007d087e38, minor:ffff81007d087e40) called kDAPL: dapl_ep_free (ffff81007f296c00) kDAPL: dapl_ep_disconnect(ffff81007f296c00, 0) kDAPL: dapl_ep_disconnect () returns 0x8007002e kDAPL: dapl_ep_free: state 0, ep ffff81007f296c00 qp_state 1 qp 7d8fbe00 kDAPL: dapl_evd_free (ffff81007de39180) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81007de39380) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff8100032ab5c0) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81007e547a80) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_pz_free(ffff81007e520680) DAT: dat_ia_close(ia:7de39480, flags:1) kDAPL: dapl_ia_query (ffff81007de39480, 0000000000000000, 0000000000000000, ffff81007d087c28) kDAPL: dapl_ia_query () returns 0x0 kDAPL: dapl_ia_close (ffff81007de39480, 1) kDAPL: dapl_evd_free (ffff81007d6395c0) kDAPL: dapl_evd_free () returns 0x0 Its probably just me being dumb again...what should I check? -JE From jlentini at netapp.com Tue Jun 14 12:47:21 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 15:47:21 -0400 (EDT) Subject: [openib-general] Re: mthca_tavor_post_send() question In-Reply-To: <528y1ckbd9.fsf@topspin.com> References: <528y1ckbd9.fsf@topspin.com> Message-ID: On Tue, 14 Jun 2005, Roland Dreier wrote: roland> James> In mthca_tavor_post_send() at line 1524 of roland> James> hw/mthca/mthca_qp.c, an int variable, "ind", is used to roland> James> store the unsigned "next_ind" value. Why does "ind" not roland> James> need to represent all of the possible "next_ind" values? roland> James> Should the type of "ind" also be unsigned? roland> roland> I don't think it makes much practical difference, since the largest roland> value they'll store is 65535. It might be cleaner to change them to roland> match, although I don't know which one should change. I understand. The mthca_wq structure is used for both Tavor and Arbel, so "next_ind" needs to be able to store the maximum number of work requests supported by either chip. I would suggest changing the "ind" variables type to unsigned so someone reading the code for the first time doesn't get sidetracked by it. As you say, this isn't of any practical importance. james From jlentini at netapp.com Tue Jun 14 13:21:40 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 16:21:40 -0400 (EDT) Subject: [openib-general] [PATCH 8/9] kDAPL: convert the hcas ia list to linux native In-Reply-To: <11187082903037@sun.com> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> <1118708288856@sun.com> <11187082893179@sun.com> <1118708290446@sun.com> <11187082903037@sun.com> Message-ID: On Mon, 13 Jun 2005, Tom Duffy wrote: tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll8/dat-provider/dapl_ia.c linux-kernel-ll9/dat-provider/dapl_ia.c tduffy> --- linux-kernel-ll8/dat-provider/dapl_ia.c 2005-06-13 16:50:30.624000000 -0700 tduffy> +++ linux-kernel-ll9/dat-provider/dapl_ia.c 2005-06-13 16:56:10.004000000 -0700 tduffy> @@ -62,7 +62,6 @@ struct dapl_ia *dapl_ia_alloc(struct dat tduffy> ia->hca = hca; tduffy> ia->async_error_evd = NULL; tduffy> ia->cleanup_async_error_evd = FALSE; tduffy> - dapl_llist_init_entry(&ia->hca_ia_list_entry); Shouldn't we replace this line with an INIT_LIST_HEAD(&ia->list)? Or is this going to be done in the next patch? tduffy> INIT_LIST_HEAD(&ia->ep_list); tduffy> INIT_LIST_HEAD(&ia->lmr_list); tduffy> INIT_LIST_HEAD(&ia->rmr_list); From tduffy at sun.com Tue Jun 14 13:26:46 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 13:26:46 -0700 Subject: [openib-general] [PATCH 8/9] kDAPL: convert the hcas ia list to linux native In-Reply-To: References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> <1118708288856@sun.com> <11187082893179@sun.com> <1118708290446@sun.com> <11187082903037@sun.com> Message-ID: <1118780806.4985.7.camel@duffman> On Tue, 2005-06-14 at 16:21 -0400, James Lentini wrote: > > On Mon, 13 Jun 2005, Tom Duffy wrote: > > tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll8/dat-provider/dapl_ia.c linux-kernel-ll9/dat-provider/dapl_ia.c > tduffy> --- linux-kernel-ll8/dat-provider/dapl_ia.c 2005-06-13 16:50:30.624000000 -0700 > tduffy> +++ linux-kernel-ll9/dat-provider/dapl_ia.c 2005-06-13 16:56:10.004000000 -0700 > tduffy> @@ -62,7 +62,6 @@ struct dapl_ia *dapl_ia_alloc(struct dat > tduffy> ia->hca = hca; > tduffy> ia->async_error_evd = NULL; > tduffy> ia->cleanup_async_error_evd = FALSE; > tduffy> - dapl_llist_init_entry(&ia->hca_ia_list_entry); > > Shouldn't we replace this line with an INIT_LIST_HEAD(&ia->list)? Or > is this going to be done in the next patch? This isn't the head. It is the entry. Therefore, it doesn't need to be initialized. -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Tue Jun 14 13:41:36 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 16:41:36 -0400 (EDT) Subject: [openib-general] [PATCH 8/9] kDAPL: convert the hcas ia list to linux native In-Reply-To: <1118780806.4985.7.camel@duffman> References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> <1118708288856@sun.com> <11187082893179@sun.com> <1118708290446@sun.com> <11187082903037@sun.com> <1118780806.4985.7.camel@duffman> Message-ID: On Tue, 14 Jun 2005, Tom Duffy wrote: tduffy> On Tue, 2005-06-14 at 16:21 -0400, James Lentini wrote: tduffy> > tduffy> > On Mon, 13 Jun 2005, Tom Duffy wrote: tduffy> > tduffy> > tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-ll8/dat-provider/dapl_ia.c linux-kernel-ll9/dat-provider/dapl_ia.c tduffy> > tduffy> --- linux-kernel-ll8/dat-provider/dapl_ia.c 2005-06-13 16:50:30.624000000 -0700 tduffy> > tduffy> +++ linux-kernel-ll9/dat-provider/dapl_ia.c 2005-06-13 16:56:10.004000000 -0700 tduffy> > tduffy> @@ -62,7 +62,6 @@ struct dapl_ia *dapl_ia_alloc(struct dat tduffy> > tduffy> ia->hca = hca; tduffy> > tduffy> ia->async_error_evd = NULL; tduffy> > tduffy> ia->cleanup_async_error_evd = FALSE; tduffy> > tduffy> - dapl_llist_init_entry(&ia->hca_ia_list_entry); tduffy> > tduffy> > Shouldn't we replace this line with an INIT_LIST_HEAD(&ia->list)? Or tduffy> > is this going to be done in the next patch? tduffy> tduffy> This isn't the head. It is the entry. Therefore, it doesn't need to be tduffy> initialized. Got it. Then I have an unnecessary INIT_LIST_HEAD on line 236 of dat/dictionary.c. From jlentini at netapp.com Tue Jun 14 13:52:19 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 16:52:19 -0400 (EDT) Subject: [openib-general] [PATCH 0/9] kDAPL: Use Linux standard linked lists In-Reply-To: <11187082863011@sun.com> References: <11187082863011@sun.com> Message-ID: Committed in revision 2601. On Mon, 13 Jun 2005, Tom Duffy wrote: > This series of patches removes the use of dapl's own linked list implementation > in favor of using standard Linux linked lists. > > Please apply in order. > > Signed-off-by: Tom Duffy > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From halr at voltaire.com Tue Jun 14 13:54:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 16:54:51 -0400 Subject: [openib-general] UC support committed to svn In-Reply-To: <52mzpskd5z.fsf@topspin.com> References: <52mzpskd5z.fsf@topspin.com> Message-ID: <1118782490.4748.21.camel@localhost.localdomain> On Tue, 2005-06-14 at 14:26, Roland Dreier wrote: > I just committed UC support to userspace/libmthca and the kernel mthca > driver (see mthca diff below). If I hack pingpong.c, this works for > me but could use some testing in a real app. Also, I would appreciate > it if someone could check the attribute for modify QP that I added to > the table in mthca. Just one minor question: On SQD->SQD for UC, should IB_QP_RNR_RETRY be included as optional ? -- Hal From jlentini at netapp.com Tue Jun 14 14:20:39 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 17:20:39 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <20050614171122.GC15215@lst.de> References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> <20050614171122.GC15215@lst.de> Message-ID: On Tue, 14 Jun 2005, Christoph Hellwig wrote: > On Tue, Jun 14, 2005 at 11:32:39AM -0400, James Lentini wrote: >>> Umm, no. Please try to get rid of dapl_common instead of adding to it. >> >> Can you elaborate on this? Is this a style or performance issue? > > Purely a style issue. Is it a personal style issue or a Linux kernel style issue? I really do see merits to the principal of using common structures, even if our implementation of it is a little off. From libor at topspin.com Tue Jun 14 14:24:39 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 14 Jun 2005 14:24:39 -0700 Subject: [openib-general] Re: [PATCH] Re: 0 op factor In-Reply-To: <20050513140655.GD320@mellanox.co.il>; from mst@mellanox.co.il on Fri, May 13, 2005 at 05:06:55PM +0300 References: <20050505155803.GS15688@aon.at> <20050505172122.B26428@topspin.com> <20050507174718.GQ15688@aon.at> <20050510174700.A5174@topspin.com> <20050512154857.D23734@topspin.com> <20050513140655.GD320@mellanox.co.il> Message-ID: <20050614142439.B5582@topspin.com> On Fri, May 13, 2005 at 05:06:55PM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > Subject: Re: [PATCH] Re: 0 op factor > > > > On Tue, May 10, 2005 at 05:47:00PM -0700, Libor Michalek wrote: > > > On Sat, May 07, 2005 at 07:47:18PM +0200, Bernhard Fischer wrote: > > > > > > > - remove expect from _sdp_cm_path_complete(). > > > > > > When I said that the SDP_EXPECT should eventually be removed, I meant > > > that the functions which have their return values checked by SDP_EXPECT > > > should either be turned into void return functions, or something > > > intelligent should be done with the return value, such as error recovery > > > or propogation. The functions which should be turned into void functions > > > are the ones which will never return anything but success. > > > > To expand on the last point. There are a lot of functions, for > > example those in sdp_buff.c, which check for incorrect function > > usage, such as checking that a buffer is not already in a queue before > > inserting it into a queue. These checks could be removed entirely or > > the checks could be made and a call to BUG() in case the condition > > is met. I prefer the later since the former would corrupt the entire > > queue. I'm not sure which is preferable to everyone else, either way > > the result is that the function becomes a void. > > IMO BUG_ON is the way to go. Attached is a patch which simplifiess a number of functions to return void, instead of a status result, and check incorrect parameters with BUG_ON. This results in the removal in a number of SDP_EXPECT instances as well as other unused error condition code. 20 files changed, 221 insertions(+), 548 deletions(-) -Libor Index: infiniband/ulp/sdp/sdp_queue.h =================================================================== --- infiniband/ulp/sdp/sdp_queue.h (revision 2588) +++ infiniband/ulp/sdp/sdp_queue.h (working copy) @@ -54,7 +54,7 @@ struct sdpc_desc *prev; /* previous structure in table */ u32 type; /* element type. (for generic queue) */ struct sdpc_desc_q *table; /* table to which this object belongs */ - int (*release)(struct sdpc_desc *element); /* release the object */ + void (*release)(struct sdpc_desc *element); /* release the object */ }; /* Index: infiniband/ulp/sdp/sdp_write.c =================================================================== --- infiniband/ulp/sdp/sdp_write.c (revision 2588) +++ infiniband/ulp/sdp/sdp_write.c (working copy) @@ -87,13 +87,11 @@ type = sdp_desc_q_type_head(&conn->w_snk); switch (type) { case SDP_DESC_TYPE_BUFF: - buff = (struct sdpc_buff *) sdp_desc_q_get_head(&conn->w_snk); - SDP_EXPECT((buff)); + buff = (struct sdpc_buff *)sdp_desc_q_get_head(&conn->w_snk); conn->send_qud -= buff->data_size; - result = sdp_buff_pool_put(buff); - SDP_EXPECT(result >= 0); + sdp_buff_pool_put(buff); break; case SDP_DESC_TYPE_IOCB: @@ -104,19 +102,13 @@ } iocb = (struct sdpc_iocb *)sdp_desc_q_get_head(&conn->w_snk); - SDP_EXPECT((iocb)); iocb->flags &= ~(SDP_IOCB_F_ACTIVE | SDP_IOCB_F_RDMA_W); SDP_CONN_STAT_WRITE_INC(conn, iocb->post); SDP_CONN_STAT_WQ_DEC(conn, iocb->size); - result = sdp_iocb_complete(iocb, 0); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>", - result, iocb->key); - goto error; - } + sdp_iocb_complete(iocb, 0); break; case SDP_DESC_TYPE_NONE: Index: infiniband/ulp/sdp/sdp_link.c =================================================================== --- infiniband/ulp/sdp/sdp_link.c (revision 2588) +++ infiniband/ulp/sdp/sdp_link.c (working copy) @@ -310,12 +310,13 @@ sdp_dbg_warn(NULL, "Error <%d> starting path record query", result); info->query = NULL; - } else { - info->qid = result; - info->flags |= SDP_LINK_F_PATH; + return result; } - return result; + info->qid = result; + info->flags |= SDP_LINK_F_PATH; + + return 0; } /* Index: infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- infiniband/ulp/sdp/sdp_rcvd.c (revision 2592) +++ infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -125,7 +125,6 @@ static int sdp_rcvd_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; - int result; /* * 1) Conn is not in source cancel mode. Send active IOCB @@ -152,8 +151,7 @@ conn->src_sent--; - result = sdp_iocb_complete(iocb, 0); - SDP_EXPECT(result >= 0); + sdp_iocb_complete(iocb, 0); } /* * Cancel complete, clear the state. @@ -169,7 +167,6 @@ { struct msg_hdr_rwch *rwch; struct sdpc_iocb *iocb; - int result; rwch = (struct msg_hdr_rwch *) buff->data; buff->data = buff->data + sizeof(struct msg_hdr_rwch); @@ -181,8 +178,7 @@ iocb = sdp_iocb_q_look(&conn->r_snk); if (!iocb) { sdp_dbg_warn(conn, "Cannot find IOCB for Write Completion."); - result = -EPROTO; - goto error; + return -EPROTO; } SDP_EXPECT((iocb->flags & SDP_IOCB_F_RDMA_W)); @@ -196,17 +192,13 @@ if (rwch->size > iocb->len) { sdp_dbg_warn(conn, "IOCB and Write size mismatch. <%d:%d>", rwch->size, iocb->len); - result = -EPROTO; - goto error; + return -EPROTO; } /* * Iocb is done, deregister memory, and generate completion. */ iocb = sdp_iocb_q_get_head(&conn->r_snk); - SDP_EXPECT((iocb)); - conn->snk_sent--; - iocb->len -= rwch->size; iocb->post += rwch->size; @@ -214,23 +206,17 @@ SDP_CONN_STAT_READ_INC(conn, iocb->post); SDP_CONN_STAT_RQ_DEC(conn, iocb->size); - result = sdp_iocb_complete(iocb, 0); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>", - result, iocb->key); - goto error; - } + conn->snk_sent--; + sdp_iocb_complete(iocb, 0); + return 0; -error: - return result; } static int sdp_rcvd_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff) { struct msg_hdr_rrch *rrch; struct sdpc_iocb *iocb; - int result; rrch = (struct msg_hdr_rrch *) buff->data; buff->data = buff->data + sizeof(struct msg_hdr_rrch); @@ -242,8 +228,7 @@ iocb = sdp_iocb_q_look(&conn->w_src); if (!iocb) { sdp_dbg_warn(conn, "Cannot find IOCB for Read Completion."); - result = -EPROTO; - goto error; + return -EPROTO; } SDP_CONN_STAT_SRC_INC(conn); @@ -257,8 +242,7 @@ if (rrch->size > iocb->len) { sdp_dbg_warn(conn, "IOCB and Read size mismatch. <%d:%d>", rrch->size, iocb->len); - result = -EPROTO; - goto error; + return -EPROTO; } /* * In combined mode the total RDMA size is going to be the buffer @@ -279,19 +263,13 @@ */ if (iocb->len <= 0) { iocb = sdp_iocb_q_get_head(&conn->w_src); - SDP_EXPECT((iocb)); conn->src_sent--; SDP_CONN_STAT_WRITE_INC(conn, iocb->post); SDP_CONN_STAT_WQ_DEC(conn, iocb->size); - result = sdp_iocb_complete(iocb, 0); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>", - result, iocb->key); - goto error; - } + sdp_iocb_complete(iocb, 0); } /* * If Source Cancel was in process, and there are no more outstanding @@ -304,8 +282,6 @@ } return 0; -error: - return result; } static int sdp_rcvd_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff) @@ -457,8 +433,7 @@ conn->src_recv--; - result = sdp_advt_destroy(advt); - SDP_EXPECT(result >= 0); + sdp_advt_destroy(advt); } /* * If there are active reads, mark the connection as being in @@ -544,8 +519,7 @@ counter++; conn->snk_recv--; - result = sdp_advt_destroy(advt); - SDP_EXPECT(result >= 0); + sdp_advt_destroy(advt); } /* * A cancel ack is sent only if we cancelled an advertisment without @@ -568,10 +542,10 @@ /* * sdp_rcvd_snk_cancel_ack - sink cancel confirmantion */ -static int sdp_rcvd_snk_cancel_ack(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_snk_cancel_ack(struct sdp_opt *conn, + struct sdpc_buff *buff) { struct sdpc_iocb *iocb; - int result; sdp_dbg_ctrl(conn, "Sink Cancel Ack. actv <%d> mode <%d> flag <%08x>", conn->snk_sent, conn->recv_mode, conn->flags); @@ -579,28 +553,20 @@ if (!(conn->flags & SDP_CONN_F_SNK_CANCEL)) { sdp_dbg_warn(conn, "Connection not in sink cancel mode <%08x>", conn->flags); - result = -EPROTO; - goto done; + return -EPROTO; } /* * drain and complete all active IOCBs */ while ((iocb = sdp_iocb_q_get_head(&conn->r_snk))) { - conn->snk_sent--; - result = sdp_iocb_complete(iocb, 0); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>", - result, iocb->key); - goto done; - } + conn->snk_sent--; + sdp_iocb_complete(iocb, 0); } /* * cancellation is complete. Cancel flag is cleared in RECV post. */ return 0; -done: - return result; } /* @@ -675,22 +641,19 @@ if (conn->send_mode != SDP_MODE_PIPE) { sdp_dbg_warn(conn, "SinkAvail, incorrect source mode <%d>", conn->send_mode); - result = -EPROTO; - goto error; + return -EPROTO; } if (SDP_MSG_MAX_ADVS == (conn->src_recv + conn->snk_recv)) { sdp_dbg_warn(conn, "SinkAvail, too many advertisments. <%d>", (conn->src_recv + conn->snk_recv)); - result = -EPROTO; - goto error; + return -EPROTO; } if (snkah->size < conn->send_size) { sdp_dbg_warn(conn, "SinkAvail too small. <%d:%d>", snkah->size, conn->send_size); - result = -EPROTO; - goto error; + return -EPROTO; } /* * Save the advertisment, if it's not stale. otherwise update @@ -716,15 +679,11 @@ * in cancel processing they need to be * completed. */ - if (!(iocb->flags & SDP_IOCB_F_CANCEL)) { - result = sdp_desc_q_put_head(&conn->send_queue, - (struct sdpc_desc *) - iocb); - SDP_EXPECT(result >= 0); - } else { - result = sdp_iocb_complete(iocb, 0); - SDP_EXPECT(result >= 0); - } + if (!(iocb->flags & SDP_IOCB_F_CANCEL)) + sdp_desc_q_put_head(&conn->send_queue, + (struct sdpc_desc *)iocb); + else + sdp_iocb_complete(iocb, 0); } /* * If Source Cancel was in process, it should now @@ -741,8 +700,7 @@ advt = sdp_advt_create(); if (!advt) { sdp_dbg_warn(conn, "SrcAvail cannot be copied."); - result = -ENOMEM; - goto error; + return -ENOMEM; } advt->post = 0; @@ -783,7 +741,6 @@ * PostRecv will take care of consuming this advertisment, based * on result. */ -error: return result; } @@ -948,7 +905,7 @@ return result; advt_error: - (void)sdp_advt_destroy(advt); + sdp_advt_destroy(advt); done: return result; } @@ -1187,8 +1144,7 @@ * process result. */ if (!result) { - result = sdp_buff_pool_put(buff); - SDP_EXPECT(result >= 0); + sdp_buff_pool_put(buff); /* * If this buffer was consumed, then make sure sufficient * recv buffers are posted. Also we might be able to move @@ -1226,7 +1182,7 @@ return 0; drop: - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); done: return result; } Index: infiniband/ulp/sdp/sdp_inet.c =================================================================== --- infiniband/ulp/sdp/sdp_inet.c (revision 2593) +++ infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -401,8 +401,7 @@ sdp_iocb_q_cancel_all_write(conn, -ECANCELED); - result = sdp_inet_disconnect(conn); - SDP_EXPECT(result >= 0); + (void)sdp_inet_disconnect(conn); } #endif } @@ -1137,10 +1136,8 @@ case TCP_NODELAY: conn->nodelay = value ? 1 : 0; - if (conn->nodelay > 0) { - result = sdp_send_flush(conn); - SDP_EXPECT(result >= 0); - } + if (conn->nodelay > 0) + (void)sdp_send_flush(conn); break; case SDP_ZCOPY_THRSH: Index: infiniband/ulp/sdp/sdp_proto.h =================================================================== --- infiniband/ulp/sdp/sdp_proto.h (revision 2593) +++ infiniband/ulp/sdp/sdp_proto.h (working copy) @@ -49,9 +49,9 @@ */ struct sdpc_buff *sdp_buff_pool_get(void); -int sdp_buff_pool_put(struct sdpc_buff *buff); +void sdp_buff_pool_put(struct sdpc_buff *buff); -int sdp_buff_pool_chain_put(struct sdpc_buff *buff, u32 count); +void sdp_buff_pool_chain_put(struct sdpc_buff *buff, u32 count); void sdp_buff_pool_chain_link(struct sdpc_buff *head, struct sdpc_buff *buff); @@ -68,8 +68,6 @@ sdp_buff_q_clear_unmap(pool, NULL, 0); } -int sdp_buff_q_remove(struct sdpc_buff *buff); - struct sdpc_buff *sdp_buff_q_get(struct sdpc_buff_q *pool); struct sdpc_buff *sdp_buff_q_get_head(struct sdpc_buff_q *pool); @@ -78,17 +76,12 @@ struct sdpc_buff *sdp_buff_q_look_head(struct sdpc_buff_q *pool); -int sdp_buff_q_put(struct sdpc_buff_q *pool, struct sdpc_buff *buff); +void sdp_buff_q_put(struct sdpc_buff_q *pool, struct sdpc_buff *buff); -int sdp_buff_q_put_head(struct sdpc_buff_q *pool, struct sdpc_buff *buff); +void sdp_buff_q_put_head(struct sdpc_buff_q *pool, struct sdpc_buff *buff); -int sdp_buff_q_put_tail(struct sdpc_buff_q *pool, struct sdpc_buff *buff); +void sdp_buff_q_put_tail(struct sdpc_buff_q *pool, struct sdpc_buff *buff); -struct sdpc_buff *sdp_buff_q_fetch_head(struct sdpc_buff_q *pool, - int (*test)(struct sdpc_buff *buff, - void *arg), - void *usr_arg); - int sdp_buff_q_trav_head(struct sdpc_buff_q *pool, int (*trav_func)(struct sdpc_buff *buff, void *arg), @@ -133,7 +126,7 @@ struct sdpc_advt *sdp_advt_create(void); -int sdp_advt_destroy(struct sdpc_advt *advt); +void sdp_advt_destroy(struct sdpc_advt *advt); struct sdpc_advt *sdp_advt_q_get(struct sdpc_advt_q *table); @@ -154,7 +147,7 @@ struct sdpc_iocb *sdp_iocb_create(void); -int sdp_iocb_destroy(struct sdpc_iocb *iocb); +void sdp_iocb_destroy(struct sdpc_iocb *iocb); struct sdpc_iocb *sdp_iocb_q_look(struct sdpc_iocb_q *table); @@ -162,9 +155,9 @@ struct sdpc_iocb *sdp_iocb_q_get_tail(struct sdpc_iocb_q *table); -int sdp_iocb_q_put_head(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb); +void sdp_iocb_q_put_head(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb); -int sdp_iocb_q_put_tail(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb); +void sdp_iocb_q_put_tail(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb); struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key); @@ -176,7 +169,7 @@ void sdp_iocb_release(struct sdpc_iocb *iocb); -int sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status); +void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status); int sdp_iocb_lock(struct sdpc_iocb *iocb); @@ -191,11 +184,11 @@ struct sdpc_desc *sdp_desc_q_get_tail(struct sdpc_desc_q *table); -int sdp_desc_q_put_head(struct sdpc_desc_q *table, - struct sdpc_desc *element); +void sdp_desc_q_put_head(struct sdpc_desc_q *table, + struct sdpc_desc *element); -int sdp_desc_q_put_tail(struct sdpc_desc_q *table, - struct sdpc_desc *element); +void sdp_desc_q_put_tail(struct sdpc_desc_q *table, + struct sdpc_desc *element); struct sdpc_desc *sdp_desc_q_look_head(struct sdpc_desc_q *table); @@ -573,7 +566,7 @@ /* * sdp_conn_stat_dump - dump stats to the log */ -static inline int sdp_conn_stat_dump(struct sdp_opt *conn) +static inline void sdp_conn_stat_dump(struct sdp_opt *conn) { #ifdef _SDP_CONN_STATS_REC int counter; @@ -590,7 +583,6 @@ conn->recv_mid[counter]); } #endif - return 0; } /* Index: infiniband/ulp/sdp/sdp_read.c =================================================================== --- infiniband/ulp/sdp/sdp_read.c (revision 2588) +++ infiniband/ulp/sdp/sdp_read.c (working copy) @@ -62,15 +62,13 @@ } advt = sdp_advt_q_get(&conn->src_actv); - SDP_EXPECT((advt)); conn->src_recv--; result = sdp_send_ctrl_rdma_rd(conn, advt->post); SDP_EXPECT(result >= 0); - result = sdp_advt_destroy(advt); - SDP_EXPECT(result >= 0); + sdp_advt_destroy(advt); /* * If a SrcAvailCancel was received, and all RDMA reads * have been flushed, perform tail processing @@ -161,15 +159,14 @@ type = sdp_desc_q_type_head(&conn->r_src); switch (type) { case SDP_DESC_TYPE_BUFF: - buff = (struct sdpc_buff *) sdp_desc_q_get_head(&conn->r_src); - SDP_EXPECT((buff)); + buff = (struct sdpc_buff *)sdp_desc_q_get_head(&conn->r_src); if (comp->wr_id != buff->wrid) { sdp_dbg_warn(conn, "work request mismatch <%llu:%llu>", (unsigned long long)comp->wr_id, (unsigned long long)buff->wrid); - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); result = -EPROTO; goto done; } @@ -191,8 +188,7 @@ sdp_dbg_warn(conn, "Error <%d> receiving buff", result); - result = sdp_buff_pool_put(buff); - SDP_EXPECT(result >= 0); + sdp_buff_pool_put(buff); } break; @@ -202,19 +198,13 @@ break; iocb = (struct sdpc_iocb *)sdp_desc_q_get_head(&conn->r_src); - SDP_EXPECT((iocb)); iocb->flags &= ~(SDP_IOCB_F_ACTIVE | SDP_IOCB_F_RDMA_R); SDP_CONN_STAT_READ_INC(conn, iocb->post); SDP_CONN_STAT_RQ_DEC(conn, iocb->size); - result = sdp_iocb_complete(iocb, 0); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>", - result, iocb->key); - goto done; - } + sdp_iocb_complete(iocb, 0); break; case SDP_DESC_TYPE_NONE: @@ -232,18 +222,10 @@ if (conn->sk->sk_rcvlowat > iocb->post) break; - iocb = sdp_iocb_q_get_head(&conn->r_pend); - SDP_EXPECT((iocb)); - SDP_CONN_STAT_READ_INC(conn, iocb->post); SDP_CONN_STAT_RQ_DEC(conn, iocb->size); - result = sdp_iocb_complete(iocb, 0); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>", - result, iocb->key); - goto done; - } + sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0); break; default: Index: infiniband/ulp/sdp/sdp_send.c =================================================================== --- infiniband/ulp/sdp/sdp_send.c (revision 2588) +++ infiniband/ulp/sdp/sdp_send.c (working copy) @@ -81,7 +81,8 @@ * the flag. This allows for at least one pending urgent message * to send early notification. */ - if ((conn->flags & SDP_CONN_F_OOB_SEND) && conn->oob_offset <= 0xFFFF) { + if ((conn->flags & SDP_CONN_F_OOB_SEND) && + conn->oob_offset <= 0xFFFF) { SDP_BSDH_SET_OOB_PEND(buff->bsdh_hdr); SDP_BUFF_F_SET_SE(buff); @@ -123,16 +124,12 @@ /* * check queue membership. (first send attempt vs. flush) */ - if (sdp_desc_q_member((struct sdpc_desc *) buff) > 0) + if (sdp_desc_q_member((struct sdpc_desc *) buff)) sdp_desc_q_remove((struct sdpc_desc *) buff); /* * save the buffer for the event handler. */ - result = sdp_buff_q_put_tail(&conn->send_post, buff); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> queueing send buffer", result); - goto done; - } + sdp_buff_q_put_tail(&conn->send_post, buff); /* * post send */ @@ -182,9 +179,9 @@ /* * sdp_send_data_buff_post - Post data for buffered transmission */ -static int sdp_send_data_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_send_data_buff_post(struct sdp_opt *conn, + struct sdpc_buff *buff) { - struct sdpc_advt *advt; int result; /* @@ -253,7 +250,7 @@ * update non-discard counter. * Make consideration for a pending sink. (can be forced by OOB) */ - if (sdp_advt_q_size(&conn->snk_pend) > 0) { + if (sdp_advt_q_size(&conn->snk_pend)) { /* * As sink advertisment needs to be discarded. We always * complete an advertisment if there is not enough room @@ -263,11 +260,7 @@ * (remember the spec makes sure that the sink is bigger * then the buffer.) */ - advt = sdp_advt_q_get(&conn->snk_pend); - SDP_EXPECT((advt)); - - result = sdp_advt_destroy(advt); - SDP_EXPECT(result >= 0); + sdp_advt_destroy(sdp_advt_q_get(&conn->snk_pend)); /* * update sink advertisments. */ @@ -281,12 +274,10 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> posting send data buffer", result); - goto error; + return result; } return 0; -error: - return result; } /* @@ -298,7 +289,6 @@ struct ib_send_wr *bad_wr; struct sdpc_advt *advt; int result; - int zcopy; /* * check state to determine OK to send: @@ -367,12 +357,7 @@ if (sdp_desc_q_member((struct sdpc_desc *) buff) > 0) sdp_desc_q_remove((struct sdpc_desc *) buff); - result = sdp_desc_q_put_tail(&conn->w_snk, (struct sdpc_desc *)buff); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> queueing write buffer. <%d>", - result, sdp_desc_q_size(&conn->w_snk)); - goto error; - } + sdp_desc_q_put_tail(&conn->w_snk, (struct sdpc_desc *)buff); /* * update send queue depth */ @@ -402,24 +387,18 @@ * advertisment. */ if (conn->send_size > advt->size) { - advt = sdp_advt_q_get(&conn->snk_pend); - SDP_EXPECT((advt)); - zcopy = advt->post; - - result = sdp_advt_destroy(advt); - SDP_EXPECT(result >= 0); - - result = sdp_send_ctrl_rdma_wr(conn, zcopy); + result = sdp_send_ctrl_rdma_wr(conn, advt->post); if (result < 0) { sdp_dbg_warn(conn, "Error <%d> completing sink. <%d>", - result, zcopy); + result, advt->post); result = -ENODEV; goto error; } /* * update sink advertisments. */ + sdp_advt_destroy(sdp_advt_q_get(&conn->snk_pend)); conn->snk_recv--; } @@ -523,19 +502,7 @@ if (conn->send_size <= advt->size) continue; - advt = sdp_advt_q_get(&conn->snk_pend); - if (!advt) { - sdp_dbg_warn(conn, "sink advertisment disappeared."); - result = -ENODEV; - goto error; - } - - zcopy = advt->post; - - result = sdp_advt_destroy(advt); - SDP_EXPECT(result >= 0); - - result = sdp_send_ctrl_rdma_wr(conn, zcopy); + result = sdp_send_ctrl_rdma_wr(conn, advt->post); if (result < 0) { sdp_dbg_warn(conn, "Error <%d> completing sink. <%d>", result, zcopy); @@ -545,6 +512,7 @@ /* * update sink advertisments. */ + sdp_advt_destroy(sdp_advt_q_get(&conn->snk_pend)); conn->snk_recv--; } @@ -715,7 +683,7 @@ error: iocb->flags &= ~(SDP_IOCB_F_RDMA_R | SDP_IOCB_F_ACTIVE); drop: - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); return result; } @@ -832,7 +800,7 @@ return iocb->len; drop: - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); error: return result; } @@ -865,13 +833,8 @@ if (sdp_desc_q_member((struct sdpc_desc *)iocb) > 0) sdp_desc_q_remove((struct sdpc_desc *)iocb); - result = sdp_desc_q_put_tail(&conn->w_snk, - (struct sdpc_desc *)iocb); - if (result < 0) { - sdp_dbg_warn(conn, - "Error <%d> queuing write IOCB.", - result); - } + sdp_desc_q_put_tail(&conn->w_snk, + (struct sdpc_desc *)iocb); } goto done; @@ -899,11 +862,7 @@ SDP_CONN_STAT_WRITE_INC(conn, iocb->post); SDP_CONN_STAT_WQ_DEC(conn, iocb->size); - result = sdp_iocb_complete(iocb, 0); - if (result < 0) - sdp_dbg_warn(conn, - "Error <%d> completing iocb <%d>", - result, iocb->key); + sdp_iocb_complete(iocb, 0); } goto done; @@ -917,11 +876,7 @@ if (sdp_desc_q_member((struct sdpc_desc *) iocb) > 0) sdp_desc_q_remove((struct sdpc_desc *)iocb); - result = sdp_iocb_q_put_tail(&conn->w_src, iocb); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> queueing write <%d:%d>", - result, iocb->key, - sdp_iocb_q_size(&conn->w_src)); + sdp_iocb_q_put_tail(&conn->w_src, iocb); } done: @@ -972,10 +927,9 @@ * non-zero result is generated. * (positive: no space; negative: error) */ - while (!result && sdp_desc_q_size(&conn->send_queue) > 0) { - element = sdp_desc_q_look_head(&conn->send_queue); - SDP_EXPECT((element)); - + while (!result && + (element = sdp_desc_q_look_head(&conn->send_queue))) { + result = sdp_send_data_queue_test(conn, element); if (!result) continue; @@ -991,11 +945,9 @@ * since called functions can dequeue the * element, and not know how to requeue it. */ - if (!sdp_desc_q_member(element)) { - result = sdp_desc_q_put_head(&conn->send_queue, - element); - SDP_EXPECT(result >= 0); - } + if (!sdp_desc_q_member(element)) + sdp_desc_q_put_head(&conn->send_queue, + element); } } @@ -1018,12 +970,8 @@ */ if (sdp_desc_q_size(&conn->send_queue) > 0 || (result = sdp_send_data_queue_test(conn, element)) > 0) { - result = sdp_desc_q_put_tail(&conn->send_queue, element); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> queueing data for send", - result); - goto done; - } + + sdp_desc_q_put_tail(&conn->send_queue, element); /* * Potentially request a switch to pipelined mode. */ @@ -1084,7 +1032,6 @@ int urg) { int result = 0; - int expect; /* * See note on send OOB implementation in SendBuffPost. @@ -1121,8 +1068,7 @@ sdp_dbg_warn(conn, "Error <%d> buffer to SEND queue.", result); - expect = sdp_buff_pool_put(buff); - SDP_EXPECT(expect >= 0); + sdp_buff_pool_put(buff); } } @@ -1171,9 +1117,8 @@ * As long as there are buffers, try to post until a non-zero * result is generated. (positive: no space; negative: error) */ - while (!result && sdp_desc_q_size(&conn->send_ctrl) > 0) { - element = sdp_desc_q_look_head(&conn->send_ctrl); - SDP_EXPECT((element)); + while (!result && + (element = sdp_desc_q_look_head(&conn->send_ctrl))) { result = sdp_send_ctrl_buff_test(conn, (struct sdpc_buff *)element); @@ -1185,11 +1130,8 @@ "Error <%d> failed to flush control msg", result); - if (!sdp_desc_q_member(element)) { - result = sdp_desc_q_put_head(&conn->send_ctrl, - element); - SDP_EXPECT(result >= 0); - } + if (!sdp_desc_q_member(element)) + sdp_desc_q_put_head(&conn->send_ctrl, element); } } @@ -1208,18 +1150,12 @@ * Either post a send, or buffer the packet in the tx queue */ if (sdp_desc_q_size(&conn->send_ctrl) > 0 || - (result = sdp_send_ctrl_buff_test(conn, buff)) > 0) { + (result = sdp_send_ctrl_buff_test(conn, buff)) > 0) /* * save the buffer for later flushing into the post queue. */ - result = sdp_desc_q_put_tail(&conn->send_ctrl, - (struct sdpc_desc *)buff); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> queueing control buff", - result); - goto error; - } - } + sdp_desc_q_put_tail(&conn->send_ctrl, + (struct sdpc_desc *)buff); if (result < 0) { sdp_dbg_warn(conn, "Error <%d> during control send posting", @@ -1278,7 +1214,7 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> posting control message", result); - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); } return result; @@ -1319,7 +1255,7 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> posting control message", result); - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); goto error; } @@ -1462,7 +1398,7 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> posting control message", result); - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); } error: @@ -1531,7 +1467,7 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> posting control message", result); - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); } error: @@ -1588,7 +1524,7 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> posting control message", result); - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); } error: @@ -1736,7 +1672,7 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> posting control message", result); - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); } error: @@ -1804,16 +1740,13 @@ sdp_desc_q_remove((struct sdpc_desc *)iocb); if (iocb->flags & SDP_IOCB_F_ACTIVE) { - if (iocb->flags & SDP_IOCB_F_RDMA_W) { - result = sdp_desc_q_put_tail(&conn->w_snk, - (struct sdpc_desc *)iocb); - SDP_EXPECT(result >= 0); - } else { + if (iocb->flags & SDP_IOCB_F_RDMA_W) + sdp_desc_q_put_tail(&conn->w_snk, + (struct sdpc_desc *)iocb); + else { SDP_EXPECT((iocb->flags & SDP_IOCB_F_RDMA_R)); - result = sdp_iocb_q_put_tail(&conn->w_src, - iocb); - SDP_EXPECT(result >= 0); + sdp_iocb_q_put_tail(&conn->w_src, iocb); } } else { /* @@ -1821,13 +1754,10 @@ * needs to be compelted. */ if (iocb->post > 0) { - result = sdp_iocb_complete(iocb, 0); - SDP_EXPECT(result >= 0); - + sdp_iocb_complete(iocb, 0); result = -EAGAIN; } else { - result = sdp_iocb_destroy(iocb); - SDP_EXPECT(result >= 0); + sdp_iocb_destroy(iocb); /* * completion reference */ @@ -1924,14 +1854,11 @@ */ advt = sdp_advt_q_look(&conn->snk_pend); if (advt && advt->post > 0) { - advt = sdp_advt_q_get(&conn->snk_pend); - SDP_EXPECT((advt)); result = sdp_send_ctrl_rdma_wr(conn, advt->post); SDP_EXPECT(result >= 0); - result = sdp_advt_destroy(advt); - SDP_EXPECT(result >= 0); + sdp_advt_destroy(sdp_advt_q_get(&conn->snk_pend)); /* * update sink advertisments. */ @@ -2080,7 +2007,7 @@ msg->msg_iov, copy); if (result < 0) { - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); goto done; } #endif @@ -2196,7 +2123,7 @@ sdp_dbg_warn(conn, "Error <%d> locking IOCB <%Zu:%d>", result, size, copied); - (void)sdp_iocb_destroy(iocb); + sdp_iocb_destroy(iocb); break; } @@ -2209,7 +2136,7 @@ sdp_dbg_warn(conn, "Error <%d> queueing write IOCB", result); - (void)sdp_iocb_destroy(iocb); + sdp_iocb_destroy(iocb); break; } Index: infiniband/ulp/sdp/sdp_actv.c =================================================================== --- infiniband/ulp/sdp/sdp_actv.c (revision 2588) +++ infiniband/ulp/sdp/sdp_actv.c (working copy) @@ -309,7 +309,7 @@ /* * Pop the hello message that was sent */ - (void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post)); + sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post)); result = sdp_cm_actv_establish(conn); if (result) { @@ -343,7 +343,6 @@ struct sdp_opt *conn = (struct sdp_opt *) arg; struct sdpc_buff *buff; int result = 0; - int expect; /* * lock the socket */ @@ -441,16 +440,7 @@ /* * save message */ - result = sdp_buff_q_put(&conn->send_post, buff); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> buffering hello msg.", result); - - expect = sdp_buff_pool_put(buff); - SDP_EXPECT(expect >= 0); - - status = -EPROTO; - goto failed; - } + sdp_buff_q_put_tail(&conn->send_post, buff); #if 1 /* * Mellanox performance bug workaround. Index: infiniband/ulp/sdp/sdp_conn.c =================================================================== --- infiniband/ulp/sdp/sdp_conn.c (revision 2588) +++ infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -758,8 +758,7 @@ result, dev_root_s.sk_entry, dev_root_s.sk_size); - result = sdp_conn_stat_dump(conn); - SDP_EXPECT(result >= 0); + sdp_conn_stat_dump(conn); /* * really there shouldn't be anything in these tables, but it's * really bad if we leave a dangling reference here. Index: infiniband/ulp/sdp/sdp_advt.c =================================================================== --- infiniband/ulp/sdp/sdp_advt.c (revision 2588) +++ infiniband/ulp/sdp/sdp_advt.c (working copy) @@ -53,12 +53,7 @@ advt = kmem_cache_alloc(sdp_advt_cache, SLAB_ATOMIC); if (advt) { - advt->next = NULL; - advt->prev = NULL; - advt->size = 0; - advt->post = 0; - advt->addr = 0; - advt->rkey = 0; + memset(advt, 0, sizeof(*advt)); advt->type = SDP_DESC_TYPE_ADVT; advt->release = sdp_advt_destroy; @@ -70,16 +65,13 @@ /* * sdp_advt_destroy - destroy an advertisment object */ -int sdp_advt_destroy(struct sdpc_advt *advt) +void sdp_advt_destroy(struct sdpc_advt *advt) { - if (advt->next || advt->prev) - return -EACCES; + BUG_ON(advt->next || advt->prev); /* * return the object to its cache */ kmem_cache_free(sdp_advt_cache, advt); - - return 0; } /* @@ -130,6 +122,8 @@ struct sdpc_advt *next; struct sdpc_advt *prev; + BUG_ON(advt->table); + if (!table->head) { advt->next = advt; advt->prev = advt; @@ -163,15 +157,11 @@ void sdp_advt_q_clear(struct sdpc_advt_q *table) { struct sdpc_advt *advt; - int result; - /* * drain the table of any objects */ - while ((advt = sdp_advt_q_get(table))) { - result = sdp_advt_destroy(advt); - SDP_EXPECT(result >= 0); - } + while ((advt = sdp_advt_q_get(table))) + sdp_advt_destroy(advt); } /* Index: infiniband/ulp/sdp/sdp_recv.c =================================================================== --- infiniband/ulp/sdp/sdp_recv.c (revision 2588) +++ infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -74,11 +74,7 @@ * actually posting the thing. Completion event can happen before * post function returns. */ - result = sdp_buff_q_put_tail(&conn->recv_post, buff); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> queuing recv buffer.", result); - goto drop; - } + sdp_buff_q_put_tail(&conn->recv_post, buff); sdp_dbg_data(conn, "POST RECV BUFF wrid <%llu> of <%u> bytes.", (unsigned long long) buff->wrid, @@ -106,7 +102,7 @@ return 0; drop: - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); conn->l_recv_bf--; error: return result; @@ -180,11 +176,7 @@ * actually posting the thing. Completion event can happen before * post function returns. */ - result = sdp_desc_q_put_tail(&conn->r_src, (struct sdpc_desc *) buff); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> queuing rdma read.", result); - goto drop; - } + sdp_desc_q_put_tail(&conn->r_src, (struct sdpc_desc *) buff); sdp_dbg_data(conn, "POST READ BUFF wrid <%llu> of <%u> bytes.", (unsigned long long) buff->wrid, @@ -216,7 +208,7 @@ return 0; drop: - (void)sdp_buff_pool_put(buff); + sdp_buff_pool_put(buff); error: done: return result; @@ -306,25 +298,11 @@ /* * if there is no more iocb space queue the it for completion */ - if (!iocb->len) { - iocb = sdp_iocb_q_get_head(&conn->r_pend); - if (!iocb) { - sdp_dbg_warn(conn, "read IOCB disappeared. <%d>", - sdp_iocb_q_size(&conn->r_pend)); - result = -ENODEV; - goto error; - } + if (!iocb->len) + sdp_desc_q_put_tail(&conn->r_src, + (struct sdpc_desc *) + sdp_iocb_q_get_head(&conn->r_pend)); - result = sdp_desc_q_put_tail(&conn->r_src, - (struct sdpc_desc *)iocb); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> queuing read IOCB", - result); - (void)sdp_iocb_destroy(iocb); - goto error; - } - } - sdp_dbg_data(conn, "POST READ IOCB wrid <%llu> bytes <%u:%d:%d>.", (unsigned long long) iocb->wrid, zcopy, iocb->len, advt->size); @@ -410,19 +388,7 @@ /* * queue IOCB */ - iocb = sdp_iocb_q_get_head(&conn->r_pend); - if (result < 0) { - sdp_dbg_warn(conn, "read IOCB missing from pending table <%d>", - sdp_iocb_q_size(&conn->r_pend)); - goto release; - } - - result = sdp_iocb_q_put_tail(&conn->r_snk, iocb); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> queueing active write IOCB", - result); - goto re_q; - } + sdp_iocb_q_put_tail(&conn->r_snk, sdp_iocb_q_get_head(&conn->r_pend)); /* * Either post a send, or buffer the packet in the tx queue */ @@ -433,21 +399,19 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> sending SnkAvail message", result); - goto de_q; + + sdp_iocb_q_put_head(&conn->r_pend, + sdp_iocb_q_get_tail(&conn->r_snk)); + + iocb->flags &= ~SDP_IOCB_F_ACTIVE; + iocb->flags &= ~SDP_IOCB_F_RDMA_W; + + goto error; } conn->snk_sent++; return 0; -de_q: - iocb = sdp_iocb_q_get_tail(&conn->r_snk); -re_q: - (void)sdp_iocb_q_put_head(&conn->r_pend, iocb); -release: - iocb->flags &= ~SDP_IOCB_F_ACTIVE; - iocb->flags &= ~SDP_IOCB_F_RDMA_W; - - sdp_iocb_release(iocb); error: return result; } @@ -727,7 +691,7 @@ result, iocb->len, (unsigned)(buff->tail - buff->data)); - (void)sdp_iocb_q_put_head(&conn->r_snk, iocb); + sdp_iocb_q_put_head(&conn->r_snk, iocb); return result; } @@ -739,10 +703,7 @@ /* * callback to complete IOCB */ - result = sdp_iocb_complete(iocb, 0); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>", - result, iocb->key); + sdp_iocb_complete(iocb, 0); return (buff->tail - buff->data); } @@ -790,18 +751,12 @@ /* * complete IOCB */ - iocb = sdp_iocb_q_get_head(&conn->r_pend); - SDP_EXPECT((iocb)); - SDP_CONN_STAT_READ_INC(conn, iocb->post); SDP_CONN_STAT_RQ_DEC(conn, iocb->size); /* * callback to complete IOCB */ - result = sdp_iocb_complete(iocb, 0); - if (result < 0) - sdp_dbg_warn(conn, "Error <%d> completing iocb. <%d>", - result, iocb->key); + sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0); } return (buff->tail - buff->data); @@ -888,10 +843,8 @@ */ buffered = buff->tail - buff->data; - if (buffered > 0) { - result = sdp_buff_q_put_tail(&conn->recv_pool, buff); - SDP_EXPECT(result >= 0); - } + if (buffered) + sdp_buff_q_put_tail(&conn->recv_pool, buff); return buffered; done: @@ -962,14 +915,11 @@ /* * callback to complete IOCB, or drop reference */ - result = sdp_iocb_complete(iocb, 0); - SDP_EXPECT(result >= 0); - + sdp_iocb_complete(iocb, 0); result = -EAGAIN; } else { - result = sdp_iocb_destroy(iocb); - SDP_EXPECT(result >= 0); + sdp_iocb_destroy(iocb); /* * completion reference */ @@ -981,16 +931,13 @@ goto unlock; } - if (iocb->flags & SDP_IOCB_F_RDMA_W) { - result = sdp_iocb_q_put_tail(&conn->r_snk, iocb); - SDP_EXPECT(result >= 0); - } + if (iocb->flags & SDP_IOCB_F_RDMA_W) + sdp_iocb_q_put_tail(&conn->r_snk, iocb); else { SDP_EXPECT((iocb->flags & SDP_IOCB_F_RDMA_R)); - result = sdp_desc_q_put_tail(&conn->r_src, - (struct sdpc_desc *)iocb); - SDP_EXPECT(result >= 0); + sdp_desc_q_put_tail(&conn->r_src, + (struct sdpc_desc *)iocb); } } /* @@ -1135,8 +1082,7 @@ sdp_inet_recv_urg_test, (void *)0); if (buff) { - result = sdp_buff_pool_put(buff); - SDP_EXPECT(result >= 0); + sdp_buff_pool_put(buff); result = sdp_recv_flush(conn); SDP_EXPECT(result >= 0); @@ -1279,12 +1225,8 @@ buff->data, copy); if (result < 0) { - expect = - sdp_buff_q_put_head(&conn-> - recv_pool, - buff); - SDP_EXPECT(expect >= 0); - + sdp_buff_q_put_head(&conn->recv_pool, + buff); goto done; } #endif @@ -1298,9 +1240,7 @@ copied += copy; if ((buff->tail - buff->data) > 0) { - expect = sdp_buff_q_put_head(&conn->recv_pool, - buff); - SDP_EXPECT(expect >= 0); + sdp_buff_q_put_head(&conn->recv_pool, buff); /* * always break, PEEK and OOB together could * throw us into a loop without a forced @@ -1312,11 +1252,9 @@ break; } - if (flags & MSG_PEEK) { - expect = sdp_buff_q_put_head(&peek_queue, - buff); - SDP_EXPECT(expect >= 0); - } else { + if (flags & MSG_PEEK) + sdp_buff_q_put_head(&peek_queue, buff); + else { if (buff->flags & SDP_BUFF_F_OOB_PRES) conn->rcv_urg_cnt -= 1; /* @@ -1459,21 +1397,13 @@ "Error <%d> IOCB lock <%Zu:%d>", result, size, copied); - (void)sdp_iocb_destroy(iocb); + sdp_iocb_destroy(iocb); break; } SDP_CONN_STAT_RQ_INC(conn, iocb->size); - result = sdp_iocb_q_put_tail(&conn->r_pend, iocb); - if (result < 0) { - sdp_dbg_warn(conn, - "Error <%d> IOCB queue <%Zu:%d>", - result, size, copied); - - (void)sdp_iocb_destroy(iocb); - break; - } + sdp_iocb_q_put_tail(&conn->r_pend, iocb); ack = 1; copied = 0; /* copied amount was saved in IOCB. */ @@ -1494,16 +1424,13 @@ expect); } - (void)sdp_buff_pool_chain_put(head, free_count); + sdp_buff_pool_chain_put(head, free_count); /* * return any peeked buffers to the recv queue, in the correct order. */ - if (flags & MSG_PEEK) { - while ((buff = sdp_buff_q_get_tail(&peek_queue))) { - expect = sdp_buff_q_put_head(&conn->recv_pool, buff); - SDP_EXPECT(expect >= 0); - } - } + if (flags & MSG_PEEK) + while ((buff = sdp_buff_q_get_tail(&peek_queue))) + sdp_buff_q_put_head(&conn->recv_pool, buff); sdp_conn_unlock(conn); return ((copied > 0) ? copied : result); Index: infiniband/ulp/sdp/sdp_advt.h =================================================================== --- infiniband/ulp/sdp/sdp_advt.h (revision 2588) +++ infiniband/ulp/sdp/sdp_advt.h (working copy) @@ -50,7 +50,7 @@ struct sdpc_advt *prev; /* previous structure in table */ u32 type; /* element type. (for generic queue) */ struct sdpc_advt_q *table; /* table to which this object belongs */ - int (*release)(struct sdpc_advt *advt); /* release the object */ + void (*release)(struct sdpc_advt *advt); /* release the object */ /* * advertisment specific */ Index: infiniband/ulp/sdp/sdp_pass.c =================================================================== --- infiniband/ulp/sdp/sdp_pass.c (revision 2588) +++ infiniband/ulp/sdp/sdp_pass.c (working copy) @@ -49,7 +49,7 @@ /* * free hello ack message */ - (void)sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post)); + sdp_buff_pool_put(sdp_buff_q_get_head(&conn->send_post)); qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL); if (!qp_attr) { @@ -108,7 +108,6 @@ struct ib_qp_attr *qp_attr; int qp_mask = 0; int result; - int expect; /* * Accept connection, build listen response headers and send * a REP message to remote peer. @@ -159,16 +158,7 @@ /* * save message */ - result = sdp_buff_q_put(&conn->send_post, buff); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> buffering hello ack packet.", - result); - - expect = sdp_buff_pool_put(buff); - SDP_EXPECT(expect >= 0); - - goto error; - } + sdp_buff_q_put_tail(&conn->send_post, buff); /* * modify QP. INIT->RTR */ Index: infiniband/ulp/sdp/sdp_sent.c =================================================================== --- infiniband/ulp/sdp/sdp_sent.c (revision 2588) +++ infiniband/ulp/sdp/sdp_sent.c (working copy) @@ -420,8 +420,7 @@ break; } - result = sdp_buff_pool_chain_put(head, free_count); - SDP_EXPECT(result >= 0); + sdp_buff_pool_chain_put(head, free_count); if (free_count <= 0 || conn->send_usig < 0) { sdp_dbg_warn(conn, @@ -443,8 +442,8 @@ return 0; drop: - (void)sdp_buff_pool_put(buff); - (void)sdp_buff_pool_chain_put(head, free_count); + sdp_buff_pool_put(buff); + sdp_buff_pool_chain_put(head, free_count); done: return result; } Index: infiniband/ulp/sdp/sdp_iocb.c =================================================================== --- infiniband/ulp/sdp/sdp_iocb.c (revision 2588) +++ infiniband/ulp/sdp/sdp_iocb.c (working copy) @@ -485,7 +485,6 @@ static void do_iocb_complete(void *arg) { struct sdpc_iocb *iocb = (struct sdpc_iocb *)arg; - int result; long value; /* * release memory @@ -511,16 +510,13 @@ /* * delete IOCB */ - result = sdp_iocb_destroy(iocb); - if (result < 0) - sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>", - result, iocb->key, iocb->status); + sdp_iocb_destroy(iocb); } /* * sdp_iocb_complete - complete an IOCB */ -int sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status) +void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status) { iocb->status = status; @@ -529,8 +525,6 @@ schedule_work(&iocb->completion); } else do_iocb_complete(iocb); - - return 0; } /* @@ -607,13 +601,12 @@ /* * sdp_iocb_destroy - destroy an IOCB object */ -int sdp_iocb_destroy(struct sdpc_iocb *iocb) +void sdp_iocb_destroy(struct sdpc_iocb *iocb) { if (!iocb) - return -EINVAL; + return; - if (iocb->next || iocb->prev) - return -EACCES; + BUG_ON(iocb->next || iocb->prev); /* * release iocb registered memory */ @@ -640,8 +633,6 @@ * return the object to its cache */ kmem_cache_free(sdp_iocb_cache, iocb); - - return 0; } /* @@ -692,12 +683,15 @@ /* * sdp_iocb_q_put - put the IOCB object at the tables tail */ -static int sdp_iocb_q_put(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb, - int head) +static void sdp_iocb_q_put(struct sdpc_iocb_q *table, + struct sdpc_iocb *iocb, + int head) { struct sdpc_iocb *next; struct sdpc_iocb *prev; + BUG_ON(iocb->table); + if (!table->head) { iocb->next = iocb; iocb->prev = iocb; @@ -718,8 +712,6 @@ table->size++; iocb->table = table; - - return 0; } /* @@ -741,17 +733,17 @@ /* * sdp_iocb_q_put_tail - put the IOCB object at the tables tail */ -int sdp_iocb_q_put_tail(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb) +void sdp_iocb_q_put_tail(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb) { - return sdp_iocb_q_put(table, iocb, 0); + sdp_iocb_q_put(table, iocb, 0); } /* * sdp_iocb_q_put_head - put the IOCB object at the tables head */ -int sdp_iocb_q_put_head(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb) +void sdp_iocb_q_put_head(struct sdpc_iocb_q *table, struct sdpc_iocb *iocb) { - return sdp_iocb_q_put(table, iocb, 1); + sdp_iocb_q_put(table, iocb, 1); } /* @@ -762,7 +754,6 @@ struct sdpc_iocb *iocb; struct sdpc_iocb *next; int counter; - int result; int total; /* @@ -780,9 +771,7 @@ iocb->post, iocb->len); sdp_iocb_q_remove(iocb); - - result = sdp_iocb_complete(iocb, comp); - SDP_EXPECT(result >= 0); + sdp_iocb_complete(iocb, comp); } iocb = next; @@ -804,15 +793,11 @@ void sdp_iocb_q_clear(struct sdpc_iocb_q *table) { struct sdpc_iocb *iocb; - int result; - /* * drain the table of any objects */ - while ((iocb = sdp_iocb_q_get_head(table))) { - result = sdp_iocb_destroy(iocb); - SDP_EXPECT(result >= 0); - } + while ((iocb = sdp_iocb_q_get_head(table))) + sdp_iocb_destroy(iocb); } /* Index: infiniband/ulp/sdp/sdp_iocb.h =================================================================== --- infiniband/ulp/sdp/sdp_iocb.h (revision 2588) +++ infiniband/ulp/sdp/sdp_iocb.h (working copy) @@ -76,7 +76,7 @@ struct sdpc_iocb *prev; /* previous structure in table */ u32 type; /* element type. (for generic queue) */ struct sdpc_iocb_q *table; /* table to which this iocb belongs */ - int (*release)(struct sdpc_iocb *iocb); /* release the object */ + void (*release)(struct sdpc_iocb *iocb); /* release the object */ /* * iocb sepcific */ Index: infiniband/ulp/sdp/sdp_buff.c =================================================================== --- infiniband/ulp/sdp/sdp_buff.c (revision 2588) +++ infiniband/ulp/sdp/sdp_buff.c (working copy) @@ -84,12 +84,11 @@ /* * do_buff_q_put - Place a buffer into a specific pool */ -static inline int do_buff_q_put(struct sdpc_buff_q *pool, - struct sdpc_buff *buff, int fifo) +static inline void do_buff_q_put(struct sdpc_buff_q *pool, + struct sdpc_buff *buff, int fifo) { /* fifo: false == tail, true == head */ - if (buff->pool) - return -EINVAL; + BUG_ON(buff->pool); if (!pool->head) { buff->next = buff; @@ -108,8 +107,6 @@ pool->size++; buff->pool = pool; - - return 0; } /* @@ -127,14 +124,13 @@ /* * do_buff_q_remove - remove a specific buffer from a specific pool */ -static inline int do_buff_q_remove(struct sdpc_buff_q *pool, - struct sdpc_buff *buff) +static inline void do_buff_q_remove(struct sdpc_buff_q *pool, + struct sdpc_buff *buff) { struct sdpc_buff *prev; struct sdpc_buff *next; - if (pool != buff->pool) - return -EINVAL; + BUG_ON(pool != buff->pool); if (buff->next == buff && buff->prev == buff) pool->head = NULL; @@ -153,8 +149,6 @@ buff->pool = NULL; buff->next = NULL; buff->prev = NULL; - - return 0; } /* @@ -167,17 +161,6 @@ } /* - * sdp_buff_q_remove - remove a specific buffer from a specific pool - */ -int sdp_buff_q_remove(struct sdpc_buff *buff) -{ - struct sdpc_buff_q *pool; - - pool = buff->pool; - return do_buff_q_remove(pool, buff); -} - -/* * sdp_buff_q_get - Get a buffer from a specific pool */ struct sdpc_buff *sdp_buff_q_get(struct sdpc_buff_q *pool) @@ -210,17 +193,6 @@ } /* - * sdp_buff_q_fetch_head - Get the pools first buffer, if the test passes - */ -struct sdpc_buff *sdp_buff_q_fetch_head(struct sdpc_buff_q *pool, - int (*test)(struct sdpc_buff *buff, - void *arg), - void *usr_arg) -{ - return do_buff_q_get(pool, 1, test, usr_arg); -} - -/* * sdp_buff_q_fetch - Get the first matching buffer from the pool */ struct sdpc_buff *sdp_buff_q_fetch(struct sdpc_buff_q *pool, @@ -243,9 +215,7 @@ counter < pool->size; counter++, buff = buff->next) { result = test(buff, usr_arg); if (result > 0) { - result = do_buff_q_remove(pool, buff); - SDP_EXPECT(result >= 0); - + do_buff_q_remove(pool, buff); return buff; } @@ -289,25 +259,25 @@ /* * sdp_buff_q_put - Place a buffer into a specific pool */ -int sdp_buff_q_put(struct sdpc_buff_q *pool, struct sdpc_buff *buff) +void sdp_buff_q_put(struct sdpc_buff_q *pool, struct sdpc_buff *buff) { - return do_buff_q_put(pool, buff, 1); + do_buff_q_put(pool, buff, 1); } /* * sdp_buff_q_put_head - Place a buffer into the head of a specific pool */ -int sdp_buff_q_put_head(struct sdpc_buff_q *pool, struct sdpc_buff *buff) +void sdp_buff_q_put_head(struct sdpc_buff_q *pool, struct sdpc_buff *buff) { - return do_buff_q_put(pool, buff, 1); + do_buff_q_put(pool, buff, 1); } /* * sdp_buff_q_put_tail - Place a buffer into the tail of a specific pool */ -int sdp_buff_q_put_tail(struct sdpc_buff_q *pool, struct sdpc_buff *buff) +void sdp_buff_q_put_tail(struct sdpc_buff_q *pool, struct sdpc_buff *buff) { - return do_buff_q_put(pool, buff, 0); + do_buff_q_put(pool, buff, 0); } /* @@ -316,7 +286,6 @@ void sdp_buff_q_clear_unmap(struct sdpc_buff_q *pool, struct device *dev, int direction) { - int result; struct sdpc_buff *buff; while ((buff = do_buff_q_get(pool, 0, NULL, NULL))) { @@ -324,10 +293,7 @@ dma_unmap_single(dev, buff->sge.addr, buff->tail - buff->data, direction); - result = sdp_buff_pool_put(buff); - if (result < 0) - sdp_dbg_err("Error <%d> returning buffer to main", - result); + sdp_buff_pool_put(buff); } } @@ -389,8 +355,6 @@ { struct sdpc_buff *buff; int total; - int result; - /* * Calculate the total number of buffers. */ @@ -428,14 +392,7 @@ buff->type = SDP_DESC_TYPE_BUFF; buff->release = sdp_buff_pool_put; - result = sdp_buff_q_put(&m_pool->pool, buff); - if (result < 0) { - sdp_warn("Failed to queue buffer. <%d>", result); - - free_page((unsigned long)buff->head); - kmem_cache_free(m_pool->buff_cache, buff); - break; - } + sdp_buff_q_put(&m_pool->pool, buff); m_pool->buff_cur++; } @@ -630,15 +587,15 @@ /* * sdp_buff_pool_put - Return a buffer to the main buffer pool */ -int sdp_buff_pool_put(struct sdpc_buff *buff) +void sdp_buff_pool_put(struct sdpc_buff *buff) { unsigned long flags; - if (!buff || buff->pool) - return -EINVAL; + if (!buff) + return; - if (buff->next || buff->prev) - return -ETOOMANYREFS; + BUG_ON(buff->pool); + BUG_ON(buff->next || buff->prev); /* * reset pointers */ @@ -665,8 +622,6 @@ sdp_buff_pool_release_check(main_pool); spin_unlock_irqrestore(&main_pool->lock, flags); - - return 0; } /* @@ -693,7 +648,7 @@ /* * sdp_buff_pool_chain_put - Return a buffer to the main buffer pool */ -int sdp_buff_pool_chain_put(struct sdpc_buff *buff, u32 count) +void sdp_buff_pool_chain_put(struct sdpc_buff *buff, u32 count) { unsigned long flags; struct sdpc_buff *next; @@ -705,7 +660,7 @@ * returned. (e.g. send completions, recv to userspace. */ if (!buff || count <= 0) - return -EINVAL; + return; spin_lock_irqsave(&main_pool->lock, flags); @@ -727,8 +682,6 @@ sdp_buff_pool_release_check(main_pool); spin_unlock_irqrestore(&main_pool->lock, flags); - - return 0; } /* Index: infiniband/ulp/sdp/sdp_queue.c =================================================================== --- infiniband/ulp/sdp/sdp_queue.c (revision 2588) +++ infiniband/ulp/sdp/sdp_queue.c (working copy) @@ -76,14 +76,13 @@ /* * sdp_desc_q_put - Place an element into a specific table */ -static inline int sdp_desc_q_put(struct sdpc_desc_q *table, +static inline void sdp_desc_q_put(struct sdpc_desc_q *table, struct sdpc_desc *element, int fifo) { /* * fifo: false == tail, true == head */ - if (element->table) - return -EINVAL; + BUG_ON(element->table); if (!table->head) { element->next = element; @@ -104,8 +103,6 @@ table->count[element->type] += ((SDP_DESC_TYPE_NONE > element->type) ? 1 : 0); element->table = table; - - return 0; } /* @@ -181,17 +178,17 @@ /* * sdp_desc_q_put_head - Place an element into the head of a table */ -int sdp_desc_q_put_head(struct sdpc_desc_q *table, struct sdpc_desc *element) +void sdp_desc_q_put_head(struct sdpc_desc_q *table, struct sdpc_desc *element) { - return sdp_desc_q_put(table, element, 1); + sdp_desc_q_put(table, element, 1); } /* * sdp_desc_q_put_tail - Place an element into the tail of a table */ -int sdp_desc_q_put_tail(struct sdpc_desc_q *table, struct sdpc_desc *element) +void sdp_desc_q_put_tail(struct sdpc_desc_q *table, struct sdpc_desc *element) { - return sdp_desc_q_put(table, element, 0); + sdp_desc_q_put(table, element, 0); } /* @@ -264,14 +261,10 @@ void sdp_desc_q_clear(struct sdpc_desc_q *table) { struct sdpc_desc *element; - int result; - /* * drain the table of any objects */ while ((element = sdp_desc_q_get_head(table))) - if (element->release) { - result = element->release(element); - SDP_EXPECT(result >= 0); - } + if (element->release) + element->release(element); } Index: infiniband/ulp/sdp/sdp_buff.h =================================================================== --- infiniband/ulp/sdp/sdp_buff.h (revision 2588) +++ infiniband/ulp/sdp/sdp_buff.h (working copy) @@ -49,7 +49,7 @@ struct sdpc_buff *prev; u32 type; /* element type. (for generic queue) */ struct sdpc_buff_q *pool; /* pool currently holding this buffer. */ - int (*release)(struct sdpc_buff *buff); /* release the object */ + void (*release)(struct sdpc_buff *buff); /* release the object */ /* * primary generic data pointers */ From mst at mellanox.co.il Tue Jun 14 14:34:29 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 00:34:29 +0300 Subject: [openib-general] Re: [PATCH] SDP: update TODO In-Reply-To: <1118771978.22484.10.camel@duffman> References: <1118771978.22484.10.camel@duffman> Message-ID: <20050614213429.GA10921@mellanox.co.il> Great, I've updated it a bit. Should there be one for mthca? Thanks, -- MST From mst at mellanox.co.il Tue Jun 14 14:35:58 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 00:35:58 +0300 Subject: [openib-general] Re: [PATCH] SDP: update TODO In-Reply-To: <20050614213429.GA10921@mellanox.co.il> References: <1118771978.22484.10.camel@duffman> <20050614213429.GA10921@mellanox.co.il> Message-ID: <20050614213558.GB10921@mellanox.co.il> Quoting r. Michael S. Tsirkin : > Subject: Re: [PATCH] SDP: update TODO > > Great, I've updated it a bit. > Should there be one for mthca? And IPoIB. Roland, OK with you? -- MST From jlentini at netapp.com Tue Jun 14 14:38:03 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 17:38:03 -0400 (EDT) Subject: [openib-general] Re: kdapltest server on x86_64 In-Reply-To: <1118770067.4748.8.camel@localhost.localdomain> References: <1118770067.4748.8.camel@localhost.localdomain> Message-ID: Hal, Can you retest with revision 2602? I've added some debug prints that should tell us if the IA handle being passed to dapl_psp_create is really invalid. james On Tue, 14 Jun 2005, Hal Rosenstock wrote: > Hi, > > With the latest changes (can't tell when this broke), I get the > following error on x86_64 when I bring up the server: > > DT_cs_Server: dat_psp_create error: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_IA > > Server_Cmd.debug: 1 > Server_Cmd.dapl_name: mthca0a > DT_cs_Server: IA mthca0a opened > DT_cs_Server: PZ created > DT_cs_Server: EP created > DT_cs_Server: dat_psp_create error: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_IA > DT_cs_Server: Waiting for clients to all go away... > DT_cs_Server: Cleaning up ... > DT_cs_Server: dat_ep_disconnect fails: DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED > DT_cs_Server: IA mthca0a closed > DT_cs_Server (mthca0a): Exiting. > TEST INSTANCE 0 > TEST return code = 1 > > This works on x86. The client can be run on either x86 or x86_64. > > With debug on the dat and ib_dat_provider modules, the following is seen: > > DAT: dat_ia_openv(name:mthca0a, async_qlen:8, ver:1.2, thrd_safe:0) > kDAPL: dapl_ia_open (mthca0a, 8, ffff810018aa7dc0, ffff810018aa7d90) > kDAPL: dapl_ia_open () returns 0x0 > kDAPL: dapl_pz_create(ffff81001a714480, ffff810018aa7d98) > kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 20, ffff810018aa7da0) > kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, A0, ffff810018aa7da8) > kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 10, ffff810018aa7db8) > kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 40, ffff810018aa7db0) > kDAPL: dapl_ep_create (ffff81001a714480, ffff81001f098780, ffff81001776b0c0, ffff81001a714980, ffff81001a714d80, 0000000000000000, ffff810018aa7dd0) > kDAPL: created qp_hndl = ffff81001ba61c00, qpn = 10406 > DAT: dat_strerror(major:ffff81003aa0de38, minor:ffff81003aa0de40) called > kDAPL: dapl_ep_disconnect(ffff81001796cc00, 0) > kDAPL: dapl_ep_disconnect () returns 0x8007002e > DAT: dat_strerror(major:ffff81003aa0de38, minor:ffff81003aa0de40) called > kDAPL: dapl_ep_free (ffff81001796cc00) > kDAPL: dapl_ep_disconnect(ffff81001796cc00, 0) > kDAPL: dapl_ep_disconnect () returns 0x8007002e > kDAPL: dapl_ep_free: state 0, ep ffff81001796cc00 qp_state 1 qp 1ba61c00 > kDAPL: dapl_evd_free (ffff81001a714d80) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001a714b80) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001a714980) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001776b0c0) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_pz_free(ffff81001f098780) > DAT: dat_ia_close(ia:1a714480, flags:1) > kDAPL: dapl_ia_query (ffff81001a714480, 0000000000000000, 0000000000000000, ffff81003aa0dc28) > kDAPL: dapl_ia_query () returns 0x0 > kDAPL: dapl_ia_close (ffff81001a714480, 1) > kDAPL: dapl_evd_free (ffff81001a714880) > kDAPL: dapl_evd_free () returns 0x0 > > -- Hal > From jlentini at netapp.com Tue Jun 14 14:38:48 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 17:38:48 -0400 (EDT) Subject: [openib-general] Re: kdapltest server on x86_64 In-Reply-To: <1118770067.4748.8.camel@localhost.localdomain> References: <1118770067.4748.8.camel@localhost.localdomain> Message-ID: Tom, Are you still using x86_64 systems? Have you seen this? james On Tue, 14 Jun 2005, Hal Rosenstock wrote: > Hi, > > With the latest changes (can't tell when this broke), I get the > following error on x86_64 when I bring up the server: > > DT_cs_Server: dat_psp_create error: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_IA > > Server_Cmd.debug: 1 > Server_Cmd.dapl_name: mthca0a > DT_cs_Server: IA mthca0a opened > DT_cs_Server: PZ created > DT_cs_Server: EP created > DT_cs_Server: dat_psp_create error: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_IA > DT_cs_Server: Waiting for clients to all go away... > DT_cs_Server: Cleaning up ... > DT_cs_Server: dat_ep_disconnect fails: DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED > DT_cs_Server: IA mthca0a closed > DT_cs_Server (mthca0a): Exiting. > TEST INSTANCE 0 > TEST return code = 1 > > This works on x86. The client can be run on either x86 or x86_64. > > With debug on the dat and ib_dat_provider modules, the following is seen: > > DAT: dat_ia_openv(name:mthca0a, async_qlen:8, ver:1.2, thrd_safe:0) > kDAPL: dapl_ia_open (mthca0a, 8, ffff810018aa7dc0, ffff810018aa7d90) > kDAPL: dapl_ia_open () returns 0x0 > kDAPL: dapl_pz_create(ffff81001a714480, ffff810018aa7d98) > kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 20, ffff810018aa7da0) > kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, A0, ffff810018aa7da8) > kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 10, ffff810018aa7db8) > kDAPL: dapl_evd_kcreate(ffff81001a714480, 8, 1, ffff81003aa0de08, 40, ffff810018aa7db0) > kDAPL: dapl_ep_create (ffff81001a714480, ffff81001f098780, ffff81001776b0c0, ffff81001a714980, ffff81001a714d80, 0000000000000000, ffff810018aa7dd0) > kDAPL: created qp_hndl = ffff81001ba61c00, qpn = 10406 > DAT: dat_strerror(major:ffff81003aa0de38, minor:ffff81003aa0de40) called > kDAPL: dapl_ep_disconnect(ffff81001796cc00, 0) > kDAPL: dapl_ep_disconnect () returns 0x8007002e > DAT: dat_strerror(major:ffff81003aa0de38, minor:ffff81003aa0de40) called > kDAPL: dapl_ep_free (ffff81001796cc00) > kDAPL: dapl_ep_disconnect(ffff81001796cc00, 0) > kDAPL: dapl_ep_disconnect () returns 0x8007002e > kDAPL: dapl_ep_free: state 0, ep ffff81001796cc00 qp_state 1 qp 1ba61c00 > kDAPL: dapl_evd_free (ffff81001a714d80) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001a714b80) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001a714980) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001776b0c0) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_pz_free(ffff81001f098780) > DAT: dat_ia_close(ia:1a714480, flags:1) > kDAPL: dapl_ia_query (ffff81001a714480, 0000000000000000, 0000000000000000, ffff81003aa0dc28) > kDAPL: dapl_ia_query () returns 0x0 > kDAPL: dapl_ia_close (ffff81001a714480, 1) > kDAPL: dapl_evd_free (ffff81001a714880) > kDAPL: dapl_evd_free () returns 0x0 > > -- Hal > From bjordan.ics at gmail.com Tue Jun 14 14:38:53 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Tue, 14 Jun 2005 17:38:53 -0400 Subject: [openib-general] How do you use IB_CM_ASSIGN_SERVICE_ID? Message-ID: <78d18e205061414386b5797a4@mail.gmail.com> If you specify a service id of IB_CM_ASSIGN_SERVICE_ID, how do you determine the service ID assigned by the SM? In the kernel, I suppose you could just dereference the ib_cm_id struct, but what about from the userspace cm? -- Bill Jordan SilverStorm Technologies From halr at voltaire.com Tue Jun 14 14:39:00 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 17:39:00 -0400 Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118778050.17668.62.camel@localhost> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> <1118761342.12164.0.camel@duffman> <1118778050.17668.62.camel@localhost> Message-ID: <1118784969.4748.88.camel@localhost.localdomain> Hi Josh, On Tue, 2005-06-14 at 15:40, Josh England wrote: > Its probably just me being dumb again...what should I check? I saw the same thing on x86_64 and also reported it. This looks like it might be the same arch. -- Hal From tduffy at sun.com Tue Jun 14 14:42:05 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 14:42:05 -0700 Subject: [openib-general] Re: kdapltest server on x86_64 In-Reply-To: References: <1118770067.4748.8.camel@localhost.localdomain> Message-ID: <1118785325.4985.14.camel@duffman> On Tue, 2005-06-14 at 17:38 -0400, James Lentini wrote: > Tom, > > Are you still using x86_64 systems? Yes. > Have you seen this? No, I haven't. I will try to reproduce. -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Tue Jun 14 14:45:12 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 17:45:12 -0400 Subject: [openib-general] Re: kdapltest server on x86_64 In-Reply-To: References: <1118770067.4748.8.camel@localhost.localdomain> Message-ID: <1118785511.4748.105.camel@localhost.localdomain> On Tue, 2005-06-14 at 17:38, James Lentini wrote: > Can you retest with revision 2602? I've added some debug prints that > should tell us if the IA handle being passed to dapl_psp_create is > really invalid. DAT: dat_ia_openv(name:mthca0a, async_qlen:8, ver:1.2, thrd_safe:0) kDAPL: dapl_ia_open (mthca0a, 8, ffff81003f3b99c0, ffff81003f3b9990) kDAPL: dapl_ia_open () returns 0x0 kDAPL: dapl_pz_create(ffff81003cac0080, ffff81003f3b9998) kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 20, ffff81003f3b99a0) kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, A0, ffff81003f3b99a8) kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 10, ffff81003f3b99b8) kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 40, ffff81003f3b99b0) kDAPL: dapl_ep_create (ffff81003cac0080, ffff81003f3b9800, ffff81001f6a6080, ffff81001776bac0, ffff81001776b5c0, 0000000000000000, ffff81003f3b99d0) kDAPL: created qp_hndl = ffff81001f6e1800, qpn = 20406 kDAPL: dapl_psp_create(ffff81003cac0080, B0DE, ffff81001776b9c0, 0, ffff81003f3b99d8) DAT: dat_strerror(major:ffff810012ee9e38, minor:ffff810012ee9e40) called kDAPL: dapl_ep_disconnect(ffff810028a49c00, 0) kDAPL: dapl_ep_disconnect () returns 0x8007002e DAT: dat_strerror(major:ffff810012ee9e38, minor:ffff810012ee9e40) called kDAPL: dapl_ep_free (ffff810028a49c00) kDAPL: dapl_ep_disconnect(ffff810028a49c00, 0) kDAPL: dapl_ep_disconnect () returns 0x8007002e kDAPL: dapl_ep_free: state 0, ep ffff810028a49c00 qp_state 1 qp 1f6e1800 kDAPL: dapl_evd_free (ffff81001776b5c0) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001776b9c0) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001776bac0) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001f6a6080) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_pz_free(ffff81003f3b9800) DAT: dat_ia_close(ia:3cac0080, flags:1) kDAPL: dapl_ia_query (ffff81003cac0080, 0000000000000000, 0000000000000000, ffff810012ee9c28) kDAPL: dapl_ia_query () returns 0x0 kDAPL: dapl_ia_close (ffff81003cac0080, 1) kDAPL: dapl_evd_free (ffff81001f6a6380) kDAPL: dapl_evd_free () returns 0x0 From tduffy at sun.com Tue Jun 14 14:49:54 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 14:49:54 -0700 Subject: [openib-general] [PATCH 8/9] kDAPL: convert the hcas ia list to linux native In-Reply-To: References: <11187082863011@sun.com> <11187082862310@sun.com> <11187082871393@sun.com> <11187082871245@sun.com> <1118708288376@sun.com> <1118708288856@sun.com> <11187082893179@sun.com> <1118708290446@sun.com> <11187082903037@sun.com> <1118780806.4985.7.camel@duffman> Message-ID: <1118785794.4985.18.camel@duffman> On Tue, 2005-06-14 at 16:41 -0400, James Lentini wrote: > Then I have an unnecessary INIT_LIST_HEAD on line 236 of > dat/dictionary.c. Shouldn't hurt anything, but it is unneeded. All INIT_LIST_HEAD() does is make the list pointer point to themselves. -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Tue Jun 14 14:52:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 17:52:02 -0400 Subject: [openib-general] How do you use IB_CM_ASSIGN_SERVICE_ID? In-Reply-To: <78d18e205061414386b5797a4@mail.gmail.com> References: <78d18e205061414386b5797a4@mail.gmail.com> Message-ID: <1118785922.4748.119.camel@localhost.localdomain> On Tue, 2005-06-14 at 17:38, William Jordan wrote: > If you specify a service id of IB_CM_ASSIGN_SERVICE_ID, how do you > determine the service ID assigned by the SM? ^^ CM > In the kernel, I suppose > you could just dereference the ib_cm_id struct, but what about from > the userspace cm? Isn't this accessible via ib_cm_attr_id in the ib_cm_attr_param structure ? -- Hal From roland at topspin.com Tue Jun 14 14:59:15 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 14 Jun 2005 14:59:15 -0700 Subject: [openib-general] UC support committed to svn In-Reply-To: <1118782490.4748.21.camel@localhost.localdomain> (Hal Rosenstock's message of "14 Jun 2005 16:54:51 -0400") References: <52mzpskd5z.fsf@topspin.com> <1118782490.4748.21.camel@localhost.localdomain> Message-ID: <52oea8ior0.fsf@topspin.com> Hal> Just one minor question: On SQD->SQD for UC, should Hal> IB_QP_RNR_RETRY be included as optional ? Nope, good catch. I committed the fix. - R. From roland at topspin.com Tue Jun 14 15:01:06 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 14 Jun 2005 15:01:06 -0700 Subject: [openib-general] Re: [PATCH] SDP: update TODO In-Reply-To: <20050614213558.GB10921@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 15 Jun 2005 00:35:58 +0300") References: <1118771978.22484.10.camel@duffman> <20050614213429.GA10921@mellanox.co.il> <20050614213558.GB10921@mellanox.co.il> Message-ID: <52is0gionx.fsf@topspin.com> Michael> Great, I've updated it a bit. Should there be one for mthca? Michael> And IPoIB. Michael> Roland, OK with you? Absolutely, it makes me very happy to see people using the wiki. - R. From halr at voltaire.com Tue Jun 14 15:00:25 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 18:00:25 -0400 Subject: [openib-general] UC support committed to svn In-Reply-To: <52mzpskd5z.fsf@topspin.com> References: <52mzpskd5z.fsf@topspin.com> Message-ID: <1118786424.4748.137.camel@localhost.localdomain> On Tue, 2005-06-14 at 14:26, Roland Dreier wrote: > I just committed UC support to userspace/libmthca and the kernel mthca > driver (see mthca diff below). If I hack pingpong.c, this works for > me but could use some testing in a real app. Also, I would appreciate > it if someone could check the attribute for modify QP that I added to > the table in mthca. Also, could you checkin uc_pingpong as another example ? -- Hal From jjengla at sandia.gov Tue Jun 14 15:04:43 2005 From: jjengla at sandia.gov (Josh England) Date: Tue, 14 Jun 2005 15:04:43 -0700 Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118784969.4748.88.camel@localhost.localdomain> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> <1118761342.12164.0.camel@duffman> <1118778050.17668.62.camel@localhost> <1118784969.4748.88.camel@localhost.localdomain> Message-ID: <1118786683.17672.78.camel@localhost> On Tue, 2005-06-14 at 17:39 -0400, Hal Rosenstock wrote: > Hi Josh, > > On Tue, 2005-06-14 at 15:40, Josh England wrote: > > Its probably just me being dumb again...what should I check? > > I saw the same thing on x86_64 and also reported it. This looks like it > might be the same arch. It is...I'll try to reproduce me original problem on x86. -JE From halr at voltaire.com Tue Jun 14 15:17:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 18:17:41 -0400 Subject: [openib-general] ucm nit Message-ID: <1118787460.4748.159.camel@localhost.localdomain> Hi Libor, In libibcm/src/cm.c::ib_cm_event_get_timed, it looks like the event parameter should be validated: int ib_cm_event_get_timed(int timeout_ms, struct ib_cm_event **event) { if (!event) return -EINVAL; ... *event = NULL; -- Hal From hch at lst.de Tue Jun 14 15:24:33 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 15 Jun 2005 00:24:33 +0200 Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flags In-Reply-To: References: <20050611093144.GA18469@lst.de> Message-ID: <20050614222433.GA20042@lst.de> On Tue, Jun 14, 2005 at 03:16:23PM -0400, James Lentini wrote: > > Christoph, > > I'm all ready to submit this minus the Makefile change. Can you > re-send with a "Signed-off-by" line? (no need to make any changes) Signed-off-by: Christoph Hellwig Index: test/dapltest/test/dapl_test_util.c =================================================================== --- test/dapltest/test/dapl_test_util.c (revision 2588) +++ test/dapltest/test/dapl_test_util.c (working copy) @@ -76,12 +76,10 @@ struct sockaddr_in *ip_addr; DT_Tdep_PT_Printf (phead, "***** DAPL Characteristics *****\n"); - DT_Tdep_PT_Printf (phead, "Provider: %s Version %d.%d DAPL %d.%d\n", + DT_Tdep_PT_Printf (phead, "Provider: %s Version %d.%d\n", pt_ptr->provider_attr.provider_name, pt_ptr->provider_attr.provider_version_major, - pt_ptr->provider_attr.provider_version_minor, - pt_ptr->provider_attr.dat_version_major, - pt_ptr->provider_attr.dat_version_minor ); + pt_ptr->provider_attr.provider_version_minor); DT_Tdep_PT_Printf (phead, "Adapter: %s by %s Version %d.%d\n", pt_ptr->ia_attr.adapter_name, pt_ptr->ia_attr.vendor_name, Index: dat-provider/dapl_ia.c =================================================================== --- dat-provider/dapl_ia.c (revision 2588) +++ dat-provider/dapl_ia.c (working copy) @@ -881,15 +881,12 @@ DAT_NAME_MAX_LENGTH); provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; - provider_attr->dat_version_major = DAT_VERSION_MAJOR; - provider_attr->dat_version_minor = DAT_VERSION_MINOR; provider_attr->lmr_mem_types_supported = DAT_MEM_TYPE_VIRTUAL | DAT_MEM_TYPE_LMR; provider_attr->iov_ownership_on_return = DAT_IOV_CONSUMER; provider_attr->dat_qos_supported = DAT_QOS_BEST_EFFORT; provider_attr->completion_flags_supported = DAT_COMPLETION_DEFAULT_FLAG; - provider_attr->is_thread_safe = FALSE; provider_attr->max_private_data_size = IB_CM_REQ_PRIVATE_DATA_SIZE; provider_attr->supports_multipath = FALSE; provider_attr->ep_creator = DAT_PSP_CREATES_EP_NEVER; Index: dat-provider/dapl_provider.c =================================================================== --- dat-provider/dapl_provider.c (revision 2588) +++ dat-provider/dapl_provider.c (working copy) @@ -244,9 +244,6 @@ (size_t) (DAT_NAME_MAX_LENGTH - 2)); provider_info->ia_name[i] = 'a' + port - 1; /* IB ports are 1 based */ provider_info->ia_name[i+1] = '\0'; - provider_info->dat_version_major = DAT_VERSION_MAJOR; - provider_info->dat_version_minor = DAT_VERSION_MINOR; - provider_info->is_thread_safe = DAPL_THREADSAFE; } static void dapl_add_port(struct ib_device *device, u8 port) Index: dat/dictionary.c =================================================================== --- dat/dictionary.c (revision 2588) +++ dat/dictionary.c (working copy) @@ -212,10 +212,6 @@ BUG_ON(NULL == new_key); strncpy(new_key->ia_name, old_key->ia_name, DAT_NAME_MAX_LENGTH); - new_key->dat_version_major = old_key->dat_version_major; - new_key->dat_version_minor = old_key->dat_version_minor; - new_key->is_thread_safe = old_key->is_thread_safe; - return DAT_SUCCESS; } @@ -226,10 +222,7 @@ BUG_ON(NULL == b); if ((strlen(a->ia_name) == strlen(b->ia_name)) && - (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name))) - && (a->dat_version_major == b->dat_version_major) - && (a->dat_version_minor == b->dat_version_minor) - && (a->is_thread_safe == b->is_thread_safe)) + (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name)))) return TRUE; else return FALSE; Index: dat/api.c =================================================================== --- dat/api.c (revision 2588) +++ dat/api.c (working copy) @@ -349,9 +349,8 @@ } } -u32 dat_ia_openv(const char *name, int async_event_qlen, - struct dat_evd **async_event_handle, struct dat_ia **ia, - u32 dat_major, u32 dat_minor, boolean_t thread_safety) +u32 dat_ia_open(const char *name, int async_event_qlen, + struct dat_evd **async_event_handle, struct dat_ia **ia) { DAT_IA_OPEN_FUNC ia_open_func; struct dat_provider_info info; @@ -360,8 +359,7 @@ dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "%s(name:%s, async_qlen:%i, ver:%x.%x, thrd_safe:%x)\n", - __func__, name, async_event_qlen, dat_major, dat_minor, - thread_safety); + __func__, name, async_event_qlen); if (DAT_IS_BAD_POINTER(name)) return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); @@ -374,10 +372,6 @@ strncpy(info.ia_name, name, len); info.ia_name[len] = '\0'; - info.dat_version_major = dat_major; - info.dat_version_minor = dat_minor; - info.is_thread_safe = thread_safety; - status = dat_dr_provider_open(&info, &ia_open_func); if (status != DAT_SUCCESS) { dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, @@ -390,7 +384,7 @@ return ia_open_func(name, async_event_qlen, async_event_handle, ia); } -EXPORT_SYMBOL(dat_ia_openv); +EXPORT_SYMBOL(dat_ia_open); u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) { @@ -423,10 +417,6 @@ strncpy(info.ia_name, ia_name, len); info.ia_name[len] = '\0'; - info.dat_version_major = provider_attr.dat_version_major; - info.dat_version_minor = provider_attr.dat_version_minor; - info.is_thread_safe = provider_attr.is_thread_safe; - status = dat_dr_provider_close(&info); if (DAT_SUCCESS != status) dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, @@ -472,10 +462,7 @@ { dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", - __func__, provider_info->ia_name, - provider_info->dat_version_major, - provider_info->dat_version_minor, - provider_info->is_thread_safe); + __func__, provider_info->ia_name); if (DAT_IS_BAD_POINTER(provider)) return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); Index: dat/Makefile =================================================================== --- dat/Makefile (revision 2588) +++ dat/Makefile (working copy) @@ -1,7 +1,6 @@ EXTRA_CFLAGS += \ - -Idrivers/dat \ - -Werror + -I$(obj) obj-$(CONFIG_DAT) += dat.o Index: dat/dat.h =================================================================== --- dat/dat.h (revision 2588) +++ dat/dat.h (working copy) @@ -33,10 +33,6 @@ #include #include -#define DAT_VERSION_MAJOR 1 -#define DAT_VERSION_MINOR 2 -#define DAT_THREADSAFE FALSE - /* * All return codes are actually a 3-way tuple: * @@ -894,9 +890,6 @@ struct dat_provider_info { char ia_name[DAT_NAME_MAX_LENGTH]; - u32 dat_version_major; - u32 dat_version_minor; - boolean_t is_thread_safe; }; struct dat_evd_param { @@ -1017,13 +1010,10 @@ char provider_name[DAT_NAME_MAX_LENGTH]; u32 provider_version_major; u32 provider_version_minor; - u32 dat_version_major; - u32 dat_version_minor; enum dat_mem_type lmr_mem_types_supported; enum dat_iov_ownership iov_ownership_on_return; enum dat_qos dat_qos_supported; enum dat_completion_flags completion_flags_supported; - boolean_t is_thread_safe; int max_private_data_size; boolean_t supports_multipath; enum dat_ep_creator_for_psp ep_creator; @@ -1361,16 +1351,10 @@ /* * DAT registry functions for consumers */ -extern u32 dat_ia_openv(const char *name, int async_event_qlen, +extern u32 dat_ia_open(const char *name, int async_event_qlen, struct dat_evd **async_event_handle, - struct dat_ia **ia, u32 dat_major, u32 dat_minor, - boolean_t thread_safety); + struct dat_ia **ia); -#define dat_ia_open(name, qlen, async_evd, ia) \ - dat_ia_openv((name), (qlen), (async_evd), (ia), \ - DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ - DAT_THREADSAFE) - extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); extern u32 dat_strerror(u32, const char **, const char **); From hch at lst.de Tue Jun 14 15:29:10 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 15 Jun 2005 00:29:10 +0200 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> <20050614171122.GC15215@lst.de> Message-ID: <20050614222910.GB20042@lst.de> On Tue, Jun 14, 2005 at 05:20:39PM -0400, James Lentini wrote: > >Purely a style issue. > > Is it a personal style issue or a Linux kernel style issue? It's a personal interpretation of the Linux style rule "don't make things complicated if they can be done simple and less confusing". Look at the structure, there's: struct dapl_ia *owner_ia; this insn't actually usefull in all users of dapl_common, and it's easy enough to use without it. No polymorphism argument either because you could just pass ->owner_ia to whatever function expects it. spinlock_t lock; now in every subsystem I looked at putting a lock into every single object was a mistake. You end up with a something as slow and undebuggable as Solaris if you do that. Note sure if it's true for the current kDAPL codebase, but in the end there shouldn't be a lock needed in every structure. unsigned long flags; /* saved lock flag values */ now this one is completely wrong. the irqflags must always be restored in the same function they're used in, and the canoncical way to do that is to put them onto the stack, as everyone in the kernel does. putting them in some object on the heap is a total wate of memory and encourages using the locks at different levels which is wrong. From hch at lst.de Tue Jun 14 15:32:01 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 15 Jun 2005 00:32:01 +0200 Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query() In-Reply-To: <1118271274.9977.41.camel@duffman> References: <1118271274.9977.41.camel@duffman> Message-ID: <20050614223201.GA20186@lst.de> On Wed, Jun 08, 2005 at 03:54:34PM -0700, Tom Duffy wrote: > I am putting this out there to see what people think about this sort of > change. This only does it for one of the functions in dat provider, but > I will change it for all if this is accepted. > > Basically, instead of using a redirected function wrapper, it calls the > function directly from the struct (how the rest of the kernel does it). Please don't do this yet, there's lots of functionality currently at the provider level that should move into the common code, like management of the various lists or the connection state. If there's entry points left where there's no work needed in the common code in the end you can still remove the then useless wrappers. From tduffy at sun.com Tue Jun 14 15:32:18 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 15:32:18 -0700 Subject: [openib-general] kdapltest server on x86_64 In-Reply-To: <1118770067.4748.8.camel@localhost.localdomain> References: <1118770067.4748.8.camel@localhost.localdomain> Message-ID: <1118788338.4985.20.camel@duffman> On Tue, 2005-06-14 at 13:27 -0400, Hal Rosenstock wrote: > Hi, > > With the latest changes (can't tell when this broke), I get the > following error on x86_64 when I bring up the server: This should fix it. Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_sp.c =================================================================== --- linux-kernel/dat-provider/dapl_sp.c (revision 2604) +++ linux-kernel/dat-provider/dapl_sp.c (working copy) @@ -509,7 +509,7 @@ u32 dapl_psp_create(struct dat_ia *dat_i dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_psp_create(%p, %X, %p, %X, %p)\n", dat_ia, conn_qual, dat_evd, psp_flags, dat_psp); - if (!ia) { + if (!dat_ia) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } From jjengla at sandia.gov Tue Jun 14 15:36:23 2005 From: jjengla at sandia.gov (Josh England) Date: Tue, 14 Jun 2005 15:36:23 -0700 Subject: [openib-general] /dev/infiniband/ucm Message-ID: <1118788583.17676.81.camel@localhost> What it the minor number for /dev/infiniband/ucm? -JE From tduffy at sun.com Tue Jun 14 15:38:25 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 15:38:25 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <20050614222910.GB20042@lst.de> References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> <20050614171122.GC15215@lst.de> <20050614222910.GB20042@lst.de> Message-ID: <1118788705.21577.1.camel@duffman> On Wed, 2005-06-15 at 00:29 +0200, Christoph Hellwig wrote: > now in every subsystem I looked at putting a lock into every single > object was a mistake. You end up with a something as slow and > undebuggable as Solaris if you do that. Hey, no low blows. Solaris actually *has* a kernel debugger, thank you very much. -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From hch at lst.de Tue Jun 14 15:40:57 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 15 Jun 2005 00:40:57 +0200 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <1118788705.21577.1.camel@duffman> References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> <20050614171122.GC15215@lst.de> <20050614222910.GB20042@lst.de> <1118788705.21577.1.camel@duffman> Message-ID: <20050614224057.GA20509@lst.de> On Tue, Jun 14, 2005 at 03:38:25PM -0700, Tom Duffy wrote: > On Wed, 2005-06-15 at 00:29 +0200, Christoph Hellwig wrote: > > now in every subsystem I looked at putting a lock into every single > > object was a mistake. You end up with a something as slow and > > undebuggable as Solaris if you do that. > > Hey, no low blows. > > Solaris actually *has* a kernel debugger, thank you very much. I'd rather prefer code that's understandable without a debugger, but that's probably philosophical differences ;-) From halr at voltaire.com Tue Jun 14 15:38:09 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 18:38:09 -0400 Subject: [openib-general] /dev/infiniband/ucm In-Reply-To: <1118788583.17676.81.camel@localhost> References: <1118788583.17676.81.camel@localhost> Message-ID: <1118788689.4748.182.camel@localhost.localdomain> On Tue, 2005-06-14 at 18:36, Josh England wrote: > What it the minor number for /dev/infiniband/ucm? minor is 255 (major 251) -- Hal From jlentini at netapp.com Tue Jun 14 15:42:00 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 18:42:00 -0400 (EDT) Subject: [openib-general] Re: kdapltest server on x86_64 In-Reply-To: <1118785511.4748.105.camel@localhost.localdomain> References: <1118770067.4748.8.camel@localhost.localdomain> <1118785511.4748.105.camel@localhost.localdomain> Message-ID: I assume that kdapltest still reports that dat_psp_create fails due to an invalid IA handle. Based on the trace: On Tue, 14 Jun 2005, Hal Rosenstock wrote: > On Tue, 2005-06-14 at 17:38, James Lentini wrote: >> Can you retest with revision 2602? I've added some debug prints that >> should tell us if the IA handle being passed to dapl_psp_create is >> really invalid. > > DAT: dat_ia_openv(name:mthca0a, async_qlen:8, ver:1.2, thrd_safe:0) > kDAPL: dapl_ia_open (mthca0a, 8, ffff81003f3b99c0, ffff81003f3b9990) > kDAPL: dapl_ia_open () returns 0x0 > kDAPL: dapl_pz_create(ffff81003cac0080, ffff81003f3b9998) > kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 20, ffff81003f3b99a0) > kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, A0, ffff81003f3b99a8) > kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 10, ffff81003f3b99b8) > kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 40, ffff81003f3b99b0) > kDAPL: dapl_ep_create (ffff81003cac0080, ffff81003f3b9800, ffff81001f6a6080, ffff81001776bac0, ffff81001776b5c0, 0000000000000000, ffff81003f3b99d0) > kDAPL: created qp_hndl = ffff81001f6e1800, qpn = 20406 > kDAPL: dapl_psp_create(ffff81003cac0080, B0DE, ffff81001776b9c0, 0, ffff81003f3b99d8) The correct IA is being passed to dapl_psp_create(). Since dapl_psp_create's IA is clearly not null, the test on line 512 of dapl_sp.c will not succeed. There is no other place in the execution of dapl_psp_create where an invalid IA handle error code is returned. dapl_ib_setup_conn_listener() did return error values without using the DAT_ERROR() macro to fill in the subtype. I wouldn't expect this to cause any problems but I've updated the code to use DAT_ERROR() for consistency. Just to make sure that dat_str_error() is correctly parsing the error code, I updated the dat_str_error() debug print. Could you please retesting with revision 2605? > DAT: dat_strerror(major:ffff810012ee9e38, minor:ffff810012ee9e40) called > kDAPL: dapl_ep_disconnect(ffff810028a49c00, 0) > kDAPL: dapl_ep_disconnect () returns 0x8007002e > DAT: dat_strerror(major:ffff810012ee9e38, minor:ffff810012ee9e40) called > kDAPL: dapl_ep_free (ffff810028a49c00) > kDAPL: dapl_ep_disconnect(ffff810028a49c00, 0) > kDAPL: dapl_ep_disconnect () returns 0x8007002e > kDAPL: dapl_ep_free: state 0, ep ffff810028a49c00 qp_state 1 qp 1f6e1800 > kDAPL: dapl_evd_free (ffff81001776b5c0) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001776b9c0) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001776bac0) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81001f6a6080) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_pz_free(ffff81003f3b9800) > DAT: dat_ia_close(ia:3cac0080, flags:1) > kDAPL: dapl_ia_query (ffff81003cac0080, 0000000000000000, 0000000000000000, ffff810012ee9c28) > kDAPL: dapl_ia_query () returns 0x0 > kDAPL: dapl_ia_close (ffff81003cac0080, 1) > kDAPL: dapl_evd_free (ffff81001f6a6380) > kDAPL: dapl_evd_free () returns 0x0 > > From halr at voltaire.com Tue Jun 14 15:49:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 18:49:51 -0400 Subject: [openib-general] kdapltest server on x86_64 In-Reply-To: <1118788338.4985.20.camel@duffman> References: <1118770067.4748.8.camel@localhost.localdomain> <1118788338.4985.20.camel@duffman> Message-ID: <1118789372.4748.199.camel@localhost.localdomain> On Tue, 2005-06-14 at 18:32, Tom Duffy wrote: > On Tue, 2005-06-14 at 13:27 -0400, Hal Rosenstock wrote: > > Hi, > > > > With the latest changes (can't tell when this broke), I get the > > following error on x86_64 when I bring up the server: > > This should fix it. It does. Thanks. -- Hal From halr at voltaire.com Tue Jun 14 15:53:04 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 18:53:04 -0400 Subject: [openib-general] Re: kdapltest server on x86_64 In-Reply-To: References: <1118770067.4748.8.camel@localhost.localdomain> <1118785511.4748.105.camel@localhost.localdomain> Message-ID: <1118789373.4748.201.camel@localhost.localdomain> On Tue, 2005-06-14 at 18:42, James Lentini wrote: > I assume that kdapltest still reports that dat_psp_create fails due to > an invalid IA handle. Yes. (I thought the added debug messages didn't change any behavior). BTW, Tom found it and his fix works. -- Hal > Based on the trace: > > On Tue, 14 Jun 2005, Hal Rosenstock wrote: > > > On Tue, 2005-06-14 at 17:38, James Lentini wrote: > >> Can you retest with revision 2602? I've added some debug prints that > >> should tell us if the IA handle being passed to dapl_psp_create is > >> really invalid. > > > > DAT: dat_ia_openv(name:mthca0a, async_qlen:8, ver:1.2, thrd_safe:0) > > kDAPL: dapl_ia_open (mthca0a, 8, ffff81003f3b99c0, ffff81003f3b9990) > > kDAPL: dapl_ia_open () returns 0x0 > > kDAPL: dapl_pz_create(ffff81003cac0080, ffff81003f3b9998) > > kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 20, ffff81003f3b99a0) > > kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, A0, ffff81003f3b99a8) > > kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 10, ffff81003f3b99b8) > > kDAPL: dapl_evd_kcreate(ffff81003cac0080, 8, 1, ffff810012ee9e08, 40, ffff81003f3b99b0) > > kDAPL: dapl_ep_create (ffff81003cac0080, ffff81003f3b9800, ffff81001f6a6080, ffff81001776bac0, ffff81001776b5c0, 0000000000000000, ffff81003f3b99d0) > > kDAPL: created qp_hndl = ffff81001f6e1800, qpn = 20406 > > kDAPL: dapl_psp_create(ffff81003cac0080, B0DE, ffff81001776b9c0, 0, ffff81003f3b99d8) > > The correct IA is being passed to dapl_psp_create(). Since > dapl_psp_create's IA is clearly not null, the test on line 512 of > dapl_sp.c will not succeed. > > There is no other place in the execution of dapl_psp_create where an > invalid IA handle error code is returned. > dapl_ib_setup_conn_listener() did return error values without using > the DAT_ERROR() macro to fill in the subtype. I wouldn't expect this > to cause any problems but I've updated the code to use DAT_ERROR() for > consistency. > > Just to make sure that dat_str_error() is correctly parsing the error > code, I updated the dat_str_error() debug print. > > Could you please retesting with revision 2605? > > > DAT: dat_strerror(major:ffff810012ee9e38, minor:ffff810012ee9e40) called > > kDAPL: dapl_ep_disconnect(ffff810028a49c00, 0) > > kDAPL: dapl_ep_disconnect () returns 0x8007002e > > DAT: dat_strerror(major:ffff810012ee9e38, minor:ffff810012ee9e40) called > > kDAPL: dapl_ep_free (ffff810028a49c00) > > kDAPL: dapl_ep_disconnect(ffff810028a49c00, 0) > > kDAPL: dapl_ep_disconnect () returns 0x8007002e > > kDAPL: dapl_ep_free: state 0, ep ffff810028a49c00 qp_state 1 qp 1f6e1800 > > kDAPL: dapl_evd_free (ffff81001776b5c0) > > kDAPL: dapl_evd_free () returns 0x0 > > kDAPL: dapl_evd_free (ffff81001776b9c0) > > kDAPL: dapl_evd_free () returns 0x0 > > kDAPL: dapl_evd_free (ffff81001776bac0) > > kDAPL: dapl_evd_free () returns 0x0 > > kDAPL: dapl_evd_free (ffff81001f6a6080) > > kDAPL: dapl_evd_free () returns 0x0 > > kDAPL: dapl_pz_free(ffff81003f3b9800) > > DAT: dat_ia_close(ia:3cac0080, flags:1) > > kDAPL: dapl_ia_query (ffff81003cac0080, 0000000000000000, 0000000000000000, ffff810012ee9c28) > > kDAPL: dapl_ia_query () returns 0x0 > > kDAPL: dapl_ia_close (ffff81003cac0080, 1) > > kDAPL: dapl_evd_free (ffff81001f6a6380) > > kDAPL: dapl_evd_free () returns 0x0 > > > > From jlentini at netapp.com Tue Jun 14 15:57:29 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 18:57:29 -0400 (EDT) Subject: [openib-general] kdapltest server on x86_64 In-Reply-To: <1118788338.4985.20.camel@duffman> References: <1118770067.4748.8.camel@localhost.localdomain> <1118788338.4985.20.camel@duffman> Message-ID: Thanks Tom. Committed in revision 2606. On Tue, 14 Jun 2005, Tom Duffy wrote: > On Tue, 2005-06-14 at 13:27 -0400, Hal Rosenstock wrote: >> Hi, >> >> With the latest changes (can't tell when this broke), I get the >> following error on x86_64 when I bring up the server: > > This should fix it. > > Signed-off-by: Tom Duffy > > Index: linux-kernel/dat-provider/dapl_sp.c > =================================================================== > --- linux-kernel/dat-provider/dapl_sp.c (revision 2604) > +++ linux-kernel/dat-provider/dapl_sp.c (working copy) > @@ -509,7 +509,7 @@ u32 dapl_psp_create(struct dat_ia *dat_i > dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_psp_create(%p, %X, %p, %X, %p)\n", > dat_ia, conn_qual, dat_evd, psp_flags, dat_psp); > > - if (!ia) { > + if (!dat_ia) { > status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); > goto bail; > } > From jjengla at sandia.gov Tue Jun 14 15:58:23 2005 From: jjengla at sandia.gov (Josh England) Date: Tue, 14 Jun 2005 15:58:23 -0700 Subject: [openib-general] /dev/infiniband/ucm In-Reply-To: <1118788689.4748.182.camel@localhost.localdomain> References: <1118788583.17676.81.camel@localhost> <1118788689.4748.182.camel@localhost.localdomain> Message-ID: <1118789903.17670.93.camel@localhost> On Tue, 2005-06-14 at 18:38 -0400, Hal Rosenstock wrote: > On Tue, 2005-06-14 at 18:36, Josh England wrote: > > What it the minor number for /dev/infiniband/ucm? > > minor is 255 (major 251) major is 231 modprobe ib_ucm cat /proc/devices ===> 231 infiniband_cm -JE From tduffy at sun.com Tue Jun 14 15:58:41 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 14 Jun 2005 15:58:41 -0700 Subject: [openib-general] [PATCH] kDAPL: make some stuff static after file consolidation Message-ID: <1118789921.21577.5.camel@duffman> These functions can be declared static now that we have consolidated some of the files. James, this patch does not, but can I delete these two functions (they seem to be unused): dapl_evd_qp_async_error_callback() dapl_evd_cq_async_error_callback() Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_ia.c =================================================================== --- linux-kernel/dat-provider/dapl_ia.c (revision 2605) +++ linux-kernel/dat-provider/dapl_ia.c (working copy) @@ -38,8 +38,8 @@ #include "dapl_sp.h" #include "dapl_cr.h" -struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, - struct dapl_hca *hca) +static struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, + struct dapl_hca *hca) { struct dapl_ia *ia; @@ -75,7 +75,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat return ia; } -u32 dapl_ia_teardown_callbacks(struct dapl_ia *ia) +static u32 dapl_ia_teardown_callbacks(struct dapl_ia *ia) { u32 dat_status = DAT_SUCCESS; @@ -100,7 +100,7 @@ bail: * Performs an abrupt close of the IA */ -u32 dapl_ia_abrupt_close(struct dapl_ia *ia) +static u32 dapl_ia_abrupt_close(struct dapl_ia *ia) { u32 dat_status = DAT_SUCCESS; struct dapl_ep *ep; @@ -271,7 +271,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia * */ -u32 dapl_ia_graceful_close(struct dapl_ia *ia) +static u32 dapl_ia_graceful_close(struct dapl_ia *ia) { u32 dat_status = DAT_SUCCESS; u32 cur_dat_status; @@ -555,8 +555,8 @@ void dapl_ia_link_rsp(struct dapl_ia *ia spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); } -u32 dapl_ia_setup_callbacks(struct dapl_ia *ia, - struct dapl_evd *async_evd) +static u32 dapl_ia_setup_callbacks(struct dapl_ia *ia, + struct dapl_evd *async_evd) { u32 dat_status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2605) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -30,6 +30,7 @@ */ #include "dapl.h" +#include "dapl_evd.h" #include "dapl_cookie.h" #include "dapl_ia.h" #include "dapl_openib_util.h" @@ -142,10 +143,10 @@ bail: /* * alloc and initialize an EVD struct */ -struct dapl_evd *dapl_evd_alloc(struct dapl_ia *ia, int qlen, - enum dat_upcall_policy upcall_policy, - const struct dat_upcall_object *upcall, - enum dat_evd_flags flags) +static struct dapl_evd *dapl_evd_alloc(struct dapl_ia *ia, int qlen, + enum dat_upcall_policy upcall_policy, + const struct dat_upcall_object *upcall, + enum dat_evd_flags flags) { struct dapl_evd *evd; @@ -627,7 +628,8 @@ static u32 dapl_evd_cq_poll_to_event(str return status; } -void dapl_evd_qp_async_error_callback(struct ib_event *cause, void *context) +static void dapl_evd_qp_async_error_callback(struct ib_event *cause, + void *context) { struct dapl_ep *ep; struct dapl_evd *async_evd; @@ -666,7 +668,8 @@ void dapl_evd_qp_async_error_callback(st "dapl_evd_qp_async_error_callback () returns\n"); } -void dapl_evd_cq_async_error_callback(struct ib_event *cause, void *context) +static void dapl_evd_cq_async_error_callback(struct ib_event *cause, + void *context) { struct dapl_evd *async_evd; struct dapl_evd *evd = context; @@ -870,7 +873,7 @@ void dapl_evd_connection_callback(struct } -void dapl_evd_dto_callback(struct ib_cq *cq, void *user_context) +static void dapl_evd_dto_callback(struct ib_cq *cq, void *user_context) { struct dapl_evd *evd; DAPL_EVD_STATE state; Index: linux-kernel/dat-provider/dapl_sp.c =================================================================== --- linux-kernel/dat-provider/dapl_sp.c (revision 2605) +++ linux-kernel/dat-provider/dapl_sp.c (working copy) @@ -36,7 +36,7 @@ #include "dapl_ia.h" #include "dapl_openib_util.h" -struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia, boolean_t is_psp) +static struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia, boolean_t is_psp) { struct dapl_sp *sp; Index: linux-kernel/dat-provider/dapl_provider.c =================================================================== --- linux-kernel/dat-provider/dapl_provider.c (revision 2605) +++ linux-kernel/dat-provider/dapl_provider.c (working copy) @@ -143,7 +143,7 @@ static struct dat_provider g_dapl_provid * * *********************************************************************/ -void dapl_provider_list_destroy(void) +static void dapl_provider_list_destroy(void) { struct list_head *cur_list, *next_list; struct dapl_provider_entry *cur_entry; @@ -156,7 +156,8 @@ void dapl_provider_list_destroy(void) } } -u32 dapl_provider_list_insert(const char *name, struct dat_provider **provider) +static u32 dapl_provider_list_insert(const char *name, + struct dat_provider **provider) { struct dapl_provider_entry *entry; u32 status = DAT_SUCCESS; @@ -210,7 +211,7 @@ error: return status; } -u32 dapl_provider_list_remove(const char *name) +static u32 dapl_provider_list_remove(const char *name) { struct list_head *cur_list, *next_list; struct dapl_provider_entry *cur_entry; From jlentini at netapp.com Tue Jun 14 16:00:20 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 19:00:20 -0400 (EDT) Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118786683.17672.78.camel@localhost> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> <1118761342.12164.0.camel@duffman> <1118778050.17668.62.camel@localhost> <1118784969.4748.88.camel@localhost.localdomain> <1118786683.17672.78.camel@localhost> Message-ID: Josh, Tom Duffy found the problem. Revision 2606 should work on x86_64. james On Tue, 14 Jun 2005, Josh England wrote: > On Tue, 2005-06-14 at 17:39 -0400, Hal Rosenstock wrote: >> Hi Josh, >> >> On Tue, 2005-06-14 at 15:40, Josh England wrote: >>> Its probably just me being dumb again...what should I check? >> >> I saw the same thing on x86_64 and also reported it. This looks like it >> might be the same arch. > > It is...I'll try to reproduce me original problem on x86. > > -JE > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From jlentini at netapp.com Tue Jun 14 16:12:41 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 19:12:41 -0400 (EDT) Subject: [openib-general] [PATCH][RFC] kDAPL: remove dat wrapper function dat_ia_query() In-Reply-To: <20050614223201.GA20186@lst.de> References: <1118271274.9977.41.camel@duffman> <20050614223201.GA20186@lst.de> Message-ID: Tom and I agreed to defer this. See: http://openib.org/pipermail/openib-general/2005-June/007578.html On Wed, 15 Jun 2005, Christoph Hellwig wrote: > On Wed, Jun 08, 2005 at 03:54:34PM -0700, Tom Duffy wrote: >> I am putting this out there to see what people think about this sort of >> change. This only does it for one of the functions in dat provider, but >> I will change it for all if this is accepted. >> >> Basically, instead of using a redirected function wrapper, it calls the >> function directly from the struct (how the rest of the kernel does it). > > Please don't do this yet, there's lots of functionality currently at the > provider level that should move into the common code, like management of > the various lists or the connection state. If there's entry points left > where there's no work needed in the common code in the end you can still > remove the then useless wrappers. > From jlentini at netapp.com Tue Jun 14 16:57:25 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 19:57:25 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: make some stuff static after file consolidation In-Reply-To: <1118789921.21577.5.camel@duffman> References: <1118789921.21577.5.camel@duffman> Message-ID: On Tue, 14 Jun 2005, Tom Duffy wrote: > These functions can be declared static now that we have consolidated > some of the files. > > James, this patch does not, but can I delete these two functions (they > seem to be unused): > > dapl_evd_qp_async_error_callback() This should be passed to ib_create_qp as the async event handler. > dapl_evd_cq_async_error_callback() This should be passed to ib_create_cq as the async event handler. I'll work on a fix for this. From jlentini at netapp.com Tue Jun 14 17:31:28 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 14 Jun 2005 20:31:28 -0400 (EDT) Subject: [openib-general] Re: [patch][kdapl] integrate dapl_mr_util function into dapl_rmr.c In-Reply-To: <20050614124913.GA7940@mellanox.co.il> References: <20050614124913.GA7940@mellanox.co.il> Message-ID: Committed in revision 2608 with a few modifications (I changed the dat_mr_ function names to dat_rmr_ for consistency). On Tue, 14 Jun 2005, Itamar wrote: > > Integrate dapl_mr_util functions into dapl_rmr.c > Delete dapl_mr_util.[h|c] files > Delete define DAPL_ATS in Makefile (not in use) > > Signed-off-by: Itamar Rabenstein > > Index: Makefile > =================================================================== > --- Makefile (revision 2595) > +++ Makefile (working copy) > @@ -9,7 +9,6 @@ > endif > > EXTRA_CFLAGS += \ > - -DDAPL_ATS \ > -Idrivers/infiniband/include \ > -Idrivers/dat > > @@ -26,7 +25,6 @@ > dapl_ia \ > dapl_llist \ > dapl_lmr \ > - dapl_mr_util \ > dapl_provider \ > dapl_pz \ > dapl_ring_buffer_util \ > Index: dapl_rmr.c > =================================================================== > --- dapl_rmr.c (revision 2595) > +++ dapl_rmr.c (working copy) > @@ -32,7 +32,6 @@ > #include "dapl.h" > #include "dapl_ep.h" > #include "dapl_ia.h" > -#include "dapl_mr_util.h" > #include "dapl_hash.h" > #include "dapl_cookie.h" > #include "dapl_openib_util.h" > @@ -92,6 +91,62 @@ > kfree(rmr); > } > > +/* > + * dapl_mr_get_address > + * > + * Returns the memory address associated with the given memory descriptor > + * > + * Input: > + * desc memory descriptor > + * type type of memory represented by desc > + * > + * Output: > + * None > + * > + */ > +u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, enum dat_mem_type type) > +{ > + struct dapl_lmr *lmr; > + > + switch (type) { > + case DAT_MEM_TYPE_VIRTUAL: > + return (u64) (unsigned long) desc.for_va; > + case DAT_MEM_TYPE_LMR: > + lmr = (struct dapl_lmr *)desc.for_lmr; > + > + /* Since this function is recoursive we cannot inline it */ > + return dapl_mr_get_address(lmr->param.region_desc, > + lmr->param.mem_type); > + case DAT_MEM_TYPE_PHYSICAL: > + return desc.for_pa; > + default: > + /* > + * The following kDAPL memory types have not been implemented: > + * DAT_MEM_TYPE_PLATFORM > + * DAT_MEM_TYPE_IA > + * DAT_MEM_TYPE_BYPASS > + */ > + dapl_os_assert(0); > + return 0; > + } > +} > + > +/* > + * dapl_mr_bounds_check > + * > + * Returns true if region B is contained within region A > + * and false otherwise > + * > + */ > +static inline boolean_t dapl_mr_bounds_check(u64 addr_a, u64 length_a, > + u64 addr_b, u64 length_b) > +{ > + if ((addr_a <= addr_b) && (addr_b + length_b) <= (addr_a + length_a)) > + return TRUE; > + else > + return FALSE; > +} > + > static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, > const struct dat_lmr_triplet *lmr_triplet, > enum dat_mem_priv_flags mem_priv, > Index: dapl_mr_util.c > =================================================================== > --- dapl_mr_util.c (revision 2595) > +++ dapl_mr_util.c (working copy) > @@ -1,79 +0,0 @@ > -/* > - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > - * > - * This Software is licensed under one of the following licenses: > - * > - * 1) under the terms of the "Common Public License 1.0" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/cpl.php. > - * > - * 2) under the terms of the "The BSD License" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/bsd-license.php. > - * > - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > - * copy of which is available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/gpl-license.php. > - * > - * Licensee has the right to choose one of the above licenses. > - * > - * Redistributions of source code must retain the above copyright > - * notice and one of the license notices. > - * > - * Redistributions in binary form must reproduce both the above copyright > - * notice, one of the license notices in the documentation > - * and/or other materials provided with the distribution. > - */ > - > -#include "dapl_mr_util.h" > - > -/* > - * $Id$ > - */ > - > -/********************************************************************* > - * * > - * Function Definitions * > - * * > - *********************************************************************/ > - > -/* > - * dapl_mr_get_address > - * > - * Returns the memory address associated with the given memory descriptor > - * > - * Input: > - * desc memory descriptor > - * type type of memory represented by desc > - * > - * Output: > - * None > - * > - */ > - > -u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, enum dat_mem_type type) > -{ > - struct dapl_lmr *lmr; > - > - switch (type) { > - case DAT_MEM_TYPE_VIRTUAL: > - return (u64) (unsigned long) desc.for_va; > - case DAT_MEM_TYPE_LMR: > - lmr = (struct dapl_lmr *)desc.for_lmr; > - > - /* Since this function is recoursive we cannot inline it */ > - return dapl_mr_get_address(lmr->param.region_desc, > - lmr->param.mem_type); > - case DAT_MEM_TYPE_PHYSICAL: > - return desc.for_pa; > - default: > - /* > - * The following kDAPL memory types have not been implemented: > - * DAT_MEM_TYPE_PLATFORM > - * DAT_MEM_TYPE_IA > - * DAT_MEM_TYPE_BYPASS > - */ > - dapl_os_assert(0); > - return 0; > - } > -} > Index: dapl_mr_util.h > =================================================================== > --- dapl_mr_util.h (revision 2595) > +++ dapl_mr_util.h (working copy) > @@ -1,58 +0,0 @@ > -/* > - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > - * > - * This Software is licensed under one of the following licenses: > - * > - * 1) under the terms of the "Common Public License 1.0" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/cpl.php. > - * > - * 2) under the terms of the "The BSD License" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/bsd-license.php. > - * > - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > - * copy of which is available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/gpl-license.php. > - * > - * Licensee has the right to choose one of the above licenses. > - * > - * Redistributions of source code must retain the above copyright > - * notice and one of the license notices. > - * > - * Redistributions in binary form must reproduce both the above copyright > - * notice, one of the license notices in the documentation > - * and/or other materials provided with the distribution. > - */ > - > -/* > - * $Id$ > - */ > - > -#ifndef DAPL_MR_UTIL_H > -#define DAPL_MR_UTIL_H > - > -#include "dapl.h" > -#include "dapl_hash.h" > - > -extern u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, > - enum dat_mem_type type); > - > -/* > - * dapl_mr_bounds_check > - * > - * Returns true if region B is contained within region A > - * and false otherwise > - * > - */ > - > -static inline boolean_t dapl_mr_bounds_check(u64 addr_a, u64 length_a, > - u64 addr_b, u64 length_b) > -{ > - if ((addr_a <= addr_b) && (addr_b + length_b) <= (addr_a + length_a)) > - return TRUE; > - else > - return FALSE; > -} > - > -#endif /* DAPL_MR_UTIL_H */ > Index: dapl_provider.c > =================================================================== > --- dapl_provider.c (revision 2595) > +++ dapl_provider.c (working copy) > @@ -36,7 +36,6 @@ > #include "dapl.h" > #include "dapl_hca_util.h" > #include "dapl_provider.h" > -#include "dapl_mr_util.h" > #include "dapl_util.h" > #include "dapl_openib_util.h" > > -- > Itamar > From jjengla at sandia.gov Tue Jun 14 17:44:32 2005 From: jjengla at sandia.gov (Josh England) Date: Tue, 14 Jun 2005 17:44:32 -0700 Subject: [openib-general] kdaptest wedges server In-Reply-To: References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> <1118761342.12164.0.camel@duffman> <1118778050.17668.62.camel@localhost> <1118784969.4748.88.camel@localhost.localdomain> <1118786683.17672.78.camel@localhost> Message-ID: <1118796272.17668.111.camel@localhost> On Tue, 2005-06-14 at 19:00 -0400, James Lentini wrote: > Josh, > > Tom Duffy found the problem. Revision 2606 should work on x86_64. That works...back to the original problem. I was just running the client too fast and hitting the server before it was ready...it doesn't wedge anymore. On another note, has anyone got a *bare-bones* kdapl/udapl example with connection setup and RDMA send/receive. -JE > james > > On Tue, 14 Jun 2005, Josh England wrote: > > > On Tue, 2005-06-14 at 17:39 -0400, Hal Rosenstock wrote: > >> Hi Josh, > >> > >> On Tue, 2005-06-14 at 15:40, Josh England wrote: > >>> Its probably just me being dumb again...what should I check? > >> > >> I saw the same thing on x86_64 and also reported it. This looks like it > >> might be the same arch. > > > > It is...I'll try to reproduce me original problem on x86. > > > > -JE > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > > From halr at voltaire.com Tue Jun 14 18:08:18 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 14 Jun 2005 21:08:18 -0400 Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118796272.17668.111.camel@localhost> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> <1118761342.12164.0.camel@duffman> <1118778050.17668.62.camel@localhost> <1118784969.4748.88.camel@localhost.localdomain> <1118786683.17672.78.camel@localhost> <1118796272.17668.111.camel@localhost> Message-ID: <1118797697.4748.343.camel@localhost.localdomain> On Tue, 2005-06-14 at 20:44, Josh England wrote: > back to the original problem. I was just running the > client too fast and hitting the server before it was ready... Can you explain what you mean by this ? -- Hal > it doesn't wedge anymore. From caitlin.bestler at gmail.com Tue Jun 14 19:09:16 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 14 Jun 2005 19:09:16 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <20050614222910.GB20042@lst.de> References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> <20050614171122.GC15215@lst.de> <20050614222910.GB20042@lst.de> Message-ID: <469958e005061419095c071a3b@mail.gmail.com> On 6/14/05, Christoph Hellwig wrote: > On Tue, Jun 14, 2005 at 05:20:39PM -0400, James Lentini wrote: > > >Purely a style issue. > > > > Is it a personal style issue or a Linux kernel style issue? > > It's a personal interpretation of the Linux style rule "don't make > things complicated if they can be done simple and less confusing". > These are attributes that are required for all sub-types that comply with a parent type. The original code makes the every so slight mistake of including the IA in that list of types, which has the horrendous defect of having an IA pointer in the IA itself. It would be guaranteed to drive a Java programmer insane, but most C programmers would figure "How many IA objects are there anyway. what's the harm in having one meaningless field in a type that there are only a handful of instances of? I'll declare one fewer type and "waste" the pointer space, I'll probably make it up in code space." Are you implying that implementing the same requirement multiple time as though it were not a common requirement somehow makes the code "simpler"? *ALL* objects created under an IA are destroyed when the IA is destroyed. If that is going to be implemented by a linked list through the header then that field truly is common. *ALL* DAT objects are required to be thread-safe, if the library advertises itself as thread-safe. How is placing the same locking mechanism in each type making the code "simpler"? Having a common subheader makes it easier to have utility code for purposes like lock/unlock. That is why such common headers are *frequently* used to *simplify* code and increases it's readability. It helps the reader know whether they are seeing something unique to this specific type or something that applies to all DAT objects. > Look at the structure, there's: > > struct dapl_ia *owner_ia; > > this insn't actually usefull in all users of dapl_common, and it's easy > enough to use without it. No polymorphism argument either because you > could just pass ->owner_ia to whatever function expects it. > Fetching the hca handle is a step common to almost all verb layer actions taken on all objects. Is there some reason why this information should be fetched differently for each type? Actually the common struct should really have the hca handle itself rather than the ia pointer, since the hca handle can be safely cached and doing so would improve the efficiency of the code by reducing one more unneeded memory fetch. The number of memory fetches is *the* metric of efficiency for RDMA. > spinlock_t lock; > > now in every subsystem I looked at putting a lock into every single > object was a mistake. You end up with a something as slow and > undebuggable as Solaris if you do that. Note sure if it's true for the > current kDAPL codebase, but in the end there shouldn't be a lock needed > in every structure. > There actually is a solution to that in DAT. It allows the DAT Consumer to hold the lock rather than individual DAT objects. It is an option that I would actually expect *most* kernel clients to prefer. That's the meaning of the "thread safety" flag, it documents *who* is responsible for making access to each DAT object thread safe: the DAT Provider or the DAT Consumer. Doing an unnecessary spinlock, particularly in an SMP kernel, is a major destroyer of efficiency. That is why DAT was designed so that the Consumer could load a "thread optimized" library (where it was responsible for serializing access to each object) or a "thread safe" library where the DAT Provider would do so. But even when the latter is needed the impact of a spinlock is lessened when the scope of the lock is no wider than needed. Fastpath operations such as dat_ep_post_send should only lock its own send queue. Locking some other object, like the IA, would cause unneeded blockages. With the lock being on the scale of the EP, 4 different processors can be concurrently posting to four different EPs without interference. If you "safe" that lock by moving it to the IA you'll limit the system to doing one post at a time no matter how many processor are available. > unsigned long flags; /* saved lock flag values */ > > now this one is completely wrong. the irqflags must always be restored > in the same function they're used in, and the canoncical way to do that > is to put them onto the stack, as everyone in the kernel does. putting > them in some object on the heap is a total wate of memory and encourages > using the locks at different levels which is wrong. > ____ That one I agree with. From jjengla at sandia.gov Tue Jun 14 20:57:16 2005 From: jjengla at sandia.gov (Josh England) Date: Tue, 14 Jun 2005 20:57:16 -0700 Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118797697.4748.343.camel@localhost.localdomain> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> <1118761342.12164.0.camel@duffman> <1118778050.17668.62.camel@localhost> <1118784969.4748.88.camel@localhost.localdomain> <1118786683.17672.78.camel@localhost> <1118796272.17668.111.camel@localhost> <1118797697.4748.343.camel@localhost.localdomain> Message-ID: <1118807836.27355.3.camel@localhost> On Tue, 2005-06-14 at 21:08 -0400, Hal Rosenstock wrote: > On Tue, 2005-06-14 at 20:44, Josh England wrote: > > back to the original problem. I was just running the > > client too fast and hitting the server before it was ready... > > Can you explain what you mean by this ? The server pauses for a couple seconds after the transaction is finished. Running the client multiple times in succession, the server shows: Dapltest: Service Point Ready - mthca0a Server: Transaction Test Finished for this client Dapltest: Service Point Ready - mthca0a Server: Transaction Test Finished for this client Dapltest: Service Point Ready - mthca0a Server: Transaction Test Finished for this client Dapltest: Service Point Ready - mthca0a If I try to run the client before the server says 'Service Point Ready', the client hangs. A week or so ago I did this and the server wedged, but it doesn't seem to do that anymore. -JE > -- Hal > > > it doesn't wedge anymore. > > > From hch at lst.de Tue Jun 14 23:50:40 2005 From: hch at lst.de (Christoph Hellwig) Date: Wed, 15 Jun 2005 08:50:40 +0200 Subject: [openib-general] Re: [PATCH] kDAPL: convert the ep list to linux native In-Reply-To: <469958e005061419095c071a3b@mail.gmail.com> References: <1118437034.29601.6.camel@duffman> <20050613211226.GA31086@lst.de> <20050614171122.GC15215@lst.de> <20050614222910.GB20042@lst.de> <469958e005061419095c071a3b@mail.gmail.com> Message-ID: <20050615065040.GA26130@lst.de> On Tue, Jun 14, 2005 at 07:09:16PM -0700, Caitlin Bestler wrote: > These are attributes that are required for all sub-types that comply > with a parent type. The original code makes the every so slight > mistake of including the IA in that list of types, which has the > horrendous defect of having an IA pointer in the IA itself. > > It would be guaranteed to drive a Java programmer insane, > but most C programmers would figure "How many IA objects > are there anyway. what's the harm in having one meaningless > field in a type that there are only a handful of instances of? > I'll declare one fewer type and "waste" the pointer space, > I'll probably make it up in code space." > > Are you implying that implementing the same requirement > multiple time as though it were not a common requirement > somehow makes the code "simpler"? If there actually were common requirements. As Mentioned only a partially-used linked list and a questionable synchronization object are common. Anyway, I think discussing this question here is rather mood because dapl_common will gradually go away anyway, the list should move to the dat layer and the lock for those objects where it makes sense aswell. > There actually is a solution to that in DAT. It allows the DAT Consumer > to hold the lock rather than individual DAT objects. It is an option that > I would actually expect *most* kernel clients to prefer. > > That's the meaning of the "thread safety" flag, it documents *who* > is responsible for making access to each DAT object thread safe: > the DAT Provider or the DAT Consumer. > > Doing an unnecessary spinlock, particularly in an SMP kernel, is > a major destroyer of efficiency. That is why DAT was designed > so that the Consumer could load a "thread optimized" library > (where it was responsible for serializing access to each object) > or a "thread safe" library where the DAT Provider would do so. > > But even when the latter is needed the impact of a spinlock > is lessened when the scope of the lock is no wider than needed. > Fastpath operations such as dat_ep_post_send should only > lock its own send queue. Locking some other object, like the > IA, would cause unneeded blockages. With the lock being > on the scale of the EP, 4 different processors can be concurrently > posting to four different EPs without interference. If you "safe" > that lock by moving it to the IA you'll limit the system to doing > one post at a time no matter how many processor are available. I completely agree on your assertation on locking, it's the direction I was thinging about when saying one lock per object is a bad idea. Once the code is cleaned up a little and more understandable we should design a sane locking scheme, and that will involve a lock in the dat_ia structure at least that can be held over multiple operations. From mst at mellanox.co.il Wed Jun 15 02:44:08 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 12:44:08 +0300 Subject: [openib-general] [PATCH] sdp_post_rdma_iocb_snk check iocb Message-ID: <20050615094408.GW21081@mellanox.co.il> In sdp_post_rdma_iocb_snk, if sdp_iocb_q_get_head returns NULL, iocb is then used without checking. Signed-off-by: Michael S. Tsirkin Index: sdp_recv.c =================================================================== --- sdp_recv.c (revision 2608) +++ sdp_recv.c (working copy) @@ -411,7 +411,7 @@ static int sdp_post_rdma_iocb_snk(struct * queue IOCB */ iocb = sdp_iocb_q_get_head(&conn->r_pend); - if (result < 0) { + if (!iocb) { sdp_dbg_warn(conn, "read IOCB missing from pending table <%d>", sdp_iocb_q_size(&conn->r_pend)); goto release; -- MST From mst at mellanox.co.il Wed Jun 15 02:47:47 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 12:47:47 +0300 Subject: [openib-general] [PATCH] positive error values cleanup Message-ID: <20050615094747.GX21081@mellanox.co.il> sdp_cm_actv_error and sdp_conn_inet_error were accepting positive result as error value, and inverting it internally. Clean this up and do a couple of cosmetic cleanups as well. Signed-off-by: Michael S. Tsirkin Index: sdp_inet.c =================================================================== --- sdp_inet.c (revision 2608) +++ sdp_inet.c (working copy) @@ -278,7 +278,7 @@ error: /* * abortive close. */ - sdp_conn_inet_error(conn, ECONNRESET); + sdp_conn_inet_error(conn, -ECONNRESET); (void)ib_send_cm_dreq(conn->cm_id, NULL, 0); return result; Index: sdp_conn.c =================================================================== --- sdp_conn.c (revision 2608) +++ sdp_conn.c (working copy) @@ -84,21 +84,21 @@ void sdp_conn_inet_error(struct sdp_opt (void)sdp_inet_accept_q_remove(conn); SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); - SDP_CONN_SET_ERR(conn, error); + SDP_CONN_SET_ERR(conn, -error); conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; if (conn->sk->sk_socket) conn->sk->sk_socket->state = SS_UNCONNECTED; - sdp_iocb_q_cancel_all(conn, -error); + sdp_iocb_q_cancel_all(conn, error); sdp_inet_wake_error(conn->sk); } void sdp_conn_abort(struct sdp_opt *conn) { int result; - int error = ECONNRESET; + int error = -ECONNRESET; sdp_dbg_ctrl(conn, "Abort send. src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -133,7 +133,7 @@ void sdp_conn_abort(struct sdp_opt *conn conn->flags &= ~SDP_CONN_F_DIS_PEND; case SDP_CONN_ST_DIS_RECV_1: - error = EPIPE; + error = -EPIPE; case SDP_CONN_ST_ESTABLISHED: /* * abortive close. @@ -150,7 +150,7 @@ void sdp_conn_abort(struct sdp_opt *conn * outstanding CM request. Mark it in error, and CM * completion needs to complete the closing. */ - error = ECONNREFUSED; + error = -ECONNREFUSED; break; case SDP_CONN_ST_ERROR: case SDP_CONN_ST_CLOSED: Index: sdp_actv.c =================================================================== --- sdp_actv.c (revision 2608) +++ sdp_actv.c (working copy) @@ -41,8 +41,6 @@ void sdp_cm_actv_error(struct sdp_opt *c { int result; /* - * error value is positive error. - * * Handle errors within active connections stream. * First generate appropriate response, REJ, DREQ or nothing. * Second the socket must be notified of the error. @@ -88,14 +86,14 @@ void sdp_cm_actv_error(struct sdp_opt *c break; } - SDP_CONN_SET_ERR(conn, error); + SDP_CONN_SET_ERR(conn, -error); conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; if (conn->sk->sk_socket) conn->sk->sk_socket->state = SS_UNCONNECTED; - sdp_iocb_q_cancel_all(conn, (0 - error)); + sdp_iocb_q_cancel_all(conn, error); sdp_inet_wake_error(conn->sk); } @@ -319,7 +317,7 @@ int sdp_cm_rep_handler(struct ib_cm_id * return 0; error: - sdp_cm_actv_error(conn, (0 - result)); + sdp_cm_actv_error(conn, result); if (conn->state == SDP_CONN_ST_CLOSED) { conn->cm_id = NULL; @@ -511,7 +509,7 @@ static void sdp_cm_path_complete(u64 id, goto done; failed: - sdp_cm_actv_error(conn, (0 - status)); + sdp_cm_actv_error(conn, status); done: sdp_conn_unlock(conn); sdp_conn_put(conn); /* address resolution reference */ Index: sdp_pass.c =================================================================== --- sdp_pass.c (revision 2608) +++ sdp_pass.c (working copy) @@ -41,7 +41,7 @@ int sdp_cm_pass_establish(struct sdp_opt { struct ib_qp_attr *qp_attr; int attr_mask = 0; - int result = 0; + int result; sdp_dbg_ctrl(conn, "Passive Establish src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -62,27 +62,26 @@ int sdp_cm_pass_establish(struct sdp_opt result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask); if (result) { - sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS", - result); + sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS", result); goto error; } result = ib_modify_qp(conn->qp, qp_attr, attr_mask); if (result) { - sdp_dbg_warn(conn, "Error <%d> modifiying QP to RTS", result); + sdp_dbg_warn(conn, "Error <%d> modifying QP to RTS", result); goto error; } conn->send_buf = SDP_INET_SEND_SIZE; result = sdp_send_flush(conn); - if (0 > result) { + if (result < 0) { sdp_dbg_warn(conn, "Error <%d> flushing sends.", result); goto error; } result = sdp_recv_flush(conn); - if (0 > result) { + if (result < 0) { sdp_dbg_warn(conn, "Error <%d> flushing receives.", result); goto error; } @@ -94,7 +93,7 @@ int sdp_cm_pass_establish(struct sdp_opt error: kfree(qp_attr); done: - sdp_conn_inet_error(conn, -result); + sdp_conn_inet_error(conn, result); return result; } /* Index: sdp_event.c =================================================================== --- sdp_event.c (revision 2608) +++ sdp_event.c (working copy) @@ -184,7 +184,7 @@ void sdp_cq_event_handler(struct ib_cq * static void sdp_cm_to_error(struct sdp_opt *conn) { - sdp_conn_inet_error(conn, ECONNRESET); + sdp_conn_inet_error(conn, -ECONNRESET); conn->cm_id = NULL; sdp_conn_put(conn); /* CM reference */ } @@ -202,11 +202,11 @@ static int sdp_cm_idle(struct ib_cm_id * */ switch (conn->state) { case SDP_CONN_ST_REQ_SENT: - sdp_cm_actv_error(conn, ECONNREFUSED); + sdp_cm_actv_error(conn, -ECONNREFUSED); break; case SDP_CONN_ST_REQ_RECV: case SDP_CONN_ST_ESTABLISHED: - sdp_conn_inet_error(conn, ECONNREFUSED); + sdp_conn_inet_error(conn, -ECONNREFUSED); break; case SDP_CONN_ST_TIME_WAIT_1: sdp_dbg_warn(conn, "Unexpected connection state"); -- MST From mst at mellanox.co.il Wed Jun 15 04:44:39 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 14:44:39 +0300 Subject: [openib-general] Re: How about ib_send_page() ? In-Reply-To: <528y2di70f.fsf@topspin.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <528y2di70f.fsf@topspin.com> Message-ID: <20050615114439.GA20041@mellanox.co.il> Quoting r. Roland Dreier : > > When I profile a system running IPoIB throughput tests, half or more > of the CPU time is going to skb_copy_and_csum() and other parts of the > core kernel's network stack. Hi, Roland! I wonder what did you use for profiling? -- MST From itamar at mellanox.co.il Wed Jun 15 06:34:03 2005 From: itamar at mellanox.co.il (Rabenstein Itamar) Date: Wed, 15 Jun 2005 16:34:03 +0300 Subject: [openib-general] [PATCH][kdapl]update dat_rmr_bind API & delete dapl_hash Message-ID: <20050615133403.GA19837@mellanox.co.il> Updated dat_rmr_bind API (added lmr_handle as input param) delete dapl_hash.[h|c] files (not needed any more) Integrate dapl_hca_util functions in ia/provider code delete dapl_hca_util.[h|c] files Signed-off-by: Itamar Rabenstein Index: test/dapltest/test/dapl_bpool.c =================================================================== --- test/dapltest/test/dapl_bpool.c (revision 2608) +++ test/dapltest/test/dapl_bpool.c (working copy) @@ -236,6 +236,7 @@ bpool_ptr->reg_addr, bpool_ptr->reg_size)); ret = dat_rmr_bind ( bpool_ptr->rmr_handle, + bpool_ptr->lmr, &iov, mflags, bpool_ptr->ep, Index: test/dapltest/kdapl/kdapl_tdep_user.c =================================================================== --- test/dapltest/kdapl/kdapl_tdep_user.c (revision 2608) +++ test/dapltest/kdapl/kdapl_tdep_user.c (working copy) @@ -76,7 +76,7 @@ } if (params_ptr->test_type == TRANSACTION_TEST) { print_ioctl.cookie = ioctl (fd, KDAPL_IOCTL_GET_STAT_T, &Client_Stats_T); - DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_iterations, + DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_threads, params_ptr->u.Transaction_Cmd.eps_per_thread); } Index: dat-provider/dapl_hca_util.c =================================================================== --- dat-provider/dapl_hca_util.c (revision 2608) +++ dat-provider/dapl_hca_util.c (working copy) @@ -1,150 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_openib_util.h" -#include "dapl_provider.h" -#include "dapl_hca_util.h" -#include "dapl_hash.h" - -/* - * dapl_hca_alloc - * - * alloc and initialize an HCA struct - * - * Input: - * name - * port - * - * Output: - * hca - * - * Returns: - * none - * - */ -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) -{ - struct dapl_hca *hca; - - hca = kmalloc(sizeof *hca, GFP_ATOMIC); - if (hca) { - memset(hca, 0, sizeof *hca); - - if (DAT_SUCCESS == - dapl_hash_create(DAPL_HASH_TABLE_DEFAULT_CAPACITY, - &hca->lmr_hash_table)) { - spin_lock_init(&hca->lock); - INIT_LIST_HEAD(&hca->ia_list); - - hca->name = dapl_os_strdup(name); - hca->ib_hca_handle = device; - hca->port_num = port; - if (hca->name == NULL) { - kfree(hca); - hca = NULL; - } - } else { - kfree(hca); - hca = NULL; - } - } - - return hca; -} - -/* - * dapl_hca_free - * - * free an IA INFO struct - * - * Input: - * hca - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_hca_free(struct dapl_hca *hca) -{ - (void)dapl_hash_free(hca->lmr_hash_table); - kfree(hca->name); - kfree(hca); -} - -/* - * dapl_hca_link_ia - * - * Add an ia to the HCA structure - * - * Input: - * hca - * ia_ptr - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia_ptr) -{ - spin_lock_irqsave(&hca->lock, hca->flags); - list_add(&ia_ptr->list, &hca->ia_list); - spin_unlock_irqrestore(&hca->lock, hca->flags); -} - -/* - * dapl_hca_unlink_ia - * - * Remove an ia from the hca info structure - * - * Input: - * hca - * ia_ptr - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia) -{ - spin_lock_irqsave(&hca->lock, hca->flags); - list_del(&ia->list); - spin_unlock_irqrestore(&hca->lock, hca->flags); -} Index: dat-provider/dapl_ia.c =================================================================== --- dat-provider/dapl_ia.c (revision 2608) +++ dat-provider/dapl_ia.c (working copy) @@ -33,7 +33,6 @@ #include "dapl_ia.h" #include "dapl_provider.h" #include "dapl_evd.h" -#include "dapl_hca_util.h" #include "dapl_openib_util.h" #include "dapl_sp.h" #include "dapl_cr.h" @@ -70,8 +69,9 @@ INIT_LIST_HEAD(&ia->psp_list); INIT_LIST_HEAD(&ia->srq_list); - dapl_hca_link_ia(hca, ia); - + spin_lock_irqsave(&hca->lock, hca->flags); + list_add(&ia->list, &hca->ia_list); + spin_unlock_irqrestore(&hca->lock, hca->flags); return ia; } @@ -378,7 +378,9 @@ dapl_os_assert(list_empty(&ia->psp_list)); dapl_os_assert(list_empty(&ia->rsp_list)); - dapl_hca_unlink_ia(ia->hca, ia); + spin_lock_irqsave(&ia->hca->lock, ia->hca->flags); + list_del(&ia->list); + spin_unlock_irqrestore(&ia->hca->lock, ia->hca->flags); /* no need to destroy ia->common.lock */ kfree(ia); Index: dat-provider/dapl_lmr.c =================================================================== --- dat-provider/dapl_lmr.c (revision 2608) +++ dat-provider/dapl_lmr.c (working copy) @@ -31,7 +31,6 @@ #include "dapl_openib_util.h" #include "dapl_ia.h" -#include "dapl_hash.h" static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, @@ -103,17 +102,6 @@ if (DAT_SUCCESS != status) goto error2; - /* if the LMR context is already in the hash table */ - status = dapl_hash_search(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, NULL); - if (status == DAT_SUCCESS) - goto error3; - - status = dapl_hash_insert(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, lmr); - if (status != DAT_SUCCESS) - goto error3; - atomic_inc(&pz->pz_ref_count); if (lmr) @@ -129,8 +117,6 @@ return DAT_SUCCESS; -error3: - (void)dapl_ib_mr_deregister(new_lmr); error2: dapl_lmr_dealloc(new_lmr); error1: @@ -167,17 +153,6 @@ if (DAT_SUCCESS != status) goto error2; - /* if the LMR context is already in the hash table */ - status = dapl_hash_search(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, NULL); - if (status == DAT_SUCCESS) - goto error3; - - status = dapl_hash_insert(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, lmr); - if (status != DAT_SUCCESS) - goto error3; - atomic_inc(&pz->pz_ref_count); if (lmr) @@ -193,8 +168,6 @@ return DAT_SUCCESS; -error3: - (void)dapl_ib_mr_deregister(new_lmr); error2: dapl_lmr_dealloc(new_lmr); error1: @@ -215,12 +188,6 @@ DAT_REGION_DESCRIPTION reg_desc; u32 status; - status = dapl_hash_search(ia->hca->lmr_hash_table, - original_lmr->param.lmr_context, - (DAPL_HASH_DATA *) &lmr); - if (status != DAT_SUCCESS) - goto error1; - reg_desc.for_lmr = (struct dat_lmr *) original_lmr; new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, @@ -235,17 +202,6 @@ if (DAT_SUCCESS != status) goto error2; - /* if the LMR context is already in the hash table */ - status = dapl_hash_search(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, NULL); - if (status == DAT_SUCCESS) - goto error3; - - status = dapl_hash_insert(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, lmr); - if (status != DAT_SUCCESS) - goto error3; - atomic_inc(&pz->pz_ref_count); if (lmr) @@ -262,8 +218,6 @@ return DAT_SUCCESS; -error3: - dapl_ib_mr_deregister(new_lmr); error2: dapl_lmr_dealloc(new_lmr); error1: @@ -371,22 +325,12 @@ if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) return DAT_INVALID_STATE; - status = dapl_hash_remove( - dapl_lmr->common.owner_ia->hca->lmr_hash_table, - dapl_lmr->param.lmr_context, NULL); - if (status != DAT_SUCCESS) - goto error; - status = dapl_ib_mr_deregister(dapl_lmr); if (status == DAT_SUCCESS) { pz = (struct dapl_pz *)dapl_lmr->param.pz; atomic_dec(&pz->pz_ref_count); dapl_lmr_dealloc(dapl_lmr); - } else /* failure; put dapl_lmr back in hash table */ - dapl_hash_insert(dapl_lmr->common.owner_ia-> - hca->lmr_hash_table, - dapl_lmr->param.lmr_context, dapl_lmr); - + } break; } case DAT_MEM_TYPE_PLATFORM: Index: dat-provider/Makefile =================================================================== --- dat-provider/Makefile (revision 2608) +++ dat-provider/Makefile (working copy) @@ -20,8 +20,6 @@ dapl_cr \ dapl_ep \ dapl_evd \ - dapl_hash \ - dapl_hca_util \ dapl_ia \ dapl_lmr \ dapl_provider \ Index: dat-provider/dapl_hca_util.h =================================================================== --- dat-provider/dapl_hca_util.h (revision 2608) +++ dat-provider/dapl_hca_util.h (working copy) @@ -1,45 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_HCA_UTIL_H -#define DAPL_HCA_UTIL_H - -#include "dapl.h" - -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port); - -void dapl_hca_free(struct dapl_hca *hca); - -void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia); - -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia); - -#endif Index: dat-provider/dapl_rmr.c =================================================================== --- dat-provider/dapl_rmr.c (revision 2608) +++ dat-provider/dapl_rmr.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_ep.h" #include "dapl_ia.h" -#include "dapl_hash.h" #include "dapl_cookie.h" #include "dapl_openib_util.h" @@ -148,26 +147,18 @@ } static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, - const struct dat_lmr_triplet *lmr_triplet, + struct dapl_lmr *lmr, + const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, struct dapl_ep *ep_ptr, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) { - struct dapl_lmr *lmr; struct dapl_cookie *cookie; u32 status; boolean_t is_signaled; - status = dapl_hash_search(rmr->common.owner_ia->hca->lmr_hash_table, - lmr_triplet->lmr_context, - (DAPL_HASH_DATA *) &lmr); - if (DAT_SUCCESS != status) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - /* * if the ep in unconnected return an error. IB requires that the * QP be connected to change a memory window binding since: @@ -352,6 +343,7 @@ * Output: */ u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, + struct dat_lmr *lmr_handle, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, @@ -359,20 +351,25 @@ DAT_RMR_CONTEXT *rmr_context) { struct dapl_rmr *rmr; + struct dapl_lmr *lmr; struct dapl_ep *ep_ptr; if (!rmr_handle) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); + if (!lmr_handle) + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); + if (!ep) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); rmr = (struct dapl_rmr *)rmr_handle; + lmr = (struct dapl_lmr *)lmr_handle; ep_ptr = (struct dapl_ep *)ep; /* if the rmr should be bound */ if (0 != lmr_triplet->segment_length) - return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, + return dapl_rmr_bind_fuse(rmr, lmr, lmr_triplet, mem_priv, ep_ptr, user_cookie, completion_flags, rmr_context); else /* the rmr should be unbound */ Index: dat-provider/dapl_util.h =================================================================== --- dat-provider/dapl_util.h (revision 2608) +++ dat-provider/dapl_util.h (working copy) @@ -40,7 +40,6 @@ #include #include #include -#include /* needed by hash functions */ #ifdef __ia64__ #include @@ -120,18 +119,6 @@ } /* - * String Functions - */ - -static inline char *dapl_os_strdup(const char *str) -{ - char *ns = kmalloc(strlen(str) + 1, GFP_KERNEL); - if (ns) - strcpy(ns, str); - return ns; -} - -/* * *printf format helper. We use the C string constant concatenation * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. @@ -143,22 +130,6 @@ #endif /* - * dapl_os_mod64 - * - * Returne the modulo of a 64 bit number. Given that this is running - * on a 32 bit platform, we need to use the kernel macro to prevent - * exceptions from killing the machine. - */ -static inline long dapl_os_mod64(uint64_t key, unsigned long hashsize) -{ -#ifdef __ia64__ - return ((uint64_t) ((key) % (hashsize))) -#else - return do_div(key, hashsize); -#endif -} - -/* * Debug Functions */ Index: dat-provider/dapl_hash.c =================================================================== --- dat-provider/dapl_hash.c (revision 2608) +++ dat-provider/dapl_hash.c (working copy) @@ -1,461 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Provides a generic hash table with chaining. - * - * $Id$ - */ - -#include "dapl_hash.h" - -/********************************************************************* - * * - * Structures * - * * - *********************************************************************/ - -/* - * A hash table element - */ -typedef struct DAPL_HASH_ELEM { - struct DAPL_HASH_ELEM *next_element; - DAPL_HASH_KEY key; - void *datum; -} DAPL_HASH_ELEM; - -/* - * The hash table - */ -struct dapl_hash_table { - unsigned long num_entries; - unsigned long tbl_size; - DAPL_HASH_ELEM *table; - spinlock_t lock; - unsigned long flags; - /* - * statistics - we tally on insert operations, counting - * the number of entries in the whole hash table, as - * well as the length of chains we walk to insert. This - * ignores empty buckets, giving us data on overall table - * occupancy, as well as max/average chain length for - * the buckets used. If our hash function results in - * hot buckets, this will show it. - */ - uint64_t hash_tbl_inserts; /* total inserts ops */ - uint64_t hash_tbl_max; /* max in entire table */ - uint64_t hash_tbl_total; /* total in table */ - uint64_t hash_chn_max; /* longest chain */ - uint64_t hash_chn_total; /* total non-0 lenghts */ -}; - -/********************************************************************* - * * - * Defines * - * * - *********************************************************************/ - -/* datum value in empty table slots (use 0UL or ~0UL as appropriate) */ -#define NO_DATUM_VALUE ((void *) 0UL) -#define NO_DATUM(value) ((value) == NO_DATUM_VALUE) - -/* Lookup macro (which falls back to function to rehash) */ -#define DAPL_HASHLOOKUP( p_table, in_key, out_datum, bucket_head) \ - do { \ - DAPL_HASH_KEY save_key = in_key; \ - DAPL_HASH_ELEM *element = \ - &((p_table)->table)[DAPL_DOHASH(in_key,(p_table)->tbl_size)]; \ - in_key = save_key; \ - if (NO_DATUM(element->datum)) { \ - (bucket_head) = (void *)0; \ - } else if (element->key == (DAPL_HASH_KEY) (in_key)) { \ - (out_datum) = element->datum; \ - (bucket_head) = (void *)element; \ - } else if (element->next_element) { \ - dapl_hash_rehash(element, \ - (in_key), \ - (void **)&(out_datum), \ - (DAPL_HASH_ELEM **)&(bucket_head)); \ - } else { \ - (bucket_head) = (void *)0; \ - }\ - } while (0) - -/********************************************************************* - * * - * Internal Functions * - * * - *********************************************************************/ - -/* - * Rehash the key (used by add and lookup functions) - * - * Inputs: element element to rehash key - * key, datum datum for key head - * head head for list - */ -static void -dapl_hash_rehash(DAPL_HASH_ELEM * element, - DAPL_HASH_KEY key, void **datum, DAPL_HASH_ELEM ** head) -{ - /* - * assume we looked at the contents of element already, - * and start with the next element. - */ - dapl_os_assert(element->next_element); - dapl_os_assert(!NO_DATUM(element->datum)); - - *head = element; - while (1) { - element = element->next_element; - if (!element) { - break; - } - if (element->key == key) { - *datum = element->datum; - return; - } - } - *head = NULL; -} - -/* - * Add a new key to the hash table - * - * Inputs: - * table, key and datum to be added - * allow_dup - TRUE if dups are allowed - * Outputs: - * report_dup - should you care to know - * Returns: - * TRUE on success - */ -static boolean_t -dapl_hash_add(struct dapl_hash_table *table, DAPL_HASH_KEY key, void *datum, - boolean_t allow_dup, boolean_t *report_dup) -{ - void *olddatum; - DAPL_HASH_KEY hashValue, save_key = key; - DAPL_HASH_ELEM *found; - boolean_t status = FALSE; - unsigned int chain_len = 0; - - if (report_dup) { - (*report_dup) = FALSE; - } - - if (NO_DATUM(datum)) { - /* - * Reserved value used for datum - */ - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_hash_add() called with magic NO_DATA " - "value (%p) used as datum!\n", datum); - return FALSE; - } - - DAPL_HASHLOOKUP(table, key, olddatum, found); - if (found) { - /* - * key exists already - */ - if (report_dup) { - *report_dup = TRUE; - } - - if (!allow_dup) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_hash_add() called with duplicate " - "key (" F64x ")\n", key); - return FALSE; - } - } - - hashValue = DAPL_DOHASH(key, table->tbl_size); - key = save_key; - if (NO_DATUM(table->table[hashValue].datum)) { - /* - * Empty head - just fill it in - */ - table->table[hashValue].key = key; - table->table[hashValue].datum = datum; - table->table[hashValue].next_element = NULL; - table->num_entries++; - status = TRUE; - } else { - DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, - GFP_ATOMIC); - /* - * Add an element to the end of the chain - */ - if (newelement) { - DAPL_HASH_ELEM *lastelement; - newelement->key = key; - newelement->datum = datum; - newelement->next_element = NULL; - for (lastelement = &table->table[hashValue]; - lastelement->next_element; - lastelement = lastelement->next_element) { - /* Walk to the end of the chain */ - chain_len++; - } - lastelement->next_element = newelement; - table->num_entries++; - status = TRUE; - } else - status = FALSE; - } - - /* - * Tally up our counters. chain_len is one less than current chain - * length. - */ - chain_len++; - table->hash_tbl_inserts++; - table->hash_tbl_total += table->num_entries; - table->hash_chn_total += chain_len; - if (table->num_entries > table->hash_tbl_max) { - table->hash_tbl_max = table->num_entries; - } - if (chain_len > table->hash_chn_max) { - table->hash_chn_max = chain_len; - } - - return status; -} - -/* - * Remove element from hash bucket - * - * Inputs: - * element, key to be deleted - * Returns: - * TRUE on success - */ -static boolean_t -dapl_hash_delete_element(DAPL_HASH_ELEM * element, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_datum) -{ - DAPL_HASH_ELEM *curelement; - DAPL_HASH_ELEM *lastelement; - - lastelement = NULL; - for (curelement = element; - curelement; - lastelement = curelement, curelement = curelement->next_element) { - if (curelement->key == key) { - if (p_datum) { - *p_datum = curelement->datum; - } - if (lastelement) { - /* - * curelement was malloc'd; free it - */ - lastelement->next_element = - curelement->next_element; - kfree(curelement); - } else { - /* - * curelement is static list head - */ - DAPL_HASH_ELEM *n = curelement->next_element; - if (n) { - /* - * If there is a next element, copy its contents into the - * head and free the original next element. - */ - curelement->key = n->key; - curelement->datum = n->datum; - curelement->next_element = - n->next_element; - kfree(n); - } else { - curelement->datum = NO_DATUM_VALUE; - } - } - break; - } - } - - return (curelement != NULL ? TRUE : FALSE); -} - -/********************************************************************* - * * - * External Functions * - * * - *********************************************************************/ - -/* - * Create a new hash table with at least 'table_size' hash buckets. - */ -u32 dapl_hash_create(int table_size, struct dapl_hash_table **pp_table) -{ - struct dapl_hash_table *p_table; - int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); - u32 dat_status = DAT_SUCCESS; - int i; - - dapl_os_assert(pp_table); - - /* Allocate hash table */ - p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); - if (!p_table) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - /* Init hash table, allocate and init and buckets */ - memset(p_table, 0, sizeof *p_table); - p_table->tbl_size = table_size; - p_table->table = kmalloc(table_length, GFP_ATOMIC); - if (!p_table->table) { - kfree(p_table); - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - spin_lock_init(&p_table->lock); - for (i = 0; i < table_size; i++) { - p_table->table[i].datum = NO_DATUM_VALUE; - p_table->table[i].key = 0; - p_table->table[i].next_element = NULL; - } - - *pp_table = p_table; - -bail: - return DAT_SUCCESS; -} - -/* - * Destroy a hash table - */ -u32 dapl_hash_free(struct dapl_hash_table *p_table) -{ - dapl_os_assert(p_table && p_table->table); - - /* no need to destroy p_table->lock */ - kfree(p_table->table); - kfree(p_table); - - return DAT_SUCCESS; -} - -/* - * Returns the number of elements stored in the table - */ - -u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size) -{ - dapl_os_assert(p_table && p_size); - - *p_size = p_table->num_entries; - - return DAT_SUCCESS; -} - -/* - * Inserts the specified data into the table with the given key. - * Duplicates are not expected, and return in error, having done nothing. - */ - -u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA data) -{ - u32 dat_status = DAT_SUCCESS; - - dapl_os_assert(p_table); - - spin_lock_irqsave(&p_table->lock, p_table->flags); - if (!dapl_hash_add(p_table, key, data, FALSE, NULL)) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - } - spin_unlock_irqrestore(&p_table->lock, p_table->flags); - - return dat_status; -} - -/* - * Searches for the given key. If found, - * DAT_SUCCESS is returned and the associated - * data is returned in the DAPL_HASH_DATA - * pointer if that pointer is not NULL. - */ -u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA *p_data) -{ - u32 dat_status; - void *olddatum; - DAPL_HASH_ELEM *found; - - dapl_os_assert(p_table); - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, 0); - - spin_lock_irqsave(&p_table->lock, p_table->flags); - DAPL_HASHLOOKUP(p_table, key, olddatum, found); - spin_unlock_irqrestore(&p_table->lock, p_table->flags); - - if (found) { - if (p_data) { - *p_data = olddatum; - } - dat_status = DAT_SUCCESS; - } - - return dat_status; -} - -u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA *p_data) -{ - u32 dat_status; - DAPL_HASH_KEY hashValue, save_key = key; - - dapl_os_assert(p_table); - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, 0); - - if (p_table->num_entries == 0) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_hash_remove () called on empty hash table!\n"); - return dat_status; - } - - hashValue = DAPL_DOHASH(key, p_table->tbl_size); - key = save_key; - spin_lock_irqsave(&p_table->lock, p_table->flags); - if (dapl_hash_delete_element(&p_table->table[hashValue], key, p_data)) { - p_table->num_entries--; - dat_status = DAT_SUCCESS; - } - spin_unlock_irqrestore(&p_table->lock, p_table->flags); - - return dat_status; -} Index: dat-provider/dapl.h =================================================================== --- dat-provider/dapl.h (revision 2608) +++ dat-provider/dapl.h (working copy) @@ -83,9 +83,6 @@ typedef void (*ib_async_handler_t) (struct ib_event *, void *); -typedef u64 DAPL_HASH_KEY; -typedef void *DAPL_HASH_DATA; - /********************************************************************* * * * Structures * @@ -124,8 +121,6 @@ u8 port_num; struct ib_device *ib_hca_handle; struct ib_cq *null_cq; /* CQ with 0 entries */ - /* Memory Subsystem Support */ - struct dapl_hash_table *lmr_hash_table; /* Limits & useful HCA attributes */ struct dat_ia_attr ia_attr; struct dat_ep_attr ep_attr; @@ -542,6 +537,7 @@ struct dat_rmr_param *rmr_args); extern u32 dapl_rmr_bind(struct dat_rmr *rmr, + struct dat_lmr *lmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, Index: dat-provider/dapl_hash.h =================================================================== --- dat-provider/dapl_hash.h (revision 2608) +++ dat-provider/dapl_hash.h (working copy) @@ -1,80 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_HASH_H -#define DAPL_HASH_H - -#include "dapl.h" - -/********************************************************************* - * * - * Defines * - * * - *********************************************************************/ - -/* - * Hash table size. - * - * Default is small; use the larger sample values for hash tables - * known to be heavily used. The sample values chosen are the - * largest primes below 2^8, 2^9, and 2^10. - */ -#define DAPL_DEF_HASHSIZE 251 -#define DAPL_MED_HASHSIZE 509 -#define DAPL_LRG_HASHSIZE 1021 - -#define DAPL_HASH_TABLE_DEFAULT_CAPACITY DAPL_DEF_HASHSIZE - -/* The hash function */ -#define DAPL_DOHASH(key,hashsize) dapl_os_mod64(key,hashsize) - -/********************************************************************* - * * - * Function Prototypes * - * * - *********************************************************************/ - -extern u32 dapl_hash_create(int capacity, struct dapl_hash_table **pp_table); - -extern u32 dapl_hash_free(struct dapl_hash_table *p_table); - -extern u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size); - -extern u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA data); - -extern u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA *p_data); - -extern u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA *p_data); - -#endif /* DAPL_HASH_H */ Index: dat-provider/dapl_provider.c =================================================================== --- dat-provider/dapl_provider.c (revision 2608) +++ dat-provider/dapl_provider.c (working copy) @@ -34,7 +34,6 @@ #include #include "dapl.h" -#include "dapl_hca_util.h" #include "dapl_provider.h" #include "dapl_util.h" #include "dapl_openib_util.h" @@ -247,6 +246,23 @@ provider_info->is_thread_safe = DAPL_THREADSAFE; } +struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) +{ + struct dapl_hca *hca; + int malloc_size = sizeof *hca + strlen(name) + 1; + hca = kmalloc(malloc_size, GFP_ATOMIC); + if (hca) { + memset(hca, 0, malloc_size); + spin_lock_init(&hca->lock); + INIT_LIST_HEAD(&hca->ia_list); + hca->name = (char *)hca + sizeof *hca; + strcpy(hca->name, name); + hca->ib_hca_handle = device; + hca->port_num = port; + } + return hca; +} + static void dapl_add_port(struct ib_device *device, u8 port) { struct dat_provider_info provider_info; @@ -306,7 +322,7 @@ (void)dapl_provider_list_remove(provider_info.ia_name); if (NULL != hca) - dapl_hca_free(hca); + kfree(hca); } } @@ -339,7 +355,7 @@ provider_info.ia_name); } - dapl_hca_free(provider->extension); + kfree(provider->extension); dapl_provider_list_remove(provider_info.ia_name); } Index: dat/dat.h =================================================================== --- dat/dat.h (revision 2608) +++ dat/dat.h (working copy) @@ -1153,7 +1153,8 @@ typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); -typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, struct dat_lmr *, + const struct dat_lmr_triplet *, enum dat_mem_priv_flags, struct dat_ep *, DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); @@ -1638,14 +1639,15 @@ } static inline u32 dat_rmr_bind(struct dat_rmr * rmr, - const struct dat_lmr_triplet *iov, + struct dat_lmr *lmr, + const struct dat_lmr_triplet *iov, enum dat_mem_priv_flags mem_flags, struct dat_ep *ep, DAT_RMR_COOKIE cookie, enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) { - return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, iov, mem_flags, ep, - cookie, comp_flags, context); + return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, lmr, iov, mem_flags, + ep, cookie, comp_flags, context); } static inline u32 dat_rmr_free(struct dat_rmr * rmr) -- Itamar From jlentini at netapp.com Wed Jun 15 07:02:45 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 15 Jun 2005 10:02:45 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: get rid of dat version and thread safetey flags In-Reply-To: <20050614222433.GA20042@lst.de> References: <20050611093144.GA18469@lst.de> <20050614222433.GA20042@lst.de> Message-ID: Committed in revision 2609. On Wed, 15 Jun 2005, Christoph Hellwig wrote: > On Tue, Jun 14, 2005 at 03:16:23PM -0400, James Lentini wrote: >> >> Christoph, >> >> I'm all ready to submit this minus the Makefile change. Can you >> re-send with a "Signed-off-by" line? (no need to make any changes) > > Signed-off-by: Christoph Hellwig > > Index: test/dapltest/test/dapl_test_util.c > =================================================================== > --- test/dapltest/test/dapl_test_util.c (revision 2588) > +++ test/dapltest/test/dapl_test_util.c (working copy) > @@ -76,12 +76,10 @@ > struct sockaddr_in *ip_addr; > > DT_Tdep_PT_Printf (phead, "***** DAPL Characteristics *****\n"); > - DT_Tdep_PT_Printf (phead, "Provider: %s Version %d.%d DAPL %d.%d\n", > + DT_Tdep_PT_Printf (phead, "Provider: %s Version %d.%d\n", > pt_ptr->provider_attr.provider_name, > pt_ptr->provider_attr.provider_version_major, > - pt_ptr->provider_attr.provider_version_minor, > - pt_ptr->provider_attr.dat_version_major, > - pt_ptr->provider_attr.dat_version_minor ); > + pt_ptr->provider_attr.provider_version_minor); > DT_Tdep_PT_Printf (phead, "Adapter: %s by %s Version %d.%d\n", > pt_ptr->ia_attr.adapter_name, > pt_ptr->ia_attr.vendor_name, > Index: dat-provider/dapl_ia.c > =================================================================== > --- dat-provider/dapl_ia.c (revision 2588) > +++ dat-provider/dapl_ia.c (working copy) > @@ -881,15 +881,12 @@ > DAT_NAME_MAX_LENGTH); > provider_attr->provider_version_major = DAPL_PROVIDER_MAJOR; > provider_attr->provider_version_minor = DAPL_PROVIDER_MINOR; > - provider_attr->dat_version_major = DAT_VERSION_MAJOR; > - provider_attr->dat_version_minor = DAT_VERSION_MINOR; > provider_attr->lmr_mem_types_supported = > DAT_MEM_TYPE_VIRTUAL | DAT_MEM_TYPE_LMR; > provider_attr->iov_ownership_on_return = DAT_IOV_CONSUMER; > provider_attr->dat_qos_supported = DAT_QOS_BEST_EFFORT; > provider_attr->completion_flags_supported = > DAT_COMPLETION_DEFAULT_FLAG; > - provider_attr->is_thread_safe = FALSE; > provider_attr->max_private_data_size = IB_CM_REQ_PRIVATE_DATA_SIZE; > provider_attr->supports_multipath = FALSE; > provider_attr->ep_creator = DAT_PSP_CREATES_EP_NEVER; > Index: dat-provider/dapl_provider.c > =================================================================== > --- dat-provider/dapl_provider.c (revision 2588) > +++ dat-provider/dapl_provider.c (working copy) > @@ -244,9 +244,6 @@ > (size_t) (DAT_NAME_MAX_LENGTH - 2)); > provider_info->ia_name[i] = 'a' + port - 1; /* IB ports are 1 based */ > provider_info->ia_name[i+1] = '\0'; > - provider_info->dat_version_major = DAT_VERSION_MAJOR; > - provider_info->dat_version_minor = DAT_VERSION_MINOR; > - provider_info->is_thread_safe = DAPL_THREADSAFE; > } > > static void dapl_add_port(struct ib_device *device, u8 port) > Index: dat/dictionary.c > =================================================================== > --- dat/dictionary.c (revision 2588) > +++ dat/dictionary.c (working copy) > @@ -212,10 +212,6 @@ > BUG_ON(NULL == new_key); > > strncpy(new_key->ia_name, old_key->ia_name, DAT_NAME_MAX_LENGTH); > - new_key->dat_version_major = old_key->dat_version_major; > - new_key->dat_version_minor = old_key->dat_version_minor; > - new_key->is_thread_safe = old_key->is_thread_safe; > - > return DAT_SUCCESS; > } > > @@ -226,10 +222,7 @@ > BUG_ON(NULL == b); > > if ((strlen(a->ia_name) == strlen(b->ia_name)) && > - (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name))) > - && (a->dat_version_major == b->dat_version_major) > - && (a->dat_version_minor == b->dat_version_minor) > - && (a->is_thread_safe == b->is_thread_safe)) > + (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name)))) > return TRUE; > else > return FALSE; > Index: dat/api.c > =================================================================== > --- dat/api.c (revision 2588) > +++ dat/api.c (working copy) > @@ -349,9 +349,8 @@ > } > } > > -u32 dat_ia_openv(const char *name, int async_event_qlen, > - struct dat_evd **async_event_handle, struct dat_ia **ia, > - u32 dat_major, u32 dat_minor, boolean_t thread_safety) > +u32 dat_ia_open(const char *name, int async_event_qlen, > + struct dat_evd **async_event_handle, struct dat_ia **ia) > { > DAT_IA_OPEN_FUNC ia_open_func; > struct dat_provider_info info; > @@ -360,8 +359,7 @@ > > dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > "%s(name:%s, async_qlen:%i, ver:%x.%x, thrd_safe:%x)\n", > - __func__, name, async_event_qlen, dat_major, dat_minor, > - thread_safety); > + __func__, name, async_event_qlen); > > if (DAT_IS_BAD_POINTER(name)) > return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); > @@ -374,10 +372,6 @@ > strncpy(info.ia_name, name, len); > info.ia_name[len] = '\0'; > > - info.dat_version_major = dat_major; > - info.dat_version_minor = dat_minor; > - info.is_thread_safe = thread_safety; > - > status = dat_dr_provider_open(&info, &ia_open_func); > if (status != DAT_SUCCESS) { > dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > @@ -390,7 +384,7 @@ > return ia_open_func(name, async_event_qlen, async_event_handle, ia); > } > > -EXPORT_SYMBOL(dat_ia_openv); > +EXPORT_SYMBOL(dat_ia_open); > > u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) > { > @@ -423,10 +417,6 @@ > strncpy(info.ia_name, ia_name, len); > info.ia_name[len] = '\0'; > > - info.dat_version_major = provider_attr.dat_version_major; > - info.dat_version_minor = provider_attr.dat_version_minor; > - info.is_thread_safe = provider_attr.is_thread_safe; > - > status = dat_dr_provider_close(&info); > if (DAT_SUCCESS != status) > dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > @@ -472,10 +462,7 @@ > { > dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, > "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", > - __func__, provider_info->ia_name, > - provider_info->dat_version_major, > - provider_info->dat_version_minor, > - provider_info->is_thread_safe); > + __func__, provider_info->ia_name); > > if (DAT_IS_BAD_POINTER(provider)) > return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); > Index: dat/Makefile > =================================================================== > --- dat/Makefile (revision 2588) > +++ dat/Makefile (working copy) > @@ -1,7 +1,6 @@ > > EXTRA_CFLAGS += \ > - -Idrivers/dat \ > - -Werror > + -I$(obj) > > obj-$(CONFIG_DAT) += dat.o > > Index: dat/dat.h > =================================================================== > --- dat/dat.h (revision 2588) > +++ dat/dat.h (working copy) > @@ -33,10 +33,6 @@ > #include > #include > > -#define DAT_VERSION_MAJOR 1 > -#define DAT_VERSION_MINOR 2 > -#define DAT_THREADSAFE FALSE > - > /* > * All return codes are actually a 3-way tuple: > * > @@ -894,9 +890,6 @@ > > struct dat_provider_info { > char ia_name[DAT_NAME_MAX_LENGTH]; > - u32 dat_version_major; > - u32 dat_version_minor; > - boolean_t is_thread_safe; > }; > > struct dat_evd_param { > @@ -1017,13 +1010,10 @@ > char provider_name[DAT_NAME_MAX_LENGTH]; > u32 provider_version_major; > u32 provider_version_minor; > - u32 dat_version_major; > - u32 dat_version_minor; > enum dat_mem_type lmr_mem_types_supported; > enum dat_iov_ownership iov_ownership_on_return; > enum dat_qos dat_qos_supported; > enum dat_completion_flags completion_flags_supported; > - boolean_t is_thread_safe; > int max_private_data_size; > boolean_t supports_multipath; > enum dat_ep_creator_for_psp ep_creator; > @@ -1361,16 +1351,10 @@ > /* > * DAT registry functions for consumers > */ > -extern u32 dat_ia_openv(const char *name, int async_event_qlen, > +extern u32 dat_ia_open(const char *name, int async_event_qlen, > struct dat_evd **async_event_handle, > - struct dat_ia **ia, u32 dat_major, u32 dat_minor, > - boolean_t thread_safety); > + struct dat_ia **ia); > > -#define dat_ia_open(name, qlen, async_evd, ia) \ > - dat_ia_openv((name), (qlen), (async_evd), (ia), \ > - DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ > - DAT_THREADSAFE) > - > extern u32 dat_ia_close(struct dat_ia *, enum dat_close_flags); > > extern u32 dat_strerror(u32, const char **, const char **); > From jlentini at netapp.com Wed Jun 15 07:14:10 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 15 Jun 2005 10:14:10 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: make some stuff static after file consolidation In-Reply-To: <1118789921.21577.5.camel@duffman> References: <1118789921.21577.5.camel@duffman> Message-ID: Committed in revision 2610. On Tue, 14 Jun 2005, Tom Duffy wrote: tduffy> These functions can be declared static now that we have consolidated tduffy> some of the files. tduffy> tduffy> James, this patch does not, but can I delete these two functions (they tduffy> seem to be unused): tduffy> tduffy> dapl_evd_qp_async_error_callback() tduffy> dapl_evd_cq_async_error_callback() tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ia.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia.c (revision 2605) tduffy> +++ linux-kernel/dat-provider/dapl_ia.c (working copy) tduffy> @@ -38,8 +38,8 @@ tduffy> #include "dapl_sp.h" tduffy> #include "dapl_cr.h" tduffy> tduffy> -struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, tduffy> - struct dapl_hca *hca) tduffy> +static struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, tduffy> + struct dapl_hca *hca) tduffy> { tduffy> struct dapl_ia *ia; tduffy> tduffy> @@ -75,7 +75,7 @@ struct dapl_ia *dapl_ia_alloc(struct dat tduffy> return ia; tduffy> } tduffy> tduffy> -u32 dapl_ia_teardown_callbacks(struct dapl_ia *ia) tduffy> +static u32 dapl_ia_teardown_callbacks(struct dapl_ia *ia) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> @@ -100,7 +100,7 @@ bail: tduffy> * Performs an abrupt close of the IA tduffy> */ tduffy> tduffy> -u32 dapl_ia_abrupt_close(struct dapl_ia *ia) tduffy> +static u32 dapl_ia_abrupt_close(struct dapl_ia *ia) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> struct dapl_ep *ep; tduffy> @@ -271,7 +271,7 @@ u32 dapl_ia_abrupt_close(struct dapl_ia tduffy> * tduffy> */ tduffy> tduffy> -u32 dapl_ia_graceful_close(struct dapl_ia *ia) tduffy> +static u32 dapl_ia_graceful_close(struct dapl_ia *ia) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> u32 cur_dat_status; tduffy> @@ -555,8 +555,8 @@ void dapl_ia_link_rsp(struct dapl_ia *ia tduffy> spin_unlock_irqrestore(&ia->common.lock, ia->common.flags); tduffy> } tduffy> tduffy> -u32 dapl_ia_setup_callbacks(struct dapl_ia *ia, tduffy> - struct dapl_evd *async_evd) tduffy> +static u32 dapl_ia_setup_callbacks(struct dapl_ia *ia, tduffy> + struct dapl_evd *async_evd) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.c (revision 2605) tduffy> +++ linux-kernel/dat-provider/dapl_evd.c (working copy) tduffy> @@ -30,6 +30,7 @@ tduffy> */ tduffy> tduffy> #include "dapl.h" tduffy> +#include "dapl_evd.h" tduffy> #include "dapl_cookie.h" tduffy> #include "dapl_ia.h" tduffy> #include "dapl_openib_util.h" tduffy> @@ -142,10 +143,10 @@ bail: tduffy> /* tduffy> * alloc and initialize an EVD struct tduffy> */ tduffy> -struct dapl_evd *dapl_evd_alloc(struct dapl_ia *ia, int qlen, tduffy> - enum dat_upcall_policy upcall_policy, tduffy> - const struct dat_upcall_object *upcall, tduffy> - enum dat_evd_flags flags) tduffy> +static struct dapl_evd *dapl_evd_alloc(struct dapl_ia *ia, int qlen, tduffy> + enum dat_upcall_policy upcall_policy, tduffy> + const struct dat_upcall_object *upcall, tduffy> + enum dat_evd_flags flags) tduffy> { tduffy> struct dapl_evd *evd; tduffy> tduffy> @@ -627,7 +628,8 @@ static u32 dapl_evd_cq_poll_to_event(str tduffy> return status; tduffy> } tduffy> tduffy> -void dapl_evd_qp_async_error_callback(struct ib_event *cause, void *context) tduffy> +static void dapl_evd_qp_async_error_callback(struct ib_event *cause, tduffy> + void *context) tduffy> { tduffy> struct dapl_ep *ep; tduffy> struct dapl_evd *async_evd; tduffy> @@ -666,7 +668,8 @@ void dapl_evd_qp_async_error_callback(st tduffy> "dapl_evd_qp_async_error_callback () returns\n"); tduffy> } tduffy> tduffy> -void dapl_evd_cq_async_error_callback(struct ib_event *cause, void *context) tduffy> +static void dapl_evd_cq_async_error_callback(struct ib_event *cause, tduffy> + void *context) tduffy> { tduffy> struct dapl_evd *async_evd; tduffy> struct dapl_evd *evd = context; tduffy> @@ -870,7 +873,7 @@ void dapl_evd_connection_callback(struct tduffy> tduffy> } tduffy> tduffy> -void dapl_evd_dto_callback(struct ib_cq *cq, void *user_context) tduffy> +static void dapl_evd_dto_callback(struct ib_cq *cq, void *user_context) tduffy> { tduffy> struct dapl_evd *evd; tduffy> DAPL_EVD_STATE state; tduffy> Index: linux-kernel/dat-provider/dapl_sp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp.c (revision 2605) tduffy> +++ linux-kernel/dat-provider/dapl_sp.c (working copy) tduffy> @@ -36,7 +36,7 @@ tduffy> #include "dapl_ia.h" tduffy> #include "dapl_openib_util.h" tduffy> tduffy> -struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia, boolean_t is_psp) tduffy> +static struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia, boolean_t is_psp) tduffy> { tduffy> struct dapl_sp *sp; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_provider.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_provider.c (revision 2605) tduffy> +++ linux-kernel/dat-provider/dapl_provider.c (working copy) tduffy> @@ -143,7 +143,7 @@ static struct dat_provider g_dapl_provid tduffy> * * tduffy> *********************************************************************/ tduffy> tduffy> -void dapl_provider_list_destroy(void) tduffy> +static void dapl_provider_list_destroy(void) tduffy> { tduffy> struct list_head *cur_list, *next_list; tduffy> struct dapl_provider_entry *cur_entry; tduffy> @@ -156,7 +156,8 @@ void dapl_provider_list_destroy(void) tduffy> } tduffy> } tduffy> tduffy> -u32 dapl_provider_list_insert(const char *name, struct dat_provider **provider) tduffy> +static u32 dapl_provider_list_insert(const char *name, tduffy> + struct dat_provider **provider) tduffy> { tduffy> struct dapl_provider_entry *entry; tduffy> u32 status = DAT_SUCCESS; tduffy> @@ -210,7 +211,7 @@ error: tduffy> return status; tduffy> } tduffy> tduffy> -u32 dapl_provider_list_remove(const char *name) tduffy> +static u32 dapl_provider_list_remove(const char *name) tduffy> { tduffy> struct list_head *cur_list, *next_list; tduffy> struct dapl_provider_entry *cur_entry; tduffy> From mst at mellanox.co.il Wed Jun 15 08:04:40 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 18:04:40 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050602005228.GP25321@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> Message-ID: <20050615150440.GB20041@mellanox.co.il> Quoting r. Grant Grundler : > Subject: [PATCH] rdma_lat-09 and results > > Michael, > > Good news: > My next cleanup of rdma_lat.c is working and patch is appended. > Summary of changes below. > > Bad News: > perf is about ~15 cycles slower since the last time I tested. > (Hrm...maybe it's time to cycle power on the TS90 switch again.) > > > Here's with the new rdma_lat.c: > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C > local address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > remote address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > Latency typical: 7140 cycles > Latency best : 6915 cycles > Latency worst : 52915.5 cycles > grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ > > And the "client" side: > grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_lat -C 10.0.0.51 > local address: LID 0x25 QPN 0x70406 PSN 0x5d4824 RKey 0x2a0434 VAddr 0x6000000000014001 > remote address: LID 0x27 QPN 0x80406 PSN 0x9188f7 RKey 0x300434 VAddr 0x6000000000014001 > Latency typical: 7140 cycles > Latency best : 6907 cycles > Latency worst : 94920 cycles > > > The previous set of rdma_lat results are here: > http://openib.org/pipermail/openib-general/2005-May/006721.html > > I'll guess the previous SVN verion was no older than r2229. > > > I get 7140 to 7151 for the original rdma_lat. Usually 7147.5. > I get 7132 to 7155 with my version of rdma_lat. Usually 7140. > No statistically significant differences. > Both essentially agree on the higher result. > Using "-n 10000" gave more consistent results * > > I use "taskset" to bind the rdma_lat test to a CPU. > But it didn't matter which CPU I bound the task to - results > where basically the same. I suspect the "stream" mode just > does not depend on or generating that many interrupts. > > > diffstat rdma_lat.c-09-diff > rdma_lat.c | 395 +++++++++++++++++++++++++++++-------------------------------- > 1 files changed, 188 insertions(+), 207 deletions(-) > > Commit Log entry/Summary of changes: > o move device lookup from main() to pp_find_dev() > o move sockfd handling code to pp_open_port() > o consolidate server/client "key exchange" code path > o enumerate return values in main() > o fixed nit: pp_*_exch_dest was called twice. > Each time it would malloc a new "rem_dest". > Code in pp_open_port() now free()'s the first one. > > Signed-off-by: Grant Grundler > > thanks, > grant OK, I accepted most of it. Thanks! There were some issues that I fixed, I also made some other small cleanups. One thing I skipped is the pp_connect_sock consolidation: I like to keep the split between client and server at the top level. Connect routines have some code duplication but I dont like spreading if (servername) all around them, and I dont mind duplication much since this is standard socket stuff. Grant, I'm not sure I understand the reason for > o enumerate return values in main() bit but I went along with it. Why do you like it? The patch also added free(tstamp) in a couple of places, but since we dont bother to free resources (qp, cq, ...) and since this was in the test part which must be cristal clear, I skipped this bit. If we ever do a proper cleanup, its probably a good idea to goto outside the loop and handle errors there. Here's what I ended up checking in: Index: rdma_lat.c =================================================================== --- rdma_lat.c (revision 2608) +++ rdma_lat.c (working copy) @@ -103,6 +103,71 @@ static uint16_t pp_get_local_lid(struct return attr.lid; } +static struct ibv_device *pp_find_dev(const char *ib_devname) +{ + struct dlist *dev_list; + struct ibv_device *ib_dev = NULL; + + dev_list = ibv_get_devices(); + + dlist_start(dev_list); + if (!ib_devname) { + ib_dev = dlist_next(dev_list); + if (!ib_dev) + fprintf(stderr, "No IB devices found\n"); + } else { + dlist_for_each_data(dev_list, ib_dev, struct ibv_device) + if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) + break; + if (!ib_dev) + fprintf(stderr, "IB device %s not found\n", ib_devname); + } + return ib_dev; +} + +#define KEY_MSG_SIZE (sizeof "0000:000000:000000:00000000:0000000000000000") +#define KEY_PRINT_FMT "%04x:%06x:%06x:%08x:%016Lx" + +static int pp_write_keys(int sockfd, const struct pingpong_dest *my_dest) +{ + char msg[KEY_MSG_SIZE]; + + sprintf(msg, KEY_PRINT_FMT, my_dest->lid, my_dest->qpn, + my_dest->psn, my_dest->rkey, my_dest->vaddr); + + if (write(sockfd, msg, sizeof msg) != sizeof msg) { + perror("client write"); + fprintf(stderr, "Couldn't send local address\n"); + return -1; + } + + return 0; +} + +static int pp_read_keys(int sockfd, const struct pingpong_dest *my_dest, + struct pingpong_dest *rem_dest) +{ + int parsed; + char msg[KEY_MSG_SIZE]; + + if (read(sockfd, msg, sizeof msg) != sizeof msg) { + perror("pp_read_keys"); + fprintf(stderr, "Couldn't read remote address\n"); + return -1; + } + + parsed = sscanf(msg, KEY_PRINT_FMT, &rem_dest->lid, &rem_dest->qpn, + &rem_dest->psn, &rem_dest->rkey, &rem_dest->vaddr); + + if (parsed != 5) { + fprintf(stderr, "Couldn't parse line <%.*s>\n", + (int)sizeof msg, msg); + return -1; + } + + return 0; +} + static int pp_client_connect(const char *servername, int port) { struct addrinfo *res, *t; @@ -141,46 +206,16 @@ static int pp_client_connect(const char return sockfd; } -struct pingpong_dest * pp_client_exch_dest(int sockfd, - const struct pingpong_dest *my_dest) +static int pp_client_exch_dest(int sockfd, const struct pingpong_dest *my_dest, + struct pingpong_dest *rem_dest) { - struct pingpong_dest *rem_dest = NULL; - char msg[sizeof "0000:000000:000000:00000000:0000000000000000"]; - int parsed; - - sprintf(msg, "%04x:%06x:%06x:%08x:%016Lx", my_dest->lid, my_dest->qpn, - my_dest->psn,my_dest->rkey,my_dest->vaddr); - if (write(sockfd, msg, sizeof msg) != sizeof msg) { - perror("client write"); - fprintf(stderr, "Couldn't send local address\n"); - goto out; - } - - if (read(sockfd, msg, sizeof msg) != sizeof msg) { - perror("client read"); - fprintf(stderr, "Couldn't read remote address\n"); - goto out; - } + if (pp_write_keys(sockfd, my_dest)) + return -1; - rem_dest = malloc(sizeof *rem_dest); - if (!rem_dest) - goto out; - - parsed = sscanf(msg, "%x:%x:%x:%x:%Lx", &rem_dest->lid, &rem_dest->qpn, - &rem_dest->psn,&rem_dest->rkey,&rem_dest->vaddr); - - if (parsed != 5) { - fprintf(stderr, "Couldn't parse line <%.*s>\n",(int)sizeof msg, - msg); - free(rem_dest); - rem_dest = NULL; - goto out; - } -out: - return rem_dest; + return pp_read_keys(sockfd, my_dest, rem_dest); } -int pp_server_connect(int port) +static int pp_server_connect(int port) { struct addrinfo *res, *t; struct addrinfo hints = { @@ -234,45 +269,14 @@ int pp_server_connect(int port) return connfd; } -static struct pingpong_dest *pp_server_exch_dest(int connfd, const struct pingpong_dest *my_dest) +static int pp_server_exch_dest(int sockfd, const struct pingpong_dest *my_dest, + const struct pingpong_dest* rem_dest) { - char msg[sizeof "0000:000000:000000:00000000:0000000000000000"]; - struct pingpong_dest *rem_dest = NULL; - int parsed; - int n; - - n = read(connfd, msg, sizeof msg); - if (n != sizeof msg) { - perror("server read"); - fprintf(stderr, "%d/%d: Couldn't read remote address\n", n, (int) sizeof msg); - goto out; - } - rem_dest = malloc(sizeof *rem_dest); - if (!rem_dest) - goto out; - - parsed = sscanf(msg, "%x:%x:%x:%x:%Lx", &rem_dest->lid, &rem_dest->qpn, - &rem_dest->psn, &rem_dest->rkey, &rem_dest->vaddr); - if (parsed != 5) { - fprintf(stderr, "Couldn't parse line <%.*s>\n",(int)sizeof msg, - msg); - free(rem_dest); - rem_dest = NULL; - goto out; - } + if (pp_read_keys(sockfd, my_dest, rem_dest)) + return -1; - sprintf(msg, "%04x:%06x:%06x:%08x:%016Lx", my_dest->lid, my_dest->qpn, - my_dest->psn, my_dest->rkey, my_dest->vaddr); - if (write(connfd, msg, sizeof msg) != sizeof msg) { - perror("server write"); - fprintf(stderr, "Couldn't send local address\n"); - free(rem_dest); - rem_dest = NULL; - goto out; - } -out: - return rem_dest; + return pp_write_keys(sockfd, my_dest); } static struct pingpong_context *pp_init_ctx(struct ibv_device *ib_dev, int size, @@ -424,6 +428,65 @@ static int pp_connect_ctx(struct pingpon return 0; } +static int pp_open_port(struct pingpong_context *ctx, const char * servername, int ib_port, int port, + struct pingpong_dest *rem_dest) +{ + char addr_fmt[] = "%8s address: LID %#04x QPN %#06x PSN %#06x RKey %#08x VAddr %#016Lx\n"; + struct pingpong_dest my_dest; + int sockfd; + int rc; + + + /* Create connection between client and server. + * We do it by exchanging data over a TCP socket connection. */ + + my_dest.lid = pp_get_local_lid(ctx, ib_port); + my_dest.qpn = ctx->qp->qp_num; + my_dest.psn = lrand48() & 0xffffff; + if (!my_dest.lid) { + fprintf(stderr, "Local lid 0x0 detected. Is an SM running?\n"); + return -1; + } + my_dest.rkey = ctx->mr->rkey; + my_dest.vaddr = (uintptr_t)ctx->buf + ctx->size; + + printf(addr_fmt, "local", my_dest.lid, my_dest.qpn, my_dest.psn, + my_dest.rkey, my_dest.vaddr); + + sockfd = servername ? pp_client_connect(servername, port) : pp_server_connect(port); + + if (sockfd < 0) { + printf("pp_connect_sock(%s,%d) failed (%d)!\n", + servername, port, sockfd); + return sockfd; + } + + rc = servername ? pp_client_exch_dest(sockfd, &my_dest) : + pp_server_exch_dest(sockfd, &my_dest); + if (rc) + return rc; + + printf(addr_fmt, "remote", rem_dest->lid, rem_dest->qpn, rem_dest->psn, + rem_dest->rkey, rem_dest->vaddr); + + if ((rc = pp_connect_ctx(ctx, ib_port, my_dest.psn, rem_dest))) + return rc; + + /* An additional handshake is required *after* moving qp to RTR. + * Arbitrarily reuse exch_dest for this purpose. + */ + + rc = servername ? pp_client_exch_dest(sockfd, &my_dest) : + pp_server_exch_dest(sockfd, &my_dest); + + if (rc) + return rc; + + write(sockfd, "done", sizeof "done"); + close(sockfd); + return 0; +} + static void usage(const char *argv0) { printf("Usage:\n"); @@ -515,30 +578,29 @@ static void print_report(struct report_o free(delta); } - int main(int argc, char *argv[]) { - struct dlist *dev_list; - struct ibv_device *ib_dev; - struct pingpong_context *ctx; - struct pingpong_dest my_dest; - struct pingpong_dest *rem_dest; - char *ib_devname = NULL; - char *servername = NULL; + const char *ib_devname = NULL; + const char *servername = NULL; int port = 18515; int ib_port = 1; int size = 1; - int tx_depth = 50; int iters = 1000; - int scnt, rcnt, ccnt; - int sockfd; - struct ibv_qp *qp; - struct ibv_send_wr *wr; - volatile char *poll_buf; - volatile char *post_buf; + int tx_depth = 50; struct report_options report = {}; - cycles_t *tstamp; + struct pingpong_context *ctx; + struct pingpong_dest rem_dest; + struct ibv_device *ib_dev; + + struct ibv_qp *qp; + struct ibv_send_wr *wr; + volatile char *poll_buf; + volatile char *post_buf; + + int scnt, rcnt, ccnt; + + cycles_t *tstamp; /* Parameter parsing. */ while (1) { @@ -578,25 +640,25 @@ int main(int argc, char *argv[]) ib_port = strtol(optarg, NULL, 0); if (ib_port < 0) { usage(argv[0]); - return 1; + return 2; } break; case 's': size = strtol(optarg, NULL, 0); - if (size < 1) { usage(argv[0]); return 1; } + if (size < 1) { usage(argv[0]); return 3; } break; case 't': tx_depth = strtol(optarg, NULL, 0); - if (tx_depth < 1) { usage(argv[0]); return 1; } + if (tx_depth < 1) { usage(argv[0]); return 4; } break; case 'n': iters = strtol(optarg, NULL, 0); if (iters < 2) { usage(argv[0]); - return 1; + return 5; } break; @@ -615,7 +677,7 @@ int main(int argc, char *argv[]) default: usage(argv[0]); - return 1; + return 5; } } @@ -623,90 +685,26 @@ int main(int argc, char *argv[]) servername = strdupa(argv[optind]); else if (optind < argc) { usage(argv[0]); - return 1; + return 6; } - - /* Done with parameter parsing. Perform setup. */ + /* + * Done with parameter parsing. Perform setup. + */ srand48(getpid() * time(NULL)); - page_size = sysconf(_SC_PAGESIZE); - dev_list = ibv_get_devices(); - - dlist_start(dev_list); - if (!ib_devname) { - ib_dev = dlist_next(dev_list); - if (!ib_dev) { - fprintf(stderr, "No IB devices found\n"); - return 1; - } - } else { - dlist_for_each_data(dev_list, ib_dev, struct ibv_device) - if (!strcmp(ibv_get_device_name(ib_dev), ib_devname)) - break; - if (!ib_dev) { - fprintf(stderr, "IB device %s not found\n", ib_devname); - return 1; - } - } + ib_dev = pp_find_dev(ib_devname); + if (!ib_dev) + return 7; ctx = pp_init_ctx(ib_dev, size, tx_depth, ib_port); if (!ctx) - return 1; - - /* Create connection between client and server. - * We do it by exchanging data over a TCP socket connection. */ - - my_dest.lid = pp_get_local_lid(ctx, ib_port); - my_dest.qpn = ctx->qp->qp_num; - my_dest.psn = lrand48() & 0xffffff; - if (!my_dest.lid) { - fprintf(stderr, "Local lid 0x0 detected. Is an SM running?\n"); - return 1; - } - my_dest.rkey = ctx->mr->rkey; - my_dest.vaddr = (uintptr_t)ctx->buf + ctx->size; - - printf(" local address: LID %#04x, QPN %#06x, PSN %#06x " - "RKey %#08x VAddr %#016Lx\n", - my_dest.lid, my_dest.qpn, my_dest.psn, - my_dest.rkey, my_dest.vaddr); - - if (servername) { - sockfd = pp_client_connect(servername, port); - if (sockfd < 0) - return 1; - rem_dest = pp_client_exch_dest(sockfd, &my_dest); - } else { - sockfd = pp_server_connect(port); - if (sockfd < 0) - return 1; - rem_dest = pp_server_exch_dest(sockfd, &my_dest); - } - - if (!rem_dest) - return 1; - - printf(" remote address: LID %#04x, QPN %#06x, PSN %#06x, " - "RKey %#08x VAddr %#016Lx\n", - rem_dest->lid, rem_dest->qpn, rem_dest->psn, - rem_dest->rkey, rem_dest->vaddr); + return 8; - if (pp_connect_ctx(ctx, ib_port, my_dest.psn, rem_dest)) - return 1; - - /* An additional handshake is required *after* moving qp to RTR. - Arbitrarily reuse exch_dest for this purpose. */ - if (servername) { - rem_dest = pp_client_exch_dest(sockfd, &my_dest); - } else { - rem_dest = pp_server_exch_dest(sockfd, &my_dest); - } - - write(sockfd, "done", sizeof "done"); - close(sockfd); + if (pp_open_port(ctx, servername, ib_port, port, &rem_dest)) + return 9; wr = &ctx->wr; ctx->list.addr = (uintptr_t) ctx->buf; @@ -723,10 +721,9 @@ int main(int argc, char *argv[]) qp = ctx->qp; tstamp = malloc(iters * sizeof *tstamp); - if (!tstamp) { perror("malloc"); - return 1; + return 10; } /* Done with setup. Start the test. */ @@ -736,8 +733,8 @@ int main(int argc, char *argv[]) /* Wait till buffer changes. */ if (rcnt < iters && !(scnt < 1 && servername)) { ++rcnt; - while (*poll_buf != (char)rcnt) { - } + while (*poll_buf != (char)rcnt) + ; /* Here the data is already in the physical memory. If we wanted to actually use it, we may need a read memory barrier here. */ @@ -751,7 +748,7 @@ int main(int argc, char *argv[]) if (ibv_post_send(qp, wr, &bad_wr)) { fprintf(stderr, "Couldn't post send: scnt=%d\n", scnt); - return 1; + return 11; } } @@ -765,7 +762,7 @@ int main(int argc, char *argv[]) if (ne < 0) { fprintf(stderr, "poll CQ failed %d\n", ne); - return 1; + return 12; } if (wc.status != IBV_WC_SUCCESS) { fprintf(stderr, "Completion wth error at %s:\n", @@ -774,13 +771,11 @@ int main(int argc, char *argv[]) wc.status, (int) wc.wr_id); fprintf(stderr, "scnt=%d, rcnt=%d, ccnt=%d\n", scnt, rcnt, ccnt); - return 1; + return 13; } } } print_report(&report, iters, tstamp); - - free(tstamp); return 0; } -- MST From mst at mellanox.co.il Wed Jun 15 08:13:53 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 18:13:53 +0300 Subject: [openib-general] Re: cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050610152440.GA10802@esmail.cup.hp.com> References: <20050610050639.GB2833@esmail.cup.hp.com> <20050610152440.GA10802@esmail.cup.hp.com> Message-ID: <20050615151353.GC20041@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: cycles_to_units is incorrect in rdma_lat, rdma_bw. > > On Thu, Jun 09, 2005 at 10:17:33PM -0700, Shirley Ma wrote: > > > Please provide "cat /proc/cpuinfo" output so I can rule out > > > broken firmware and look for something else. > > > > processor : 0 > > cpu : PPC970, altivec supported > > clock : 1600.000000MHz > > revision : 2.2 > > Ah no wonder...every other arch uses "cpu MHz". > > > I modified the get_cpu_mhz() to for PPC. > > rc = sscanf(buf, "clock : %lf", &m); > > Is 1600Mhz correct? I'm assuming it is. > Does the attached patch work for you? > > It will allow PPC to change to "cpu MHz" and it will still work. > Or accomodate other arches that might have cloned PPC code. > > grant I checked that in. Thanks! -- MST From mst at mellanox.co.il Wed Jun 15 08:16:53 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 18:16:53 +0300 Subject: [openib-general] Re: cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050611042727.GB12389@esmail.cup.hp.com> References: <20050610003352.GE12434@esmail.cup.hp.com> <20050610215952.GK10802@esmail.cup.hp.com> <20050611042727.GB12389@esmail.cup.hp.com> Message-ID: <20050615151653.GD20041@mellanox.co.il> Quoting r. Grant Grundler : > Patch below adds "-g" (gettimeofday) and "-M=" parameters > in case someone's firmware is less accurate than HP's. I dont think we shall care about that until this comes about. Maybe there shall be a separate utility to compare get_cycles and gettimeofday results, may be useful for debug. Anyway, if get_cpu_mhz isnt reliable people can just dump raw cycles data and do the math outside the tool. -- MST From mst at mellanox.co.il Wed Jun 15 08:21:12 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 18:21:12 +0300 Subject: [openib-general] Re: Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest In-Reply-To: <20050608234637.GE20219@esmail.cup.hp.com> References: <20050525145727.08DE92283D7@openib.ca.sandia.gov> <20050601232123.GN25321@esmail.cup.hp.com> <20050604213422.GA23153@mellanox.co.il> <20050608002426.GI15489@esmail.cup.hp.com> <20050608055030.GA7809@mellanox.co.il> <20050608234057.GC20219@esmail.cup.hp.com> <20050608234637.GE20219@esmail.cup.hp.com> Message-ID: <20050615152112.GE20041@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: Re: [openib-commits] r2480 - gen2/trunk/src/userspace/perftest > > On Wed, Jun 08, 2005 at 04:40:57PM -0700, Grant Grundler wrote: > > Index: rdma_bw.c > > =================================================================== > > --- rdma_bw.c (revision 2570) > > +++ rdma_bw.c (working copy) > > @@ -61,10 +61,6 @@ > ... > > Sorry - just realized I forgot the: > Signed-off-by: Grant Grundler > > thanks, > grant Applied. Thanks! -- MST From gyakusimei at sw.isao.net Tue Jun 14 04:23:02 2005 From: gyakusimei at sw.isao.net (=?ISO-2022-JP?B?GyRCPi5APjktSH4bKEIg?=) Date: Tue, 14 Jun 2005 20:23:02 +0900 Subject: [openib-general] =?iso-2022-jp?b?UmU6GyRCNls1XiEqGyhC?= Message-ID: <20050614.1123020406@gyakusimei-sw.isao.net> 真奈美さんが今週会える男性を募集しています。 ☆プロフィール☆ 27歳、血液型A型、160cm-46kg、スリーサイズ《86・60・86》写メ有。 『真剣なお願いがあります。貴方の精子をください！子供が出来なくて困ってます。 絶対迷惑はかけませんので中出ししてください。一回10万円で、 妊娠できたら100万円お礼として払います。詳しくはすぐに連絡先を教えます。 出来れば本日中に返事をください。』 http://lovepop.hosting-geomax.jp/secret.php?pr=hs0506 完全無料登録です。サイト内(登録、メールのやり取りすべての利用, 検索〜出会いまで自由解放しております。) 不法な請求は一切ございません。 ※真奈美さんのプロフィールでパスワード【1107】を入力すると 無料で携帯番号・アドレス見ることができます。 これは貴方様限定のお知らせですので他者には絶対に教えないで下さい！ http://lovepop.hosting-geomax.jp/secret.php?pr=hs0506 From roland at topspin.com Wed Jun 15 08:54:52 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 08:54:52 -0700 Subject: [openib-general] Re: How about ib_send_page() ? In-Reply-To: <20050615114439.GA20041@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 15 Jun 2005 14:44:39 +0300") References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <528y2di70f.fsf@topspin.com> <20050615114439.GA20041@mellanox.co.il> Message-ID: <52fyvjhayb.fsf@topspin.com> Michael> Hi, Roland! I wonder what did you use for profiling? oprofile. - R. From Thomas.Talpey at netapp.com Wed Jun 15 09:15:20 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Wed, 15 Jun 2005 12:15:20 -0400 Subject: [openib-general] kdaptest wedges server In-Reply-To: <1118796272.17668.111.camel@localhost> References: <91DB792C7985D411BEC300B40080D29CC35C72@mtvex01.mtv.mtl.com> <1118708297.17672.32.camel@localhost> <1118761342.12164.0.camel@duffman> <1118778050.17668.62.camel@localhost> <1118784969.4748.88.camel@localhost.localdomain> <1118786683.17672.78.camel@localhost> <1118796272.17668.111.camel@localhost> Message-ID: <6.2.1.2.2.20050615100657.04202eb0@exnane01.nane.netapp.com> At 08:44 PM 6/14/2005, Josh England wrote: >On another note, has anyone got a *bare-bones* kdapl/udapl example with >connection setup and RDMA send/receive. Define bare bones? If you go to the Sourceforge version of dapl (http://sourceforge.net/projects/dapl) you can take a look at the dat_echo directory. Also, the kdapltest code has some pretty basic examples for kdapl, and regular dapltest for udapl. Tom. From roland at topspin.com Wed Jun 15 10:12:35 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 10:12:35 -0700 Subject: [openib-general] UC support committed to svn In-Reply-To: <1118786424.4748.137.camel@localhost.localdomain> (Hal Rosenstock's message of "14 Jun 2005 18:00:25 -0400") References: <52mzpskd5z.fsf@topspin.com> <1118786424.4748.137.camel@localhost.localdomain> Message-ID: <52wtovfssc.fsf@topspin.com> OK, I added a uc_pingpong example, and renamed the original pingpong to rc_pingpong for consistency. - R. From tduffy at sun.com Wed Jun 15 10:11:52 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 15 Jun 2005 10:11:52 -0700 Subject: [openib-general] [PATCH] kDAPL: anal lawyer update Message-ID: <1118855512.29970.6.camel@duffman> James, the Sun lawyers are getting anal and want any code that I have modified substantially on OpenIB to have a Sun copyright on it. I have gone through the DAPL stuff and grepped the svn logs for sun.com and updated those files with a Sun copyright. Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_client.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_performance_client.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_transaction_stats.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_stats.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_transaction_stats.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_cnxn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_cnxn.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_cnxn.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_server.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_server.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_thread.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_thread.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_thread.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_test_data.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_data.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_test_data.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_mem.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_mem.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_mem.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_limit.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_server_info.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_server_info.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_server_info.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_performance_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_server.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_performance_server.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_connmgt.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_memlist.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_memlist.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_memlist.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_util.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_endpoint.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_util.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_pz.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_hwconn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (revision 2622) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_client_info.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_client_info.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_client_info.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_server_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_server_cmd.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_server_cmd.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_tdep.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_tdep.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_global.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_global.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_global.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_limit_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_limit_cmd.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_limit_cmd.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_fft_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_cmd.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_fft_cmd.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_quit_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_quit_cmd.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_quit_cmd.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_test_data.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_test_data.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_test_data.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_transaction_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_cmd.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_transaction_cmd.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_bpool.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_fft_util.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_performance_stats.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_stats.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_performance_stats.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_server_info.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_server_info.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_server_info.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_params.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_params.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_params.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_performance_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_cmd.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_performance_cmd.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/include/dapl_common.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_common.h (revision 2622) +++ linux-kernel/test/dapltest/include/dapl_common.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_qos_util.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_qos_util.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_qos_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_netaddr.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_netaddr.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_netaddr.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_params.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_params.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_params.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_test_data.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_test_data.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_test_data.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/cmd/dapl_server_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_server_cmd.c (revision 2622) +++ linux-kernel/test/dapltest/cmd/dapl_server_cmd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2622) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2622) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h (revision 2622) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h (revision 2622) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/common/dapl_endian.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_endian.c (revision 2622) +++ linux-kernel/test/dapltest/common/dapl_endian.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/common/dapl_global.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_global.c (revision 2622) +++ linux-kernel/test/dapltest/common/dapl_global.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c (revision 2622) +++ linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c (revision 2622) +++ linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/kdapl/kdapl_module.c =================================================================== --- linux-kernel/test/dapltest/kdapl/kdapl_module.c (revision 2622) +++ linux-kernel/test/dapltest/kdapl/kdapl_module.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2622) +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -1,4 +1,7 @@ -/* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. * +/* + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * * This Software is licensed under one of the following licenses: * * 1) under the terms of the "Common Public License 1.0" a copy of which is Index: linux-kernel/dat-provider/dapl_provider.h =================================================================== --- linux-kernel/dat-provider/dapl_provider.h (revision 2622) +++ linux-kernel/dat-provider/dapl_provider.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_openib_cm.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.h (revision 2622) +++ linux-kernel/dat-provider/dapl_openib_cm.h (working copy) @@ -3,6 +3,7 @@ * Copyright (c) 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004-2005, Mellanox Technologies, Inc. All rights reserved. * Copyright (c) 2003 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_srq.c =================================================================== --- linux-kernel/dat-provider/dapl_srq.c (revision 2622) +++ linux-kernel/dat-provider/dapl_srq.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel/dat-provider/dapl_cookie.c (revision 2622) +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2622) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 1999-2005, Mellanox Technologies, Inc. All rights reserved. * Copyright (c) 2005 Voltaire Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2622) +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 1999-2005, Mellanox Technologies, Inc. All rights reserved. * Copyright (c) 2005 Voltaire Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_cookie.h =================================================================== --- linux-kernel/dat-provider/dapl_cookie.h (revision 2622) +++ linux-kernel/dat-provider/dapl_cookie.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_openib_util.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.h (revision 2622) +++ linux-kernel/dat-provider/dapl_openib_util.h (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 1999-2005, Mellanox Technologies, Inc. All rights reserved. * Copyright (c) 2005 Voltaire Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_ring_buffer_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ring_buffer_util.c (revision 2622) +++ linux-kernel/dat-provider/dapl_ring_buffer_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_openib_dto.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_dto.h (revision 2622) +++ linux-kernel/dat-provider/dapl_openib_dto.h (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 1999-2005, Mellanox Technologies, Inc. All rights reserved. * Copyright (c) 2005 Voltaire Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_hca_util.c =================================================================== --- linux-kernel/dat-provider/dapl_hca_util.c (revision 2622) +++ linux-kernel/dat-provider/dapl_hca_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_pz.c =================================================================== --- linux-kernel/dat-provider/dapl_pz.c (revision 2622) +++ linux-kernel/dat-provider/dapl_pz.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_ia.c =================================================================== --- linux-kernel/dat-provider/dapl_ia.c (revision 2622) +++ linux-kernel/dat-provider/dapl_ia.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_lmr.c =================================================================== --- linux-kernel/dat-provider/dapl_lmr.c (revision 2622) +++ linux-kernel/dat-provider/dapl_lmr.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_ring_buffer_util.h =================================================================== --- linux-kernel/dat-provider/dapl_ring_buffer_util.h (revision 2622) +++ linux-kernel/dat-provider/dapl_ring_buffer_util.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_hca_util.h =================================================================== --- linux-kernel/dat-provider/dapl_hca_util.h (revision 2622) +++ linux-kernel/dat-provider/dapl_hca_util.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_ia.h =================================================================== --- linux-kernel/dat-provider/dapl_ia.h (revision 2622) +++ linux-kernel/dat-provider/dapl_ia.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_rmr.c =================================================================== --- linux-kernel/dat-provider/dapl_rmr.c (revision 2622) +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2622) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_cr.c =================================================================== --- linux-kernel/dat-provider/dapl_cr.c (revision 2622) +++ linux-kernel/dat-provider/dapl_cr.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_ep.h =================================================================== --- linux-kernel/dat-provider/dapl_ep.h (revision 2622) +++ linux-kernel/dat-provider/dapl_ep.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_cr.h =================================================================== --- linux-kernel/dat-provider/dapl_cr.h (revision 2622) +++ linux-kernel/dat-provider/dapl_cr.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_util.c =================================================================== --- linux-kernel/dat-provider/dapl_util.c (revision 2622) +++ linux-kernel/dat-provider/dapl_util.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2622) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_util.h =================================================================== --- linux-kernel/dat-provider/dapl_util.h (revision 2622) +++ linux-kernel/dat-provider/dapl_util.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_sp.c =================================================================== --- linux-kernel/dat-provider/dapl_sp.c (revision 2622) +++ linux-kernel/dat-provider/dapl_sp.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_hash.c =================================================================== --- linux-kernel/dat-provider/dapl_hash.c (revision 2622) +++ linux-kernel/dat-provider/dapl_hash.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_evd.h =================================================================== --- linux-kernel/dat-provider/dapl_evd.h (revision 2622) +++ linux-kernel/dat-provider/dapl_evd.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2622) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_sp.h =================================================================== --- linux-kernel/dat-provider/dapl_sp.h (revision 2622) +++ linux-kernel/dat-provider/dapl_sp.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_hash.h =================================================================== --- linux-kernel/dat-provider/dapl_hash.h (revision 2622) +++ linux-kernel/dat-provider/dapl_hash.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_provider.c =================================================================== --- linux-kernel/dat-provider/dapl_provider.c (revision 2622) +++ linux-kernel/dat-provider/dapl_provider.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2622) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -3,6 +3,7 @@ * Copyright (c) 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004-2005, Mellanox Technologies, Inc. All rights reserved. * Copyright (c) 2003 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat/dictionary.c =================================================================== --- linux-kernel/dat/dictionary.c (revision 2622) +++ linux-kernel/dat/dictionary.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat/dr.c =================================================================== --- linux-kernel/dat/dr.c (revision 2622) +++ linux-kernel/dat/dr.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat/core.c =================================================================== --- linux-kernel/dat/core.c (revision 2622) +++ linux-kernel/dat/core.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2622) +++ linux-kernel/dat/api.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat/dr.h =================================================================== --- linux-kernel/dat/dr.h (revision 2622) +++ linux-kernel/dat/dr.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat/dictionary.h =================================================================== --- linux-kernel/dat/dictionary.h (revision 2622) +++ linux-kernel/dat/dictionary.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2622) +++ linux-kernel/dat/dat.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * Index: linux-kernel/dat/core.h =================================================================== --- linux-kernel/dat/core.h (revision 2622) +++ linux-kernel/dat/core.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This Software is licensed under one of the following licenses: * From iod00d at hp.com Wed Jun 15 10:30:58 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 15 Jun 2005 10:30:58 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050615150440.GB20041@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050615150440.GB20041@mellanox.co.il> Message-ID: <20050615173058.GB4379@esmail.cup.hp.com> On Wed, Jun 15, 2005 at 06:04:40PM +0300, Michael S. Tsirkin wrote: > OK, I accepted most of it. Thanks! Welcome! Excellent! > There were some issues that I fixed, I also made some other small > cleanups. > > One thing I skipped is the pp_connect_sock consolidation: I like to keep the > split between client and server at the top level. Connect routines have some > code duplication but I dont like spreading if (servername) all around them, > and I dont mind duplication much since this is standard socket stuff. I moved the servername stuff down one level for several reasons: 1) reduce the size of main() to < 100 lines. Why? It's supposed to be an example of how to use verbs. The main routine should outline what has to happen, not clutter with test setup stuff. 2) I working towards making the server side a daemon. ie move stuff out of main: o parse commandline parameters o loop o get parameters via socket o set up ctx o run test_lat (or test_bw maybe eventually) o tear down ctx (e.g release qp, cq, etc) o report results o clear "global" state (ie free(tstamp), etc) > Grant, I'm not sure I understand the reason for > > o enumerate return values in main() > bit but I went along with it. Why do you like it? In general I like unique return values from programs when they fail. Makes debugging alot easier. e.g. rdma_lat just spews the usage msg when a bad parameter is passed to it. It may not be obvious which parameter gets rejected depending on how complex the command line is. rdma_lat is otherwise generally pretty good about printing a somewhat unique (that's different than "meaningful") error messages. > The patch also added free(tstamp) in a couple of places, but since > we dont bother to free resources (qp, cq, ...) and since this was in > the test part which must be cristal clear, I skipped this bit. Yeah - this needs to be fixed. I was looking at what needs to happen for teardown so the test can iterate. But I'm clueless at this point what needs to happen. > If we ever do a proper cleanup, its probably a good idea to goto > outside the loop and handle errors there. Yes - agree. Can you add that? It's not obvious to me right now what needs to be cleaned up. ibv_pingpong wasn't helpful in this case either. :^( > Here's what I ended up checking in: looks good - I'll have the next patch ready later today. thanks! grant From libor at topspin.com Wed Jun 15 10:36:35 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 15 Jun 2005 10:36:35 -0700 Subject: [openib-general] Re: [PATCH] sdp_post_rdma_iocb_snk check iocb In-Reply-To: <20050615094408.GW21081@mellanox.co.il>; from mst@mellanox.co.il on Wed, Jun 15, 2005 at 12:44:08PM +0300 References: <20050615094408.GW21081@mellanox.co.il> Message-ID: <20050615103635.A7232@topspin.com> On Wed, Jun 15, 2005 at 12:44:08PM +0300, Michael S. Tsirkin wrote: > In sdp_post_rdma_iocb_snk, if sdp_iocb_q_get_head returns NULL, iocb is then > used without checking. Michael, This is done in a number of places, and the check for NULL iocb should not be necessary. Earlier in each function where it's done, we perform a look at the head of the queue and only continue if there is an iocb at the head of the queue: iocb = sdp_iocb_q_look(&conn->r_pend); The reason I did it this way is to avoid having to return the iocb back into the same queue if it is not entirely consumed or on a recoverable error, depending on the function. -Libor > Index: sdp_recv.c > =================================================================== > --- sdp_recv.c (revision 2608) > +++ sdp_recv.c (working copy) > @@ -411,7 +411,7 @@ static int sdp_post_rdma_iocb_snk(struct > * queue IOCB > */ > iocb = sdp_iocb_q_get_head(&conn->r_pend); > - if (result < 0) { > + if (!iocb) { > sdp_dbg_warn(conn, "read IOCB missing from pending table <%d>", > sdp_iocb_q_size(&conn->r_pend)); > goto release; > > -- > MST From iod00d at hp.com Wed Jun 15 10:40:24 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 15 Jun 2005 10:40:24 -0700 Subject: [openib-general] Re: cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050615151653.GD20041@mellanox.co.il> References: <20050610003352.GE12434@esmail.cup.hp.com> <20050610215952.GK10802@esmail.cup.hp.com> <20050611042727.GB12389@esmail.cup.hp.com> <20050615151653.GD20041@mellanox.co.il> Message-ID: <20050615174024.GC4379@esmail.cup.hp.com> On Wed, Jun 15, 2005 at 06:16:53PM +0300, Michael S. Tsirkin wrote: > Quoting r. Grant Grundler : > > Patch below adds "-g" (gettimeofday) and "-M=" parameters > > in case someone's firmware is less accurate than HP's. > > I dont think we shall care about that until this comes about. Shirley Ma (IBM) has asserted that was the case for machines she was testing on. That's what started this email thread. Shirley, did the patch to get_clock.c fix the problem? > Maybe there shall be a separate utility to compare > get_cycles and gettimeofday results, may be useful for debug. > > Anyway, if get_cpu_mhz isnt reliable people can just dump raw > cycles data and do the math outside the tool. Yeah, that's a valid alternative. And the extra code doesn't make this a better example. OTOH, it's more convenient to have the math integrated in the test - one less step that can introduce errors. Any one else have an opinion? Shirley? thanks, grant From tduffy at sun.com Wed Jun 15 10:39:34 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 15 Jun 2005 10:39:34 -0700 Subject: [openib-general] [PATCH] IPoIB: anal lawyer update Message-ID: <1118857174.29970.8.camel@duffman> Roland, I have grepped the ipoib logs for any files I have changed and added a Sun copyright. Signed-off-by: Tom Duffy Index: linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_main.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_main.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_main.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_multicast.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_multicast.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_multicast.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_ib.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_ib.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/ipoib/ipoib_ib.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU From tduffy at sun.com Wed Jun 15 10:41:15 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 15 Jun 2005 10:41:15 -0700 Subject: [openib-general] [PATCH] mthca: anal lawyer update Message-ID: <1118857275.3845.1.camel@duffman> Roland, I have grepped the mthca logs for any files I have changed and added a Sun copyright. Signed-off-by: Tom Duffy Index: linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_dev.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_dev.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_dev.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_main.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_main.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_main.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_provider.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_provider.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_provider.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_cq.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_cq.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_cq.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_av.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_av.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_av.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_doorbell.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_doorbell.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/hw/mthca/mthca_doorbell.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU From roland at topspin.com Wed Jun 15 10:42:27 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 10:42:27 -0700 Subject: [openib-general] Re: [PATCH] IPoIB: anal lawyer update In-Reply-To: <1118857174.29970.8.camel@duffman> (Tom Duffy's message of "Wed, 15 Jun 2005 10:39:34 -0700") References: <1118857174.29970.8.camel@duffman> Message-ID: <52ll5bfrek.fsf@topspin.com> Thanks, applied. From tduffy at sun.com Wed Jun 15 10:43:40 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 15 Jun 2005 10:43:40 -0700 Subject: [openib-general] [PATCH] core: anal lawyer update Message-ID: <1118857420.3845.4.camel@duffman> Sean, I have grepped the core and include logs for any files I have changed and added a Sun copyright. Signed-off-by: Tom Duffy Index: linux-2.6.11-openib/drivers/infiniband/core/agent.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/agent.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/agent.c (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004, 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004, 2005 Topspin Corporation. All rights reserved. * Copyright (c) 2004, 2005 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/user_mad.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/user_mad.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/user_mad.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. * Copyright (c) 2005 Voltaire, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/device.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/device.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/device.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/cm.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/cm.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/cm.c (working copy) @@ -2,6 +2,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/cache.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/cache.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/cache.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/packer.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/packer.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/packer.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/mad_priv.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/mad_priv.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/mad_priv.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005, Voltaire, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/sysfs.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/sysfs.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/sysfs.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/fmr_pool.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/fmr_pool.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/fmr_pool.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/ud_header.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/ud_header.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/ud_header.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/verbs.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/verbs.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/verbs.c (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/smi.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/smi.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/smi.c (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004, 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004, 2005 Topspin Corporation. All rights reserved. * Copyright (c) 2004, 2005 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/core/ping.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/ping.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/core/ping.c (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004, 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004, 2005 Topspin Corporation. All rights reserved. * Copyright (c) 2004, 2005 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/include/ib_cm.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/include/ib_cm.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/include/ib_cm.h (working copy) @@ -2,6 +2,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/include/ib_verbs.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/include/ib_verbs.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/include/ib_verbs.h (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/include/ib_cache.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/include/ib_cache.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/include/ib_cache.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/include/ib_fmr_pool.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/include/ib_fmr_pool.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/include/ib_fmr_pool.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU From roland at topspin.com Wed Jun 15 10:43:39 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 10:43:39 -0700 Subject: [openib-general] Re: [PATCH] mthca: anal lawyer update In-Reply-To: <1118857275.3845.1.camel@duffman> (Tom Duffy's message of "Wed, 15 Jun 2005 10:41:15 -0700") References: <1118857275.3845.1.camel@duffman> Message-ID: <52fyvjfrck.fsf@topspin.com> thanks, applied. From tduffy at sun.com Wed Jun 15 10:45:45 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 15 Jun 2005 10:45:45 -0700 Subject: [openib-general] [PATCH] SDP: anal lawyer update Message-ID: <1118857545.3845.6.camel@duffman> Libor, I have added a Sun copyright to all the SDP files. Please apply. Signed-off-by: Tom Duffy Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_queue.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_queue.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_queue.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_main.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_main.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_main.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_write.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_write.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_write.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_sock.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_sock.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_sock.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proto.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proto.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proto.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_read.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_read.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_read.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_msgs.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_msgs.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_msgs.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_send.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_send.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_send.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_actv.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_advt.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_advt.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_advt.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_recv.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_advt.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_advt.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_advt.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_pass.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_sent.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_sent.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_sent.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_event.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_event.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_event.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_dev.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_queue.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_queue.c (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_queue.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.h (revision 2624) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU From tduffy at sun.com Wed Jun 15 10:49:09 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 15 Jun 2005 10:49:09 -0700 Subject: [openib-general] [Fwd: Returned mail: see transcript for details] Message-ID: <1118857749.3845.8.camel@duffman> Ha ha Intel. -------------- next part -------------- An embedded message was scrubbed... From: Mail Delivery Subsystem Subject: Returned mail: see transcript for details Date: Wed, 15 Jun 2005 11:43:43 -0600 (MDT) Size: 16819 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From atorrez at lanl.gov Wed Jun 15 11:41:58 2005 From: atorrez at lanl.gov (Alfred Torrez) Date: Wed, 15 Jun 2005 12:41:58 -0600 Subject: [openib-general] Rev 2617 Compile Error on 2.6.12-rc4 Message-ID: <6.0.0.22.2.20050615122926.01caf760@cic-mail.lanl.gov> I am trying to compile rev 2617 on 2.6.12-rc4. I applied the Tom Duffy's SDP patch for compiling on 2.6.12-rc4 (early May) but it looks like at least one file does not exist anymore (sdp_wall.c) and patch is failing on sdp_pass.c. Does a newer patch exist, or do I need to apply additional patches? I successfully built ver. 2386 a month or so ago on 2.6.12-rc4. Thanks, Alfred From xma at us.ibm.com Wed Jun 15 11:55:07 2005 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 15 Jun 2005 11:55:07 -0700 Subject: [openib-general] Re: cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: <20050615174024.GC4379@esmail.cup.hp.com> Message-ID: > in case someone's firmware is less accurate than HP's. > Shirley, did the patch to get_clock.c fix the problem? Cycles is not equal to the clock rate. If you have a processor with internal doubling or pipelining or whatever, the clock rate isn't the same. > Maybe there shall be a separate utility to compare > get_cycles and gettimeofday results, may be useful for debug. > > Anyway, if get_cpu_mhz isnt reliable people can just dump raw > cycles data and do the math outside the tool. Agree if the rdma_lat, and rdma_bw are only used for debugging/regression testing not for performance measurement. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tduffy at sun.com Wed Jun 15 12:29:48 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 15 Jun 2005 12:29:48 -0700 Subject: [openib-general] Rev 2617 Compile Error on 2.6.12-rc4 In-Reply-To: <6.0.0.22.2.20050615122926.01caf760@cic-mail.lanl.gov> References: <6.0.0.22.2.20050615122926.01caf760@cic-mail.lanl.gov> Message-ID: <1118863788.10140.1.camel@duffman> On Wed, 2005-06-15 at 12:41 -0600, Alfred Torrez wrote: > I am trying to compile rev 2617 on 2.6.12-rc4. I applied the Tom Duffy's > SDP patch for compiling on 2.6.12-rc4 (early May) but it looks like at > least one file does not exist anymore (sdp_wall.c) and patch is failing on > sdp_pass.c. Does a newer patch exist, or do I need to apply additional > patches? I successfully built ver. 2386 a month or so ago on 2.6.12-rc4. Yes, please apply version 6 of the patch. http://openib.org/pipermail/openib-general/2005-June/007567.html -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mst at mellanox.co.il Wed Jun 15 13:01:19 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 23:01:19 +0300 Subject: [openib-general] Re: [PATCH] sdp_post_rdma_iocb_snk check iocb In-Reply-To: <20050615103635.A7232@topspin.com> References: <20050615094408.GW21081@mellanox.co.il> <20050615103635.A7232@topspin.com> Message-ID: <20050615200119.GB21860@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: [PATCH] sdp_post_rdma_iocb_snk check iocb > > On Wed, Jun 15, 2005 at 12:44:08PM +0300, Michael S. Tsirkin wrote: > > In sdp_post_rdma_iocb_snk, if sdp_iocb_q_get_head returns NULL, iocb is then > > used without checking. > > Michael, > > This is done in a number of places, and the check for NULL iocb should > not be necessary. Earlier in each function where it's done, we perform a > look at the head of the queue and only continue if there is an iocb at > the head of the queue: > > iocb = sdp_iocb_q_look(&conn->r_pend); > > The reason I did it this way is to avoid having to return the iocb back > into the same queue if it is not entirely consumed or on a recoverable > error, depending on the function. > > -Libor Okay, but the test if (result < 0) is clearly bogus since result isnt set? > > Index: sdp_recv.c > > =================================================================== > > --- sdp_recv.c (revision 2608) > > +++ sdp_recv.c (working copy) > > @@ -411,7 +411,7 @@ static int sdp_post_rdma_iocb_snk(struct > > * queue IOCB > > */ > > iocb = sdp_iocb_q_get_head(&conn->r_pend); > > - if (result < 0) { > > + if (!iocb) { > > sdp_dbg_warn(conn, "read IOCB missing from pending table <%d>", > > sdp_iocb_q_size(&conn->r_pend)); > > goto release; > > > > -- > > MST > -- MST From mst at mellanox.co.il Wed Jun 15 13:18:05 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 15 Jun 2005 23:18:05 +0300 Subject: [openib-general] Re: cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: References: <20050615174024.GC4379@esmail.cup.hp.com> Message-ID: <20050615201805.GC21860@mellanox.co.il> Quoting r. Shirley Ma : > Subject: Re: cycles_to_units is incorrect in rdma_lat, rdma_bw. > > > in case someone's firmware is less accurate than HP's. > > Shirley, did the patch to get_clock.c fix the problem? > > Cycles is not equal to the clock rate. If you have a processor with internal > doubling or pipelining or whatever, the clock rate isn't the same. The name get_cycles was chosen to match linux kernel API. (get_cycles / cpu_mhz) shall return time in microseconds. If it doesnt get_cycles can just be fixed/renamed. Do you see an actual machine where thats not the case? > > Maybe there shall be a separate utility to compare > > get_cycles and gettimeofday results, may be useful for debug. > > > > Anyway, if get_cpu_mhz isnt reliable people can just dump raw > > cycles data and do the math outside the tool. > > Agree if the rdma_lat, and rdma_bw are only used for debugging/regression > testing not for performance measurement. Its just a work-around in case someone has a problem. But no one complained yet. -- MST From libor at topspin.com Wed Jun 15 13:29:20 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 15 Jun 2005 13:29:20 -0700 Subject: [openib-general] Re: [PATCH] sdp_post_rdma_iocb_snk check iocb In-Reply-To: <20050615200119.GB21860@mellanox.co.il>; from mst@mellanox.co.il on Wed, Jun 15, 2005 at 11:01:19PM +0300 References: <20050615094408.GW21081@mellanox.co.il> <20050615103635.A7232@topspin.com> <20050615200119.GB21860@mellanox.co.il> Message-ID: <20050615132920.B7232@topspin.com> On Wed, Jun 15, 2005 at 11:01:19PM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > Subject: Re: [PATCH] sdp_post_rdma_iocb_snk check iocb > > > > On Wed, Jun 15, 2005 at 12:44:08PM +0300, Michael S. Tsirkin wrote: > > > In sdp_post_rdma_iocb_snk, if sdp_iocb_q_get_head returns NULL, iocb > > > is then used without checking. > > > > This is done in a number of places, and the check for NULL iocb should > > not be necessary. Earlier in each function where it's done, we perform a > > look at the head of the queue and only continue if there is an iocb at > > the head of the queue: > > > > iocb = sdp_iocb_q_look(&conn->r_pend); > > > > The reason I did it this way is to avoid having to return the iocb back > > into the same queue if it is not entirely consumed or on a recoverable > > error, depending on the function. > > Okay, but the test if (result < 0) is clearly bogus since result > isnt set? Yes, that's true. In the patch I sent out yesterday and committed today that is resolved. -Libor From roland at topspin.com Wed Jun 15 13:43:16 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 13:43:16 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API Message-ID: <52slzje4gr.fsf@topspin.com> Here's a first stab at the changes to the API in required to support shared receive queues (SRQs). SRQs are described in version 1.2 of the IBA spec; specifically, sections 10.2.9 and 11.2.3 describe the semantics and verbs interface. I did diverge from the spec in that instead of a single "Post Receive Request" verb, there are separate ib_post_recv() and ib_post_srq_recv() functions. This avoids complications from having a polymorphic function that takes either a QP or an SRQ, and matches the interface Mellanox used in VAPI. Any comments? Thanks, Roland --- infiniband/include/ib_verbs.h (revision 2631) +++ infiniband/include/ib_verbs.h (working copy) @@ -261,6 +261,7 @@ struct ib_event { union { struct ib_cq *cq; struct ib_qp *qp; + struct ib_srq *srq; u8 port_num; } element; enum ib_event_type event; @@ -381,6 +382,17 @@ enum ib_cq_notify { IB_CQ_NEXT_COMP }; +enum ib_srq_attr_mask { + IB_SRQ_MAX_WR = 1 << 0, + IB_SRQ_LIMIT = 1 << 1, +}; + +struct ib_srq_attr { + u32 max_wr; + u32 max_sge; + u32 srq_limit; +}; + struct ib_qp_cap { u32 max_send_wr; u32 max_recv_wr; @@ -818,6 +830,17 @@ struct ib_device { int (*query_ah)(struct ib_ah *ah, struct ib_ah_attr *ah_attr); int (*destroy_ah)(struct ib_ah *ah); + struct ib_srq * (*create_srq)(struct ib_pd *pd, + struct ib_srq_attr *srq_attr); + int (*modify_srq)(struct ib_srp *srq, + struct ib_srp_attr *srq_attr, + enum ib_srq_attr_mask srq_attr_mask); + int (*query_srq)(struct ib_srq *srq, + struct ib_srq_attr *srq_attr); + int (*destroy_srq)(struct ib_srq *srq); + int (*post_srq_recv)(struct ib_srq *srq, + struct ib_recv_wr *recv_wr, + struct ib_recv_wr **bad_recv_wr); struct ib_qp * (*create_qp)(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr, const void __user *udata, int udatalen); @@ -1021,6 +1044,68 @@ int ib_query_ah(struct ib_ah *ah, struct int ib_destroy_ah(struct ib_ah *ah); /** + * ib_create_srq - Creates a SRQ associated with the specified protection + * domain. + * @pd: The protection domain associated with the SRQ. + * @srq_attr: A list of initial attributes required to create the SRQ. + * + * srq_attr->max_wr and srq_attr->max_sge are read the determine the + * requested size of the SRQ, and set to the actual values allocated + * on return. If ib_create_srq() succeeds, then max_wr and max_sge + * will always be at least as large as the requested values. + */ +struct ib_srq *ib_create_srq(struct ib_pd *pd, + struct ib_srq_attr *srq_attr); + +/** + * ib_modify_srq - Modifies the attributes for the specified SRQ and then + * transitions the SRQ to the given state. + * @srq: The SRQ to modify. + * @srq_attr: On input, specifies the SRQ attributes to modify. On output, + * the current values of selected SRQ attributes are returned. + * @srq_attr_mask: A bit-mask used to specify which attributes of the SRQ + * are being modified. + * + * The mask may contain IB_SRQ_MAX_WR to resize the SRQ and/or + * IB_SRQ_LIMIT to set the SRQ's limit and request notification when + * the number of receives queued drops below the limit. + */ +int ib_modify_srq(struct ib_srq *srq, + struct ib_srq_attr *srq_attr, + enum ib_srq_attr_mask srq_attr_mask); + +/** + * ib_query_srq - Returns the attribute list and current values for the + * specified SRQ. + * @srq: The SRQ to query. + * @srq_attr: The attributes of the specified SRQ. + */ +int ib_query_srq(struct ib_srq *srq, + struct ib_srq_attr *srq_attr, + int srq_attr_mask, + struct ib_srq_init_attr *srq_init_attr); + +/** + * ib_destroy_srq - Destroys the specified SRQ. + * @srq: The SRQ to destroy. + */ +int ib_destroy_srq(struct ib_srq *srq); + +/** + * ib_post_srq_recv - Posts a list of work requests to the specified SRQ. + * @srq: The SRQ to post the work request on. + * @recv_wr: A list of work requests to post on the receive queue. + * @bad_recv_wr: On an immediate failure, this parameter will reference + * the work request that failed to be posted on the QP. + */ +static inline int ib_post_srq_recv(struct ib_srq *srq, + struct ib_recv_wr *recv_wr, + struct ib_recv_wr **bad_recv_wr) +{ + return srq->device->post_srq_recv(srq, recv_wr, bad_recv_wr); +} + +/** * ib_create_qp - Creates a QP associated with the specified protection * domain. * @pd: The protection domain associated with the QP. From johnip at sgi.com Wed Jun 15 13:47:15 2005 From: johnip at sgi.com (John Partridge) Date: Wed, 15 Jun 2005 15:47:15 -0500 Subject: [openib-general] SDP debug Message-ID: <42B093D3.8020609@sgi.com> I am trying to get SDP working on SGI Altix. I know that the InfiniBand is working as I have ipoib and ibv_pingpong etc all working fine. My question is how do I prove that an nttcp (or whatever I use) is using SDP rather than ipoib ? I have used strace and can see the libsdp.conf file being read. So I know I am going through libsdp.so (as defined in my LD_PRELOAD). Is there some DEBUG or counter I can use to tell me what is going on ? Thanks John John Partridge Silicon Graphics Inc Tel: 651-683-3428 Vnet: 233-3428 E-Mail: johnip at sgi.com From roland at topspin.com Wed Jun 15 13:53:18 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 13:53:18 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API In-Reply-To: <52slzje4gr.fsf@topspin.com> (Roland Dreier's message of "Wed, 15 Jun 2005 13:43:16 -0700") References: <52slzje4gr.fsf@topspin.com> Message-ID: <52mzpre401.fsf@topspin.com> Here's a second stab with a bunch of typos corrected so it actually builds... (Little programming tip for all the kids out there: test compile the same tree you are editing) - R. --- infiniband/include/ib_verbs.h (revision 2631) +++ infiniband/include/ib_verbs.h (working copy) @@ -261,6 +261,7 @@ struct ib_event { union { struct ib_cq *cq; struct ib_qp *qp; + struct ib_srq *srq; u8 port_num; } element; enum ib_event_type event; @@ -381,6 +382,17 @@ enum ib_cq_notify { IB_CQ_NEXT_COMP }; +enum ib_srq_attr_mask { + IB_SRQ_MAX_WR = 1 << 0, + IB_SRQ_LIMIT = 1 << 1, +}; + +struct ib_srq_attr { + u32 max_wr; + u32 max_sge; + u32 srq_limit; +}; + struct ib_qp_cap { u32 max_send_wr; u32 max_recv_wr; @@ -818,6 +830,17 @@ struct ib_device { int (*query_ah)(struct ib_ah *ah, struct ib_ah_attr *ah_attr); int (*destroy_ah)(struct ib_ah *ah); + struct ib_srq * (*create_srq)(struct ib_pd *pd, + struct ib_srq_attr *srq_attr); + int (*modify_srq)(struct ib_srq *srq, + struct ib_srq_attr *srq_attr, + enum ib_srq_attr_mask srq_attr_mask); + int (*query_srq)(struct ib_srq *srq, + struct ib_srq_attr *srq_attr); + int (*destroy_srq)(struct ib_srq *srq); + int (*post_srq_recv)(struct ib_srq *srq, + struct ib_recv_wr *recv_wr, + struct ib_recv_wr **bad_recv_wr); struct ib_qp * (*create_qp)(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr, const void __user *udata, int udatalen); @@ -1021,6 +1044,66 @@ int ib_query_ah(struct ib_ah *ah, struct int ib_destroy_ah(struct ib_ah *ah); /** + * ib_create_srq - Creates a SRQ associated with the specified protection + * domain. + * @pd: The protection domain associated with the SRQ. + * @srq_attr: A list of initial attributes required to create the SRQ. + * + * srq_attr->max_wr and srq_attr->max_sge are read the determine the + * requested size of the SRQ, and set to the actual values allocated + * on return. If ib_create_srq() succeeds, then max_wr and max_sge + * will always be at least as large as the requested values. + */ +struct ib_srq *ib_create_srq(struct ib_pd *pd, + struct ib_srq_attr *srq_attr); + +/** + * ib_modify_srq - Modifies the attributes for the specified SRQ and then + * transitions the SRQ to the given state. + * @srq: The SRQ to modify. + * @srq_attr: On input, specifies the SRQ attributes to modify. On output, + * the current values of selected SRQ attributes are returned. + * @srq_attr_mask: A bit-mask used to specify which attributes of the SRQ + * are being modified. + * + * The mask may contain IB_SRQ_MAX_WR to resize the SRQ and/or + * IB_SRQ_LIMIT to set the SRQ's limit and request notification when + * the number of receives queued drops below the limit. + */ +int ib_modify_srq(struct ib_srq *srq, + struct ib_srq_attr *srq_attr, + enum ib_srq_attr_mask srq_attr_mask); + +/** + * ib_query_srq - Returns the attribute list and current values for the + * specified SRQ. + * @srq: The SRQ to query. + * @srq_attr: The attributes of the specified SRQ. + */ +int ib_query_srq(struct ib_srq *srq, + struct ib_srq_attr *srq_attr); + +/** + * ib_destroy_srq - Destroys the specified SRQ. + * @srq: The SRQ to destroy. + */ +int ib_destroy_srq(struct ib_srq *srq); + +/** + * ib_post_srq_recv - Posts a list of work requests to the specified SRQ. + * @srq: The SRQ to post the work request on. + * @recv_wr: A list of work requests to post on the receive queue. + * @bad_recv_wr: On an immediate failure, this parameter will reference + * the work request that failed to be posted on the QP. + */ +static inline int ib_post_srq_recv(struct ib_srq *srq, + struct ib_recv_wr *recv_wr, + struct ib_recv_wr **bad_recv_wr) +{ + return srq->device->post_srq_recv(srq, recv_wr, bad_recv_wr); +} + +/** * ib_create_qp - Creates a QP associated with the specified protection * domain. * @pd: The protection domain associated with the QP. From iod00d at hp.com Wed Jun 15 14:14:21 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 15 Jun 2005 14:14:21 -0700 Subject: [openib-general] SDP debug In-Reply-To: <42B093D3.8020609@sgi.com> References: <42B093D3.8020609@sgi.com> Message-ID: <20050615211421.GQ4379@esmail.cup.hp.com> On Wed, Jun 15, 2005 at 03:47:15PM -0500, John Partridge wrote: > I am trying to get SDP working on SGI Altix. I know that the InfiniBand is > working as I have ipoib and ibv_pingpong etc all working fine. > > My question is how do I prove that an nttcp (or whatever I use) is using SDP > rather than ipoib ? > > I have used strace and can see the libsdp.conf file being read. So I know I > am going through libsdp.so (as defined in my LD_PRELOAD). > > Is there some DEBUG or counter I can use to tell me what is going on ? Start the "server" side of the test and see that it's using the right address family and port. I expect "netstat -a" will tell you. Its been a few monthes since I've poked at this. grant From ftillier at silverstorm.com Wed Jun 15 14:17:49 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Wed, 15 Jun 2005 14:17:49 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API In-Reply-To: <52slzje4gr.fsf@topspin.com> Message-ID: <000301c571ef$af9a3ad0$9c5aa8c0@infiniconsys.com> > From: Roland Dreier [mailto:roland at topspin.com] > Sent: Wednesday, June 15, 2005 1:43 PM > > Here's a first stab at the changes to the API in required > to support shared receive queues (SRQs). SRQs are described in > version 1.2 of the IBA spec; specifically, sections 10.2.9 and 11.2.3 > describe the semantics and verbs interface. I'm a bit confused still about how WR get processed. If a QP goes to the error state, does it pull all WRs off of the SRQ? I would expect not. How does one flush WRs from an SRQ? It seems that to use an SRQ, the user has to keep a parallel list of all WRs posted to track them so they may be properly freed when the SRQ is destroyed. This also means the completion path now must be serialized with the requesting path so that accesses to this tracking list are properly synchronized. > > I did diverge from the spec in that instead of a single "Post Receive > Request" verb, there are separate ib_post_recv() and > ib_post_srq_recv() functions. This avoids complications from having a > polymorphic function that takes either a QP or an SRQ, and matches the > interface Mellanox used in VAPI. This sounds sane. - Fab From libor at topspin.com Wed Jun 15 14:28:34 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 15 Jun 2005 14:28:34 -0700 Subject: [openib-general] SDP debug In-Reply-To: <42B093D3.8020609@sgi.com>; from johnip@sgi.com on Wed, Jun 15, 2005 at 03:47:15PM -0500 References: <42B093D3.8020609@sgi.com> Message-ID: <20050615142834.C7232@topspin.com> On Wed, Jun 15, 2005 at 03:47:15PM -0500, John Partridge wrote: > I am trying to get SDP working on SGI Altix. I know that the InfiniBand is > working as I have ipoib and ibv_pingpong etc all working fine. > > My question is how do I prove that an nttcp (or whatever I use) is using SDP > rather than ipoib ? > > I have used strace and can see the libsdp.conf file being read. So I know I > am going through libsdp.so (as defined in my LD_PRELOAD). > > Is there some DEBUG or counter I can use to tell me what is going on ? John, The easiest way is to look at /proc/net/sdp/devices, the second line contains the number of connections in the SDP connection table. There will be a connection for every SDP socket that's open, including listen sockets. There are also a few files in the same /proc directory which have a line of statistics per connection in the table. -Libor From hozer at hozed.org Wed Jun 15 14:30:47 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Wed, 15 Jun 2005 16:30:47 -0500 Subject: [openib-general] SDP: device mthca0 does not support fast memory regions Message-ID: <20050615213047.GM577@kalmia.hozed.org> I'm getting a 'SDP: device mthca0 does nto support fast memory regions' error on PPC64 systems.. Is there something that needs to be done for PPC, or could I have an older version of the mthca module hanging around? From libor at topspin.com Wed Jun 15 14:33:40 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 15 Jun 2005 14:33:40 -0700 Subject: [openib-general] SDP debug In-Reply-To: <20050615211421.GQ4379@esmail.cup.hp.com>; from iod00d@hp.com on Wed, Jun 15, 2005 at 02:14:21PM -0700 References: <42B093D3.8020609@sgi.com> <20050615211421.GQ4379@esmail.cup.hp.com> Message-ID: <20050615143340.D7232@topspin.com> On Wed, Jun 15, 2005 at 02:14:21PM -0700, Grant Grundler wrote: > On Wed, Jun 15, 2005 at 03:47:15PM -0500, John Partridge wrote: > > I am trying to get SDP working on SGI Altix. I know that the InfiniBand is > > working as I have ipoib and ibv_pingpong etc all working fine. > > > > My question is how do I prove that an nttcp (or whatever I use) is > > using SDP rather than ipoib ? > > > > Is there some DEBUG or counter I can use to tell me what is going on ? > > Start the "server" side of the test and see that it's using the > right address family and port. I expect "netstat -a" will tell you. > Its been a few monthes since I've poked at this. Actually 'netstat -a' will let you know that you're using SDP only in that the connection will not appear in the list. If you see the connection/port in the list then it's still using TCP. netstat would need to be expanded to open and use the data in /proc/net/sdp for it to display the connections. -Libor From roland at topspin.com Wed Jun 15 14:35:12 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 14:35:12 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API In-Reply-To: <000301c571ef$af9a3ad0$9c5aa8c0@infiniconsys.com> (Fab Tillier's message of "Wed, 15 Jun 2005 14:17:49 -0700") References: <000301c571ef$af9a3ad0$9c5aa8c0@infiniconsys.com> Message-ID: <52ekb3e227.fsf@topspin.com> Fab> I'm a bit confused still about how WR get processed. If a QP Fab> goes to the error state, does it pull all WRs off of the SRQ? Fab> I would expect not. Work requests are only taken from an SRQ when a message is received on a QP attached to the SRQ. The semantics are mostly described in chapter 10 of the IBA spec, eg in sections 10.2.9 and 10.8.3. Fab> How does one flush WRs from an SRQ? It seems that to use an Fab> SRQ, the user has to keep a parallel list of all WRs posted Fab> to track them so they may be properly freed when the SRQ is Fab> destroyed. This also means the completion path now must be Fab> serialized with the requesting path so that accesses to this Fab> tracking list are properly synchronized. The only way to flush the WRs is to destroy the SRQ. This does mean that the consumer needs to keep track of resources to be freed. However, one doesn't necesarily have to synchronize completions and work requests; for example independent head and tail pointers into a circular buffer could be used instead. - R. From caitlin.bestler at gmail.com Wed Jun 15 14:42:35 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 15 Jun 2005 14:42:35 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API In-Reply-To: <000301c571ef$af9a3ad0$9c5aa8c0@infiniconsys.com> References: <52slzje4gr.fsf@topspin.com> <000301c571ef$af9a3ad0$9c5aa8c0@infiniconsys.com> Message-ID: <469958e0050615144225f88557@mail.gmail.com> On 6/15/05, Fab Tillier wrote: > > From: Roland Dreier [mailto:roland at topspin.com] > > Sent: Wednesday, June 15, 2005 1:43 PM > > > > Here's a first stab at the changes to the API in required > > to support shared receive queues (SRQs). SRQs are described in > > version 1.2 of the IBA spec; specifically, sections 10.2.9 and 11.2.3 > > describe the semantics and verbs interface. > > I'm a bit confused still about how WR get processed. If a QP goes to the error > state, does it pull all WRs off of the SRQ? I would expect not. > > How does one flush WRs from an SRQ? It seems that to use an SRQ, the user has > to keep a parallel list of all WRs posted to track them so they may be properly > freed when the SRQ is destroyed. This also means the completion path now must > be serialized with the requesting path so that accesses to this tracking list > are properly synchronized. > For iWARP each WR is allocated to a specific QP when it is needed (i.e. when part of the message arrives, or potenitally when a later message arrives). WRs that have been assigned to a QP are flushed when the QP goes to an error state or is discconnected. I haven't studied the IB 1.2 spec on SRQs at horrendous lenght, but I strongly suspect that they match the iWARP SRQ specs fairly closely since I believe the same people wrote them. In which case one of the inherited defects is that there is no reliable method of draining an SRQ. The closest you can do is to do a loopback connection and send yourself short messages until you fail because the SRQ is empty. >From a theoretical viewpoint this is a glaring hole. From a realistic viewpoint virtually all applications that use SRQs create them and then keep them until the application exits. From caitlin.bestler at gmail.com Wed Jun 15 14:49:08 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 15 Jun 2005 14:49:08 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API In-Reply-To: <52slzje4gr.fsf@topspin.com> References: <52slzje4gr.fsf@topspin.com> Message-ID: <469958e005061514491dc5b491@mail.gmail.com> On 6/15/05, Roland Dreier wrote: > Here's a first stab at the changes to the API in required > to support shared receive queues (SRQs). SRQs are described in > version 1.2 of the IBA spec; specifically, sections 10.2.9 and 11.2.3 > describe the semantics and verbs interface. > > I did diverge from the spec in that instead of a single "Post Receive > Request" verb, there are separate ib_post_recv() and > ib_post_srq_recv() functions. This avoids complications from having a > polymorphic function that takes either a QP or an SRQ, and matches the > interface Mellanox used in VAPI. > > Any comments? I definitely agree on having two methods. If you want one method then it should accept the QP pointer all the time and merely redirect to the SRQ. Two totally distinct methods with signatures calling for the specific handle type allows the code to get right to work, not act as a sub-dispatcher. Polymorphism at run-time should only apply when the same actions are taken no matter what the type is. I would recommend defining warning events associated with SRQ usage. Omitting them creates glaring security vulnerabilities. DAT/IT-API define an SRQ low-watermark (event when number of buffers in SRQ falls below set level), per RQ soft high-watermark (event when one QP allocates more than N uncompleted buffers) and a per RQ hard high-watermark (connection terminated when more than N buffers required for one QP). The first two are defined for iWARP/RDMAC, the third is optional under RNIC-PI. Without such events a server attempting to use an SRQ to support multiple client will be subject to a Denial of Service attack from one client that threatens to kill or stall other connections. But servers supporting multiple clients are *exactly* the type of application that would benefit the most from SRQs. Of course you can't force any hardware that doesn't generate those warnings to do so, but if the API doesn't have the concept then you can *never* get those warnings. From mst at mellanox.co.il Wed Jun 15 14:53:34 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 16 Jun 2005 00:53:34 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050615173058.GB4379@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050615150440.GB20041@mellanox.co.il> <20050615173058.GB4379@esmail.cup.hp.com> Message-ID: <20050615215334.GA22691@mellanox.co.il> Quoting r. Grant Grundler : > 2) I working towards making the server side a daemon. I dont see this necessarily as an improvement. I'm happy running server and client with ssh from a script with multiple parameters, this works well for me - actually better than netserver. But we shall see. -- MST From roland at topspin.com Wed Jun 15 14:56:15 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 14:56:15 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API In-Reply-To: <469958e005061514491dc5b491@mail.gmail.com> (Caitlin Bestler's message of "Wed, 15 Jun 2005 14:49:08 -0700") References: <52slzje4gr.fsf@topspin.com> <469958e005061514491dc5b491@mail.gmail.com> Message-ID: <523brje134.fsf@topspin.com> Caitlin> I would recommend defining warning events associated with Caitlin> SRQ usage. Omitting them creates glaring security Caitlin> vulnerabilities. DAT/IT-API define an SRQ low-watermark Caitlin> (event when number of buffers in SRQ falls below set Caitlin> level), per RQ soft high-watermark (event when one QP Caitlin> allocates more than N uncompleted buffers) and a per RQ Caitlin> hard high-watermark (connection terminated when more than Caitlin> N buffers required for one QP). The first two are defined Caitlin> for iWARP/RDMAC, the third is optional under RNIC-PI. You're right. I misread the enum we already have and thought that we had SRQ events. I'll add the SRQ low-water event, since that's part of the IB spec supported by current HW, and we can add whatever other events are required as support for different HW is added to the tree. - R. From roland at topspin.com Wed Jun 15 15:08:33 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 15 Jun 2005 15:08:33 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API In-Reply-To: <523brje134.fsf@topspin.com> (Roland Dreier's message of "Wed, 15 Jun 2005 14:56:15 -0700") References: <52slzje4gr.fsf@topspin.com> <469958e005061514491dc5b491@mail.gmail.com> <523brje134.fsf@topspin.com> Message-ID: <52slzjcly6.fsf@topspin.com> OK, I added the below to my patch: enum ib_event_type { IB_EVENT_CQ_ERR, + IB_EVENT_SRQ_ERR, + IB_EVENT_SRQ_LIMIT_REACHED, IB_EVENT_QP_FATAL, IB_EVENT_QP_REQ_ERR, IB_EVENT_QP_ACCESS_ERR, + IB_EVENT_QP_LAST_WQE_REACHED, IB_EVENT_COMM_EST, IB_EVENT_SQ_DRAINED, IB_EVENT_PATH_MIG, - R. From mst at mellanox.co.il Wed Jun 15 15:11:39 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 16 Jun 2005 01:11:39 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050615173058.GB4379@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050615150440.GB20041@mellanox.co.il> <20050615173058.GB4379@esmail.cup.hp.com> Message-ID: <20050615221139.GA22860@mellanox.co.il> Quoting r. Grant Grundler : > 2) I working towards making the server side a daemon. > ie move stuff out of main: I have another idea: lets move all the socket stuff out of perftests altogether. Each test will just read/write standard input/output. Reports will go to standard error. On top of this, we'll be able to have a script/program, or multiple different programs, that will perform the server/client communication in whatever way we want it to. One simple way would be to dup the socket descriptor to standard input/output. It seems this would also let the same test use cm for connection setup, but I didnt look at ucm, yet. -- MST From iod00d at hp.com Wed Jun 15 16:50:55 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 15 Jun 2005 16:50:55 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050615215334.GA22691@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050615150440.GB20041@mellanox.co.il> <20050615173058.GB4379@esmail.cup.hp.com> <20050615215334.GA22691@mellanox.co.il> Message-ID: <20050615235055.GF5992@esmail.cup.hp.com> On Thu, Jun 16, 2005 at 12:53:34AM +0300, Michael S. Tsirkin wrote: > Quoting r. Grant Grundler : > > 2) I working towards making the server side a daemon. > > I dont see this necessarily as an improvement. > I'm happy running server and client with ssh from a script > with multiple parameters, this works well for me - actually better > than netserver. I don't allow root to ssh across my test environment since several of my machines have NICs are facing a very hostile internet. So that doesn't work for me. grant From iod00d at hp.com Wed Jun 15 17:33:13 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 15 Jun 2005 17:33:13 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050615221139.GA22860@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050615150440.GB20041@mellanox.co.il> <20050615173058.GB4379@esmail.cup.hp.com> <20050615221139.GA22860@mellanox.co.il> Message-ID: <20050616003313.GG5992@esmail.cup.hp.com> On Thu, Jun 16, 2005 at 01:11:39AM +0300, Michael S. Tsirkin wrote: > Quoting r. Grant Grundler : > > 2) I working towards making the server side a daemon. > > ie move stuff out of main: > > I have another idea: lets move all the socket stuff out of perftests > altogether. Each test will just read/write standard input/output. > Reports will go to standard error. That sounds good to me. > On top of this, we'll be able to have a script/program, or > multiple different programs, that will perform the > server/client communication in whatever way we want it to. I've been heading down this path already (splitting out the test) and have my latest version posted on ftp://gsyprf10.external.hp.com/pub/openib/perftest/rdma_lat.c-10 Sorry - I haven't been able to prepare a patch based on the current SVN content and probably won't tomorrow either. The above is the best I can offer right now. > One simple way would be to dup the socket descriptor > to standard input/output. I'm not a good network applications programmer...ie I'd be more comfortable taking netperf/netserver and hacking it to handle the "high level" stuff. I'm just not quite sure how to wedge the test into netperf at this point. > It seems this would also let the same test use cm for connection setup, > but I didnt look at ucm, yet. sorry - neither "cm" nor "ucm" mean anything to me (yet). Infiniband has more acronyms than a stray dog has fleas. :^) thanks, grant From iod00d at hp.com Wed Jun 15 18:03:23 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 15 Jun 2005 18:03:23 -0700 Subject: [openib-general] Re: cycles_to_units is incorrect in rdma_lat, rdma_bw. In-Reply-To: References: <20050615174024.GC4379@esmail.cup.hp.com> Message-ID: <20050616010323.GH5992@esmail.cup.hp.com> On Wed, Jun 15, 2005 at 11:55:07AM -0700, Shirley Ma wrote: > > in case someone's firmware is less accurate than HP's. > > Shirley, did the patch to get_clock.c fix the problem? > > Cycles is not equal to the clock rate. If you have a processor with > internal doubling or pipelining or whatever, the clock rate isn't the > same. This test assumes the ITC is running at some fixed rate. Linux expects that rate to be reported in /proc/cpuinfo with "cpu Mhz". Does your machine's cycle counter run at 1600 Mhz like the /proc/cpuinfo output indicated? hyperthreading, pipelining, superscaler, etc are all orthogonal issues. The ITC is what matters. > > Anyway, if get_cpu_mhz isnt reliable people can just dump raw > > cycles data and do the math outside the tool. > > Agree if the rdma_lat, and rdma_bw are only used for debugging/regression > testing not for performance measurement. It would be nice if the test would determine the report result was more than X% (e.g. 2%) off from the what the system clock reported. It would be appropriate to warn the user about bogus results. But this is a benchmarking issue. If rdma_lat is primarily a micro-benchmark and secondarily an RDMA code example, then I think it would be good to address this. grant From mst at mellanox.co.il Wed Jun 15 22:44:46 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 16 Jun 2005 08:44:46 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050615235055.GF5992@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050615150440.GB20041@mellanox.co.il> <20050615173058.GB4379@esmail.cup.hp.com> <20050615215334.GA22691@mellanox.co.il> <20050615235055.GF5992@esmail.cup.hp.com> Message-ID: <20050616054446.GA30238@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [PATCH] rdma_lat-09 and results > > On Thu, Jun 16, 2005 at 12:53:34AM +0300, Michael S. Tsirkin wrote: > > Quoting r. Grant Grundler : > > > 2) I working towards making the server side a daemon. > > > > I dont see this necessarily as an improvement. > > I'm happy running server and client with ssh from a script > > with multiple parameters, this works well for me - actually better > > than netserver. > > I don't allow root to ssh across my test environment since several > of my machines have NICs are facing a very hostile internet. > So that doesn't work for me. > > grant > Dont run the test as root then. uverbs work for a regular user. -- MST From itamar at mellanox.co.il Thu Jun 16 01:39:00 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 16 Jun 2005 11:39:00 +0300 Subject: [openib-general] kdapl locking problem Message-ID: <91DB792C7985D411BEC300B40080D29CC35C9C@mtvex01.mtv.mtl.com> Hi Hal, I am trying to understand what is going here and i still dont see how this happan . This prints are only set in UP mode .(is this your system UP?) the code is (function: dapl_evd_connection_callback): spin_lock_irqsave(&ep->common.lock, ep->common.flags); case on the event type disconnect: dapl_ib_disconnect_clean(ep, TRUE); spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); from some reason in the middle between the lock and the unlock there is a call to consumer function (dat_ep_disconnetc) that try to disconnect the same ep and the lock fail. the evd_cb function is either an interupt from the CM so i dont see how the consumer can call dat_ib_disconnect in the middle or the user called twice to dat_ib_disconnect on the same ep and youe kernel give preemption i dont understand both (;-) can you try to run it with some debug? at least ot know who called to dapl_evd_connection_callback ? Itamar > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Tuesday, June 14, 2005 8:37 PM > To: James Lentini > Cc: openib-general at openib.org > Subject: [openib-general] kdapl locking problem > > > Hi, > > When running in loopback mode (client and server on same > machine (x86)): > kdapltest -T T -s -D mthca0a -d -t 2 -w 8 -i 20 > client SR server SR > I see the following locking problem: > > Jun 14 13:30:08 localhost kernel: > drivers/infiniband/ulp/dat-provider/dapl_ep.c:1111: > spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c44b1c > 18) already locked by > drivers/infiniband/ulp/dat-provider/dapl_evd.c/756 > Jun 14 13:30:08 localhost kernel: > drivers/infiniband/ulp/dat-provider/dapl_evd.c:797: > spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c44b > 1c18) not locked > > -- Hal > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From halr at voltaire.com Thu Jun 16 05:40:19 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 Jun 2005 08:40:19 -0400 Subject: [openib-general] [IPoIB] Add support for MTU module parameter Message-ID: <1118925570.4506.1052.camel@localhost.localdomain> [IPoIB] Add support for MTU module parameter. This is so there can be a non default MTU at boot up if the administrator so desires (prior to being able to invoke ifconfig). Signed-off-by: Hal Rosenstock Index: ipoib_main.c =================================================================== --- ipoib_main.c (revision 2631) +++ ipoib_main.c (working copy) @@ -58,6 +58,10 @@ MODULE_PARM_DESC(debug_level, "Enable debug tracing if > 0"); #endif +static unsigned int mtu = IPOIB_PACKET_SIZE - IPOIB_ENCAP_LEN; +module_param(mtu, int, 0); +MODULE_PARM_DESC(mtu, "Override the default MTU value of 2044."); + static const u8 ipv4_bcast_addr[] = { 0x00, 0xff, 0xff, 0xff, 0xff, 0x12, 0x40, 0x1b, 0x00, 0x00, 0x00, 0x00, @@ -824,7 +828,7 @@ dev->features = NETIF_F_VLAN_CHALLENGED | NETIF_F_LLTX; /* MTU will be reset when mcast join happens */ - dev->mtu = IPOIB_PACKET_SIZE - IPOIB_ENCAP_LEN; + dev->mtu = mtu; priv->mcast_mtu = priv->admin_mtu = dev->mtu; memcpy(dev->broadcast, ipv4_bcast_addr, INFINIBAND_ALEN); From Diego at Mellanox.com Thu Jun 16 06:08:29 2005 From: Diego at Mellanox.com (Diego Crupnicoff) Date: Thu, 16 Jun 2005 06:08:29 -0700 Subject: [openib-general] [RFC] [PATCH] SRQ API Message-ID: <25AE7F432672D511B8DC00B0D0DF11DA04D57461@MTIEX01> > -----Original Message----- > From: Caitlin Bestler [mailto:caitlin.bestler at gmail.com] > Sent: Wednesday, June 15, 2005 6:49 PM > To: Roland Dreier > Cc: openib-general at openib.org > Subject: Re: [openib-general] [RFC] [PATCH] SRQ API > > ... > per RQ soft high-watermark (event when one QP allocates more > than N uncompleted buffers) and a > per RQ hard high-watermark (connection terminated when more > than N buffers required for one QP). The first two are > defined for iWARP/RDMAC, the third is optional under RNIC-PI. > In IB, at any given time, one QP will never have more than 1 receive WQE allocated. So these last two events would never trigger on an IB implementation. Diego -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlentini at netapp.com Thu Jun 16 08:08:32 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 16 Jun 2005 11:08:32 -0400 (EDT) Subject: [openib-general] kdapl locking problem In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C9C@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C9C@mtvex01.mtv.mtl.com> Message-ID: As Itamar observed, the first message is printed out by _raw_spin_lock() when the kernel is compiled for UP and spin lock debugging. dapl_ep_disconnect() is trying to obtain a lock that is already locked. The message indicates that the lock was taken in dapl_evd_connection_callback(). There is no control flow path from dapl_evd_connection_callback() that reaches dapl_ep_disconnect(). I'm also unsure of how execution could have reached dapl_ep_disconnect() with the spin lock locked. We are using spin_lock_irqsave(). My understanding is that interrupts will be masked until spin_unlock_irqrestore() is called. That would imply that it is not possible for the control flow to change to another context that calls dapl_ep_disconnect(). The second message is a by-product of the first problem. dapl_ep_disconnect() unlocks the spin lock, so when control returns to dapl_evd_connection_callback(), the lock is already unlocked. So we just need to fix the first problem. Are we using spin_lock_irqsave() incorrectly? james On Thu, 16 Jun 2005, Itamar Rabenstein wrote: > Hi Hal, > I am trying to understand what is going here and i still dont see how this > happan . > > This prints are only set in UP mode .(is this your system UP?) > the code is (function: dapl_evd_connection_callback): > spin_lock_irqsave(&ep->common.lock, ep->common.flags); > case on the event type > disconnect: dapl_ib_disconnect_clean(ep, TRUE); > spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); > > from some reason in the middle between the lock and the unlock there is a > call to consumer > function (dat_ep_disconnetc) that try to disconnect the same ep and the lock > fail. > > the evd_cb function is either an interupt from the CM so i dont see how the > consumer can call > dat_ib_disconnect in the middle > or the user called twice to dat_ib_disconnect on the same ep and youe kernel > give preemption > > i dont understand both (;-) > > can you try to run it with some debug? > at least ot know who called to dapl_evd_connection_callback ? > > Itamar > > >> -----Original Message----- >> From: Hal Rosenstock [mailto:halr at voltaire.com] >> Sent: Tuesday, June 14, 2005 8:37 PM >> To: James Lentini >> Cc: openib-general at openib.org >> Subject: [openib-general] kdapl locking problem >> >> >> Hi, >> >> When running in loopback mode (client and server on same >> machine (x86)): >> kdapltest -T T -s -D mthca0a -d -t 2 -w 8 -i 20 >> client SR server SR >> I see the following locking problem: >> >> Jun 14 13:30:08 localhost kernel: >> drivers/infiniband/ulp/dat-provider/dapl_ep.c:1111: >> spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c44b1c >> 18) already locked by >> drivers/infiniband/ulp/dat-provider/dapl_evd.c/756 >> Jun 14 13:30:08 localhost kernel: >> drivers/infiniband/ulp/dat-provider/dapl_evd.c:797: >> spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c44b >> 1c18) not locked >> >> -- Hal >> >> _______________________________________________ >> openib-general mailing list >> openib-general at openib.org >> http://openib.org/mailman/listinfo/openib-general >> >> To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From taniguchi at make-love.cx Thu Jun 16 08:24:55 2005 From: taniguchi at make-love.cx (taniguchi at make-love.cx) Date: Fri, 17 Jun 2005 00:24:55 +0900 Subject: [openib-general] =?iso-2022-jp?b?GyRCJCQkRCRiJCpAJE9DJEsbKEI=?= =?iso-2022-jp?b?GyRCJEokaiReJDkhIxsoQk1BS0UtVVAgTE9WRRskQiFaGyhC?= =?iso-2022-jp?b?GyRCJWElJCUvISYlaSVWIVsbKEJWb2wuMw==?= Message-ID: <200506170024796.SM00848@192.168.1.21> お久しぶりです。はじめましての方もいらっしゃると思いますが改めまして ご挨拶させていただきます。MAKE-UP LOVE　代表 藁科美智子でございます。 最初に申しておきますが、当クラブ「MAKE-UP LOVE」の会報メールがご不要な 場合にはそのまま破棄してください。 尚、「配信不要」とお送りいただければ速報メールは二度と配信致しません。 それでは今回の緊急速報の内容に移りたいと思います。 この間のアンケートの結果が出ました。 お題は「今あなたが抱えている悩み」でした。 たくさんのご意見ありがとうございました。 そこでいくつか頂いたメールをご紹介したいと思います。 ☆まずは東京都にお住まいの武さん(仮)のメッセージです☆ ---------------------------------------------------------------------------- 僕には彼女がいたのですがなんと浮気ばかりしているので別れてしまいました。 婚約までしていたのでやりきれない思いでいっぱりです。 藁科さん、真面目な出会いがしたいです。 ---------------------------------------------------------------------------- ☆続いては愛知県名古屋市にお住まいの阿部さん32歳さんからのメッセージです☆ ---------------------------------------------------------------------------- 私の悩みを聞いてください。最近までキャバクラ通いをしていたのです。 キャバクラ嬢と仲良くなれたのはいいのですが・・・。 自分の意思の弱さに漬け込まれてしまい借金の保証人になってしまったのです。 金額はもう口には出せないくらいの額です。 もう絶望的です。 ---------------------------------------------------------------------------- 多数寄せられたご意見の中から２件のメッセージをご紹介させていただきました。 これをお読みになられている皆さんの中にもこのお二方と同じく女性関係、金銭の ことで悩んでいる方もいるのではありませんか？ そこで皆さんのお悩みをすぐにでも解決させていただきたく今回の速報メールを 送らせていただきました。 そう私達「MAKE-UP LOVE」はそんな皆様のために存在するのです。 このクラブでは男女同士の大人の出会いを提供いたします。 ですが・・・ここで１つ問題があるのです。 それは実は私達「MAKE-UP LOVE」は女性会員8256名に対し、男性会員883名 と男性会員様の人数があまりに少ないのです。 そこで今回このメールをお読みの皆様から男性会員を募集することに至りました。 当クラブでは『身元の確かな出会いに縁のない女性会員様と誠実で秘密厳守を 守れる男性会員様』の出会いを第一に考えております。 また「MAKE-UP LOVE」では女性会員様の会費で運営されております。 お会いする場合にはくれぐれも失礼のないようお願い致します。 また男性会員様におきまして登録料・紹介料は一切いただいておりません。 ☆前回のメール読者様で会員様になられた方がいらっしゃるのですが、その方から メッセージが届いております☆ ---------------------------------------------------------------------------- MAKE-UP LOVEメールを読んでるみんな！はじめまして。前回のメールで会員になり ました岡崎明雄と言います！僕はたまたまMAKE-UP LOVEメールに登録して前回届いた メールを読んだんだけど最初は何だ？って疑問に思ったよ。これを読んでる皆の中 にも「何だこれは？」とか「こんなの登録した覚えないぞ」とか思ってる人もいる んじゃないかな？でもこのメールをもらったみんなはすごいラッキーだよ！ 僕は多額の借金があって二進も三進も行かない状態だったんだけどこのクラブに 女性を紹介してもらったおかげ人生何度でもやり直しが利くんだって気がついたんだよ。 紹介された女性の方は某企業の重役だったんだけど困ってる僕に会うたびにサポート してくれて。今でも関係は続いてるんだ。もう僕にとってなくてはならない存在です。 こんな僕でも素敵な方と出会えました。メール読者のみんなも登録は無料だし騙され たと思って試してみるのはどうかな？それでは岡崎明雄でした。またね！ -------------------------------------------------------------------------------- 皆様にも岡崎明雄さんのようなビックチャンスが待っております。 当クラブには様々な女性が在籍しております。 学生、ＯＬ、フリーター、セレブ、淫乱マダム、ＳＭ、３Ｐなどなど盛りだくさんです。 女性会員様のご年齢は18歳〜65歳です。 日本全国に女性会員様が多数在籍しておりますので、 まずは貴方様の地域に該当している女性会員様をご紹介させて頂きたいと思います。 簡単なシートですので、まずはお気軽にお書き下さいませ。 Ｑ：ＮＯ,１ 　　貴方様のお会いするにあたってのご希望の地域(都道府県等) Ｑ：ＮＯ,２ 　 当クラブをご利用するにあたってのご利用タイプ 【タイプ1】真面目に一般女性とお付き合い　　（真面目な交際・結婚前提） 【タイプ2】女性スポンサーとお付き合い　　　（事業資金などサポートして欲しい） 【タイプ3】割り切った大人のお付き合い　　　（愛人・セフレ・人妻・ＳＭなど） 【タイプ4】４０歳以上同士の新しい交際　 （４０歳以上同士の交際） 以上のご質問にお答え頂きご返信いただければ、当クラブの各地域担当女性から 貴方様のご希望に沿った女性をご案内させて頂きます。 『MAKE-UP LOVE』代表　藁科美智子 From halr at voltaire.com Thu Jun 16 08:35:20 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 Jun 2005 11:35:20 -0400 Subject: [openib-general] [PATCH] [AT] Initial changes to work with user AT Message-ID: <1118936120.4506.1267.camel@localhost.localdomain> Initial changes to work with user AT. Also, some other changes based on code inspection. Signed-off-by: Hal Rosenstock Index: at.c =================================================================== --- at.c (revision 2610) +++ at.c (working copy) @@ -118,7 +118,7 @@ int sa_id; }; -struct async pending_reqs; /* dummy head for cyclic list */ +static struct async pending_reqs; /* dummy head for cyclic list */ struct ib_at_src { u32 ip; @@ -286,7 +286,7 @@ spin_lock_irqsave(&pending_reqs.lock, flags); new_id = ++req_id; - if (!new_id) + if (!new_id) /* 0 is not used as req_id (reserved value) */ new_id = ++req_id; spin_unlock_irqrestore(&pending_reqs.lock, flags); @@ -319,9 +319,13 @@ break; default: WARN("bad async req type %d", pend->type); + if (pend->sa_query) { + ib_sa_cancel_query(pend->sa_id, pend->sa_query); + pend->sa_query = NULL; + } pend->status = IB_AT_STATUS_INVALID; pend->type = IBAT_REQ_NONE; - pend->sa_query = NULL; + break; } } @@ -377,16 +381,16 @@ DEBUG("pend %p nrec %d async %p", pend, nrec, q); + if (pend->sa_query) { + ib_sa_cancel_query(pend->sa_id, pend->sa_query); + pend->sa_query = NULL; + } + if (pend->status != IB_AT_STATUS_PENDING) WARN("pend %p already completed? status %d", pend, pend->status); pend->status = nrec < 0 ? IB_AT_STATUS_ERROR : IB_AT_STATUS_COMPLETED; - if (pend->sa_query) { - ib_sa_cancel_query(pend->sa_id, pend->sa_query); - pend->sa_query = NULL; - } - if (q) spin_lock_irqsave(&q->lock, flags); @@ -464,9 +468,12 @@ DEBUG("free async %p req %p", async, req); + if (req->pend.sa_query) { + ib_sa_cancel_query(req->pend.sa_id, req->pend.sa_query); + req->pend.sa_query = NULL; + } req->pend.status = IB_AT_STATUS_INVALID; req->pend.type = IBAT_REQ_NONE; - req->pend.sa_query = NULL; kmem_cache_free(route_req_cache, req); } @@ -477,9 +484,12 @@ DEBUG("free async %p req %p", async, req); + if (req->pend.sa_query) { + ib_sa_cancel_query(req->pend.sa_id, req->pend.sa_query); + req->pend.sa_query = NULL; + } req->pend.status = IB_AT_STATUS_INVALID; req->pend.type = IBAT_REQ_NONE; - req->pend.sa_query = NULL; kmem_cache_free(path_req_cache, req); } @@ -494,7 +504,7 @@ DEBUG("lookup in q %p pending %p", q, new); spin_lock_irqsave(&q->lock, flags); for (a = q->next; a != q; a = a->next) { - DEBUG("%d %d", a->type, type); + DEBUG("req type %d %d", a->type, type); if (a->type == type && same_fn(a, new)) break; } @@ -508,7 +518,7 @@ unsigned long flags; struct async *a; - DEBUG("lookup in q %p id %llx", q, id); + DEBUG("lookup in q %p id 0x%llx", q, id); spin_lock_irqsave(&q->lock, flags); for (a = q->next; a != q; a = a->next) if (a->id == id) @@ -551,8 +561,7 @@ { int n = min(npath, nelem); - DEBUG("fill ib_sa_path_rec %p output %d records", out, n); - + DEBUG("output ib_sa_path_rec %p %d records", out, n); memcpy(out, resp, n * sizeof (struct ib_sa_path_rec)); return n; } @@ -589,15 +598,15 @@ DEBUG("req %p status %d", req, status); + req->pend.sa_query = NULL; + if (req->pend.parent) { WARN("for child req %p???", req); return; } - req->pend.sa_query = NULL; - if (status) { - DEBUG("status %d - check if should retry", status); + DEBUG("status %d - checking if should retry", status); if (status == -ETIMEDOUT && jiffies - req->pend.start < IB_AT_REQ_TIMEOUT) resolve_path(req); @@ -625,7 +634,7 @@ struct path_req *preq; unsigned long flags; - DEBUG("start sweeping"); + DEBUG("start sweep"); spin_lock_irqsave(&pending_reqs.lock, flags); for (pend = pending_reqs.next; pend != &pending_reqs; pend = next) { @@ -638,7 +647,7 @@ DEBUG("examining route req %p pend %p", req, pend); if (jiffies > pend->start + IB_AT_REQ_TIMEOUT) { - DEBUG("req delete <%d.%d.%d.%d> <%lu:%lu>", + DEBUG("delete route <%d.%d.%d.%d> <%lu:%lu>", (req->dst_ip & 0x000000ff), (req->dst_ip & 0x0000ff00) >> 8, (req->dst_ip & 0x00ff0000) >> 16, @@ -653,7 +662,7 @@ DEBUG("examining path req %p pend %p", preq, pend); if (jiffies > pend->start + IB_AT_REQ_TIMEOUT) { - DEBUG("req delete path <%lu:%lu>", + DEBUG("delete path <%lu:%lu>", jiffies, pend->start); req_end(pend, -ETIMEDOUT, NULL); @@ -661,6 +670,7 @@ break; default: WARN("unknown async req type %d", pend->type); + break; } } @@ -722,7 +732,8 @@ &req->pend.sa_query); if (req->pend.sa_id < 0) { - WARN("ib_sa_path_rec_get %d", req->pend.sa_id); + WARN("ib_sa_path_rec_get failed %d", req->pend.sa_id); + req->pend.sa_query = NULL; return req->pend.sa_id; } From jlentini at netapp.com Thu Jun 16 08:48:19 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 16 Jun 2005 11:48:19 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl]update dat_rmr_bind API & delete dapl_hash In-Reply-To: <20050615133403.GA19837@mellanox.co.il> References: <20050615133403.GA19837@mellanox.co.il> Message-ID: Itamar, Can you seperate the RMR api change and hash table removal into a seperate patch? Merging the HCA and IA code into the dapl_ia.[ch] files doesn't feel right to me. I think of the HCA as a DAT object just like IAs, EPs, LMRs, etc, and hence it should have its own file. james On Wed, 15 Jun 2005, Rabenstein Itamar wrote: > Updated dat_rmr_bind API (added lmr_handle as input param) > delete dapl_hash.[h|c] files (not needed any more) > Integrate dapl_hca_util functions in ia/provider code > delete dapl_hca_util.[h|c] files > > Signed-off-by: Itamar Rabenstein > > Index: test/dapltest/test/dapl_bpool.c > =================================================================== > --- test/dapltest/test/dapl_bpool.c (revision 2608) > +++ test/dapltest/test/dapl_bpool.c (working copy) > @@ -236,6 +236,7 @@ > bpool_ptr->reg_addr, bpool_ptr->reg_size)); > > ret = dat_rmr_bind ( bpool_ptr->rmr_handle, > + bpool_ptr->lmr, > &iov, > mflags, > bpool_ptr->ep, > Index: test/dapltest/kdapl/kdapl_tdep_user.c > =================================================================== > --- test/dapltest/kdapl/kdapl_tdep_user.c (revision 2608) > +++ test/dapltest/kdapl/kdapl_tdep_user.c (working copy) > @@ -76,7 +76,7 @@ > } > if (params_ptr->test_type == TRANSACTION_TEST) { > print_ioctl.cookie = ioctl (fd, KDAPL_IOCTL_GET_STAT_T, &Client_Stats_T); > - DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_iterations, > + DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_threads, > params_ptr->u.Transaction_Cmd.eps_per_thread); > } > > Index: dat-provider/dapl_hca_util.c > =================================================================== > --- dat-provider/dapl_hca_util.c (revision 2608) > +++ dat-provider/dapl_hca_util.c (working copy) > @@ -1,150 +0,0 @@ > -/* > - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > - * > - * This Software is licensed under one of the following licenses: > - * > - * 1) under the terms of the "Common Public License 1.0" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/cpl.php. > - * > - * 2) under the terms of the "The BSD License" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/bsd-license.php. > - * > - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > - * copy of which is available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/gpl-license.php. > - * > - * Licensee has the right to choose one of the above licenses. > - * > - * Redistributions of source code must retain the above copyright > - * notice and one of the license notices. > - * > - * Redistributions in binary form must reproduce both the above copyright > - * notice, one of the license notices in the documentation > - * and/or other materials provided with the distribution. > - */ > - > -/* > - * $Id$ > - */ > - > -#include "dapl.h" > -#include "dapl_openib_util.h" > -#include "dapl_provider.h" > -#include "dapl_hca_util.h" > -#include "dapl_hash.h" > - > -/* > - * dapl_hca_alloc > - * > - * alloc and initialize an HCA struct > - * > - * Input: > - * name > - * port > - * > - * Output: > - * hca > - * > - * Returns: > - * none > - * > - */ > -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) > -{ > - struct dapl_hca *hca; > - > - hca = kmalloc(sizeof *hca, GFP_ATOMIC); > - if (hca) { > - memset(hca, 0, sizeof *hca); > - > - if (DAT_SUCCESS == > - dapl_hash_create(DAPL_HASH_TABLE_DEFAULT_CAPACITY, > - &hca->lmr_hash_table)) { > - spin_lock_init(&hca->lock); > - INIT_LIST_HEAD(&hca->ia_list); > - > - hca->name = dapl_os_strdup(name); > - hca->ib_hca_handle = device; > - hca->port_num = port; > - if (hca->name == NULL) { > - kfree(hca); > - hca = NULL; > - } > - } else { > - kfree(hca); > - hca = NULL; > - } > - } > - > - return hca; > -} > - > -/* > - * dapl_hca_free > - * > - * free an IA INFO struct > - * > - * Input: > - * hca > - * > - * Output: > - * none > - * > - * Returns: > - * none > - * > - */ > -void dapl_hca_free(struct dapl_hca *hca) > -{ > - (void)dapl_hash_free(hca->lmr_hash_table); > - kfree(hca->name); > - kfree(hca); > -} > - > -/* > - * dapl_hca_link_ia > - * > - * Add an ia to the HCA structure > - * > - * Input: > - * hca > - * ia_ptr > - * > - * Output: > - * none > - * > - * Returns: > - * none > - * > - */ > -void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia_ptr) > -{ > - spin_lock_irqsave(&hca->lock, hca->flags); > - list_add(&ia_ptr->list, &hca->ia_list); > - spin_unlock_irqrestore(&hca->lock, hca->flags); > -} > - > -/* > - * dapl_hca_unlink_ia > - * > - * Remove an ia from the hca info structure > - * > - * Input: > - * hca > - * ia_ptr > - * > - * Output: > - * none > - * > - * Returns: > - * none > - * > - */ > -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia) > -{ > - spin_lock_irqsave(&hca->lock, hca->flags); > - list_del(&ia->list); > - spin_unlock_irqrestore(&hca->lock, hca->flags); > -} > Index: dat-provider/dapl_ia.c > =================================================================== > --- dat-provider/dapl_ia.c (revision 2608) > +++ dat-provider/dapl_ia.c (working copy) > @@ -33,7 +33,6 @@ > #include "dapl_ia.h" > #include "dapl_provider.h" > #include "dapl_evd.h" > -#include "dapl_hca_util.h" > #include "dapl_openib_util.h" > #include "dapl_sp.h" > #include "dapl_cr.h" > @@ -70,8 +69,9 @@ > INIT_LIST_HEAD(&ia->psp_list); > INIT_LIST_HEAD(&ia->srq_list); > > - dapl_hca_link_ia(hca, ia); > - > + spin_lock_irqsave(&hca->lock, hca->flags); > + list_add(&ia->list, &hca->ia_list); > + spin_unlock_irqrestore(&hca->lock, hca->flags); > return ia; > } > > @@ -378,7 +378,9 @@ > dapl_os_assert(list_empty(&ia->psp_list)); > dapl_os_assert(list_empty(&ia->rsp_list)); > > - dapl_hca_unlink_ia(ia->hca, ia); > + spin_lock_irqsave(&ia->hca->lock, ia->hca->flags); > + list_del(&ia->list); > + spin_unlock_irqrestore(&ia->hca->lock, ia->hca->flags); > /* no need to destroy ia->common.lock */ > > kfree(ia); > Index: dat-provider/dapl_lmr.c > =================================================================== > --- dat-provider/dapl_lmr.c (revision 2608) > +++ dat-provider/dapl_lmr.c (working copy) > @@ -31,7 +31,6 @@ > > #include "dapl_openib_util.h" > #include "dapl_ia.h" > -#include "dapl_hash.h" > > static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, > enum dat_mem_type mem_type, > @@ -103,17 +102,6 @@ > if (DAT_SUCCESS != status) > goto error2; > > - /* if the LMR context is already in the hash table */ > - status = dapl_hash_search(ia->hca->lmr_hash_table, > - new_lmr->param.lmr_context, NULL); > - if (status == DAT_SUCCESS) > - goto error3; > - > - status = dapl_hash_insert(ia->hca->lmr_hash_table, > - new_lmr->param.lmr_context, lmr); > - if (status != DAT_SUCCESS) > - goto error3; > - > atomic_inc(&pz->pz_ref_count); > > if (lmr) > @@ -129,8 +117,6 @@ > > return DAT_SUCCESS; > > -error3: > - (void)dapl_ib_mr_deregister(new_lmr); > error2: > dapl_lmr_dealloc(new_lmr); > error1: > @@ -167,17 +153,6 @@ > if (DAT_SUCCESS != status) > goto error2; > > - /* if the LMR context is already in the hash table */ > - status = dapl_hash_search(ia->hca->lmr_hash_table, > - new_lmr->param.lmr_context, NULL); > - if (status == DAT_SUCCESS) > - goto error3; > - > - status = dapl_hash_insert(ia->hca->lmr_hash_table, > - new_lmr->param.lmr_context, lmr); > - if (status != DAT_SUCCESS) > - goto error3; > - > atomic_inc(&pz->pz_ref_count); > > if (lmr) > @@ -193,8 +168,6 @@ > > return DAT_SUCCESS; > > -error3: > - (void)dapl_ib_mr_deregister(new_lmr); > error2: > dapl_lmr_dealloc(new_lmr); > error1: > @@ -215,12 +188,6 @@ > DAT_REGION_DESCRIPTION reg_desc; > u32 status; > > - status = dapl_hash_search(ia->hca->lmr_hash_table, > - original_lmr->param.lmr_context, > - (DAPL_HASH_DATA *) &lmr); > - if (status != DAT_SUCCESS) > - goto error1; > - > reg_desc.for_lmr = (struct dat_lmr *) original_lmr; > > new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, > @@ -235,17 +202,6 @@ > if (DAT_SUCCESS != status) > goto error2; > > - /* if the LMR context is already in the hash table */ > - status = dapl_hash_search(ia->hca->lmr_hash_table, > - new_lmr->param.lmr_context, NULL); > - if (status == DAT_SUCCESS) > - goto error3; > - > - status = dapl_hash_insert(ia->hca->lmr_hash_table, > - new_lmr->param.lmr_context, lmr); > - if (status != DAT_SUCCESS) > - goto error3; > - > atomic_inc(&pz->pz_ref_count); > > if (lmr) > @@ -262,8 +218,6 @@ > > return DAT_SUCCESS; > > -error3: > - dapl_ib_mr_deregister(new_lmr); > error2: > dapl_lmr_dealloc(new_lmr); > error1: > @@ -371,22 +325,12 @@ > if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) > return DAT_INVALID_STATE; > > - status = dapl_hash_remove( > - dapl_lmr->common.owner_ia->hca->lmr_hash_table, > - dapl_lmr->param.lmr_context, NULL); > - if (status != DAT_SUCCESS) > - goto error; > - > status = dapl_ib_mr_deregister(dapl_lmr); > if (status == DAT_SUCCESS) { > pz = (struct dapl_pz *)dapl_lmr->param.pz; > atomic_dec(&pz->pz_ref_count); > dapl_lmr_dealloc(dapl_lmr); > - } else /* failure; put dapl_lmr back in hash table */ > - dapl_hash_insert(dapl_lmr->common.owner_ia-> > - hca->lmr_hash_table, > - dapl_lmr->param.lmr_context, dapl_lmr); > - > + } > break; > } > case DAT_MEM_TYPE_PLATFORM: > Index: dat-provider/Makefile > =================================================================== > --- dat-provider/Makefile (revision 2608) > +++ dat-provider/Makefile (working copy) > @@ -20,8 +20,6 @@ > dapl_cr \ > dapl_ep \ > dapl_evd \ > - dapl_hash \ > - dapl_hca_util \ > dapl_ia \ > dapl_lmr \ > dapl_provider \ > Index: dat-provider/dapl_hca_util.h > =================================================================== > --- dat-provider/dapl_hca_util.h (revision 2608) > +++ dat-provider/dapl_hca_util.h (working copy) > @@ -1,45 +0,0 @@ > -/* > - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > - * > - * This Software is licensed under one of the following licenses: > - * > - * 1) under the terms of the "Common Public License 1.0" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/cpl.php. > - * > - * 2) under the terms of the "The BSD License" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/bsd-license.php. > - * > - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > - * copy of which is available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/gpl-license.php. > - * > - * Licensee has the right to choose one of the above licenses. > - * > - * Redistributions of source code must retain the above copyright > - * notice and one of the license notices. > - * > - * Redistributions in binary form must reproduce both the above copyright > - * notice, one of the license notices in the documentation > - * and/or other materials provided with the distribution. > - */ > - > -/* > - * $Id$ > - */ > - > -#ifndef DAPL_HCA_UTIL_H > -#define DAPL_HCA_UTIL_H > - > -#include "dapl.h" > - > -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port); > - > -void dapl_hca_free(struct dapl_hca *hca); > - > -void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia); > - > -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia); > - > -#endif > Index: dat-provider/dapl_rmr.c > =================================================================== > --- dat-provider/dapl_rmr.c (revision 2608) > +++ dat-provider/dapl_rmr.c (working copy) > @@ -32,7 +32,6 @@ > #include "dapl.h" > #include "dapl_ep.h" > #include "dapl_ia.h" > -#include "dapl_hash.h" > #include "dapl_cookie.h" > #include "dapl_openib_util.h" > > @@ -148,26 +147,18 @@ > } > > static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, > - const struct dat_lmr_triplet *lmr_triplet, > + struct dapl_lmr *lmr, > + const struct dat_lmr_triplet *lmr_triplet, > enum dat_mem_priv_flags mem_priv, > struct dapl_ep *ep_ptr, > DAT_RMR_COOKIE user_cookie, > enum dat_completion_flags completion_flags, > DAT_RMR_CONTEXT *rmr_context) > { > - struct dapl_lmr *lmr; > struct dapl_cookie *cookie; > u32 status; > boolean_t is_signaled; > > - status = dapl_hash_search(rmr->common.owner_ia->hca->lmr_hash_table, > - lmr_triplet->lmr_context, > - (DAPL_HASH_DATA *) &lmr); > - if (DAT_SUCCESS != status) { > - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); > - goto bail; > - } > - > /* > * if the ep in unconnected return an error. IB requires that the > * QP be connected to change a memory window binding since: > @@ -352,6 +343,7 @@ > * Output: > */ > u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, > + struct dat_lmr *lmr_handle, > const struct dat_lmr_triplet *lmr_triplet, > enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, > DAT_RMR_COOKIE user_cookie, > @@ -359,20 +351,25 @@ > DAT_RMR_CONTEXT *rmr_context) > { > struct dapl_rmr *rmr; > + struct dapl_lmr *lmr; > struct dapl_ep *ep_ptr; > > if (!rmr_handle) > return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); > > + if (!lmr_handle) > + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); > + > if (!ep) > return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); > > rmr = (struct dapl_rmr *)rmr_handle; > + lmr = (struct dapl_lmr *)lmr_handle; > ep_ptr = (struct dapl_ep *)ep; > > /* if the rmr should be bound */ > if (0 != lmr_triplet->segment_length) > - return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, > + return dapl_rmr_bind_fuse(rmr, lmr, lmr_triplet, mem_priv, ep_ptr, > user_cookie, completion_flags, > rmr_context); > else /* the rmr should be unbound */ > Index: dat-provider/dapl_util.h > =================================================================== > --- dat-provider/dapl_util.h (revision 2608) > +++ dat-provider/dapl_util.h (working copy) > @@ -40,7 +40,6 @@ > #include > #include > #include > -#include /* needed by hash functions */ > > #ifdef __ia64__ > #include > @@ -120,18 +119,6 @@ > } > > /* > - * String Functions > - */ > - > -static inline char *dapl_os_strdup(const char *str) > -{ > - char *ns = kmalloc(strlen(str) + 1, GFP_KERNEL); > - if (ns) > - strcpy(ns, str); > - return ns; > -} > - > -/* > * *printf format helper. We use the C string constant concatenation > * ability to define 64 bit formats, which unfortunatly are non standard > * in the C compiler world. > @@ -143,22 +130,6 @@ > #endif > > /* > - * dapl_os_mod64 > - * > - * Returne the modulo of a 64 bit number. Given that this is running > - * on a 32 bit platform, we need to use the kernel macro to prevent > - * exceptions from killing the machine. > - */ > -static inline long dapl_os_mod64(uint64_t key, unsigned long hashsize) > -{ > -#ifdef __ia64__ > - return ((uint64_t) ((key) % (hashsize))) > -#else > - return do_div(key, hashsize); > -#endif > -} > - > -/* > * Debug Functions > */ > > Index: dat-provider/dapl_hash.c > =================================================================== > --- dat-provider/dapl_hash.c (revision 2608) > +++ dat-provider/dapl_hash.c (working copy) > @@ -1,461 +0,0 @@ > -/* > - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > - * > - * This Software is licensed under one of the following licenses: > - * > - * 1) under the terms of the "Common Public License 1.0" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/cpl.php. > - * > - * 2) under the terms of the "The BSD License" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/bsd-license.php. > - * > - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > - * copy of which is available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/gpl-license.php. > - * > - * Licensee has the right to choose one of the above licenses. > - * > - * Redistributions of source code must retain the above copyright > - * notice and one of the license notices. > - * > - * Redistributions in binary form must reproduce both the above copyright > - * notice, one of the license notices in the documentation > - * and/or other materials provided with the distribution. > - */ > - > -/* > - * Provides a generic hash table with chaining. > - * > - * $Id$ > - */ > - > -#include "dapl_hash.h" > - > -/********************************************************************* > - * * > - * Structures * > - * * > - *********************************************************************/ > - > -/* > - * A hash table element > - */ > -typedef struct DAPL_HASH_ELEM { > - struct DAPL_HASH_ELEM *next_element; > - DAPL_HASH_KEY key; > - void *datum; > -} DAPL_HASH_ELEM; > - > -/* > - * The hash table > - */ > -struct dapl_hash_table { > - unsigned long num_entries; > - unsigned long tbl_size; > - DAPL_HASH_ELEM *table; > - spinlock_t lock; > - unsigned long flags; > - /* > - * statistics - we tally on insert operations, counting > - * the number of entries in the whole hash table, as > - * well as the length of chains we walk to insert. This > - * ignores empty buckets, giving us data on overall table > - * occupancy, as well as max/average chain length for > - * the buckets used. If our hash function results in > - * hot buckets, this will show it. > - */ > - uint64_t hash_tbl_inserts; /* total inserts ops */ > - uint64_t hash_tbl_max; /* max in entire table */ > - uint64_t hash_tbl_total; /* total in table */ > - uint64_t hash_chn_max; /* longest chain */ > - uint64_t hash_chn_total; /* total non-0 lenghts */ > -}; > - > -/********************************************************************* > - * * > - * Defines * > - * * > - *********************************************************************/ > - > -/* datum value in empty table slots (use 0UL or ~0UL as appropriate) */ > -#define NO_DATUM_VALUE ((void *) 0UL) > -#define NO_DATUM(value) ((value) == NO_DATUM_VALUE) > - > -/* Lookup macro (which falls back to function to rehash) */ > -#define DAPL_HASHLOOKUP( p_table, in_key, out_datum, bucket_head) \ > - do { \ > - DAPL_HASH_KEY save_key = in_key; \ > - DAPL_HASH_ELEM *element = \ > - &((p_table)->table)[DAPL_DOHASH(in_key,(p_table)->tbl_size)]; \ > - in_key = save_key; \ > - if (NO_DATUM(element->datum)) { \ > - (bucket_head) = (void *)0; \ > - } else if (element->key == (DAPL_HASH_KEY) (in_key)) { \ > - (out_datum) = element->datum; \ > - (bucket_head) = (void *)element; \ > - } else if (element->next_element) { \ > - dapl_hash_rehash(element, \ > - (in_key), \ > - (void **)&(out_datum), \ > - (DAPL_HASH_ELEM **)&(bucket_head)); \ > - } else { \ > - (bucket_head) = (void *)0; \ > - }\ > - } while (0) > - > -/********************************************************************* > - * * > - * Internal Functions * > - * * > - *********************************************************************/ > - > -/* > - * Rehash the key (used by add and lookup functions) > - * > - * Inputs: element element to rehash key > - * key, datum datum for key head > - * head head for list > - */ > -static void > -dapl_hash_rehash(DAPL_HASH_ELEM * element, > - DAPL_HASH_KEY key, void **datum, DAPL_HASH_ELEM ** head) > -{ > - /* > - * assume we looked at the contents of element already, > - * and start with the next element. > - */ > - dapl_os_assert(element->next_element); > - dapl_os_assert(!NO_DATUM(element->datum)); > - > - *head = element; > - while (1) { > - element = element->next_element; > - if (!element) { > - break; > - } > - if (element->key == key) { > - *datum = element->datum; > - return; > - } > - } > - *head = NULL; > -} > - > -/* > - * Add a new key to the hash table > - * > - * Inputs: > - * table, key and datum to be added > - * allow_dup - TRUE if dups are allowed > - * Outputs: > - * report_dup - should you care to know > - * Returns: > - * TRUE on success > - */ > -static boolean_t > -dapl_hash_add(struct dapl_hash_table *table, DAPL_HASH_KEY key, void *datum, > - boolean_t allow_dup, boolean_t *report_dup) > -{ > - void *olddatum; > - DAPL_HASH_KEY hashValue, save_key = key; > - DAPL_HASH_ELEM *found; > - boolean_t status = FALSE; > - unsigned int chain_len = 0; > - > - if (report_dup) { > - (*report_dup) = FALSE; > - } > - > - if (NO_DATUM(datum)) { > - /* > - * Reserved value used for datum > - */ > - dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - "dapl_hash_add() called with magic NO_DATA " > - "value (%p) used as datum!\n", datum); > - return FALSE; > - } > - > - DAPL_HASHLOOKUP(table, key, olddatum, found); > - if (found) { > - /* > - * key exists already > - */ > - if (report_dup) { > - *report_dup = TRUE; > - } > - > - if (!allow_dup) { > - dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - "dapl_hash_add() called with duplicate " > - "key (" F64x ")\n", key); > - return FALSE; > - } > - } > - > - hashValue = DAPL_DOHASH(key, table->tbl_size); > - key = save_key; > - if (NO_DATUM(table->table[hashValue].datum)) { > - /* > - * Empty head - just fill it in > - */ > - table->table[hashValue].key = key; > - table->table[hashValue].datum = datum; > - table->table[hashValue].next_element = NULL; > - table->num_entries++; > - status = TRUE; > - } else { > - DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, > - GFP_ATOMIC); > - /* > - * Add an element to the end of the chain > - */ > - if (newelement) { > - DAPL_HASH_ELEM *lastelement; > - newelement->key = key; > - newelement->datum = datum; > - newelement->next_element = NULL; > - for (lastelement = &table->table[hashValue]; > - lastelement->next_element; > - lastelement = lastelement->next_element) { > - /* Walk to the end of the chain */ > - chain_len++; > - } > - lastelement->next_element = newelement; > - table->num_entries++; > - status = TRUE; > - } else > - status = FALSE; > - } > - > - /* > - * Tally up our counters. chain_len is one less than current chain > - * length. > - */ > - chain_len++; > - table->hash_tbl_inserts++; > - table->hash_tbl_total += table->num_entries; > - table->hash_chn_total += chain_len; > - if (table->num_entries > table->hash_tbl_max) { > - table->hash_tbl_max = table->num_entries; > - } > - if (chain_len > table->hash_chn_max) { > - table->hash_chn_max = chain_len; > - } > - > - return status; > -} > - > -/* > - * Remove element from hash bucket > - * > - * Inputs: > - * element, key to be deleted > - * Returns: > - * TRUE on success > - */ > -static boolean_t > -dapl_hash_delete_element(DAPL_HASH_ELEM * element, > - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_datum) > -{ > - DAPL_HASH_ELEM *curelement; > - DAPL_HASH_ELEM *lastelement; > - > - lastelement = NULL; > - for (curelement = element; > - curelement; > - lastelement = curelement, curelement = curelement->next_element) { > - if (curelement->key == key) { > - if (p_datum) { > - *p_datum = curelement->datum; > - } > - if (lastelement) { > - /* > - * curelement was malloc'd; free it > - */ > - lastelement->next_element = > - curelement->next_element; > - kfree(curelement); > - } else { > - /* > - * curelement is static list head > - */ > - DAPL_HASH_ELEM *n = curelement->next_element; > - if (n) { > - /* > - * If there is a next element, copy its contents into the > - * head and free the original next element. > - */ > - curelement->key = n->key; > - curelement->datum = n->datum; > - curelement->next_element = > - n->next_element; > - kfree(n); > - } else { > - curelement->datum = NO_DATUM_VALUE; > - } > - } > - break; > - } > - } > - > - return (curelement != NULL ? TRUE : FALSE); > -} > - > -/********************************************************************* > - * * > - * External Functions * > - * * > - *********************************************************************/ > - > -/* > - * Create a new hash table with at least 'table_size' hash buckets. > - */ > -u32 dapl_hash_create(int table_size, struct dapl_hash_table **pp_table) > -{ > - struct dapl_hash_table *p_table; > - int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); > - u32 dat_status = DAT_SUCCESS; > - int i; > - > - dapl_os_assert(pp_table); > - > - /* Allocate hash table */ > - p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); > - if (!p_table) { > - dat_status = > - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); > - goto bail; > - } > - > - /* Init hash table, allocate and init and buckets */ > - memset(p_table, 0, sizeof *p_table); > - p_table->tbl_size = table_size; > - p_table->table = kmalloc(table_length, GFP_ATOMIC); > - if (!p_table->table) { > - kfree(p_table); > - dat_status = > - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); > - goto bail; > - } > - > - spin_lock_init(&p_table->lock); > - for (i = 0; i < table_size; i++) { > - p_table->table[i].datum = NO_DATUM_VALUE; > - p_table->table[i].key = 0; > - p_table->table[i].next_element = NULL; > - } > - > - *pp_table = p_table; > - > -bail: > - return DAT_SUCCESS; > -} > - > -/* > - * Destroy a hash table > - */ > -u32 dapl_hash_free(struct dapl_hash_table *p_table) > -{ > - dapl_os_assert(p_table && p_table->table); > - > - /* no need to destroy p_table->lock */ > - kfree(p_table->table); > - kfree(p_table); > - > - return DAT_SUCCESS; > -} > - > -/* > - * Returns the number of elements stored in the table > - */ > - > -u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size) > -{ > - dapl_os_assert(p_table && p_size); > - > - *p_size = p_table->num_entries; > - > - return DAT_SUCCESS; > -} > - > -/* > - * Inserts the specified data into the table with the given key. > - * Duplicates are not expected, and return in error, having done nothing. > - */ > - > -u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, > - DAPL_HASH_DATA data) > -{ > - u32 dat_status = DAT_SUCCESS; > - > - dapl_os_assert(p_table); > - > - spin_lock_irqsave(&p_table->lock, p_table->flags); > - if (!dapl_hash_add(p_table, key, data, FALSE, NULL)) { > - dat_status = > - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); > - } > - spin_unlock_irqrestore(&p_table->lock, p_table->flags); > - > - return dat_status; > -} > - > -/* > - * Searches for the given key. If found, > - * DAT_SUCCESS is returned and the associated > - * data is returned in the DAPL_HASH_DATA > - * pointer if that pointer is not NULL. > - */ > -u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, > - DAPL_HASH_DATA *p_data) > -{ > - u32 dat_status; > - void *olddatum; > - DAPL_HASH_ELEM *found; > - > - dapl_os_assert(p_table); > - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, 0); > - > - spin_lock_irqsave(&p_table->lock, p_table->flags); > - DAPL_HASHLOOKUP(p_table, key, olddatum, found); > - spin_unlock_irqrestore(&p_table->lock, p_table->flags); > - > - if (found) { > - if (p_data) { > - *p_data = olddatum; > - } > - dat_status = DAT_SUCCESS; > - } > - > - return dat_status; > -} > - > -u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, > - DAPL_HASH_DATA *p_data) > -{ > - u32 dat_status; > - DAPL_HASH_KEY hashValue, save_key = key; > - > - dapl_os_assert(p_table); > - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, 0); > - > - if (p_table->num_entries == 0) { > - dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - "dapl_hash_remove () called on empty hash table!\n"); > - return dat_status; > - } > - > - hashValue = DAPL_DOHASH(key, p_table->tbl_size); > - key = save_key; > - spin_lock_irqsave(&p_table->lock, p_table->flags); > - if (dapl_hash_delete_element(&p_table->table[hashValue], key, p_data)) { > - p_table->num_entries--; > - dat_status = DAT_SUCCESS; > - } > - spin_unlock_irqrestore(&p_table->lock, p_table->flags); > - > - return dat_status; > -} > Index: dat-provider/dapl.h > =================================================================== > --- dat-provider/dapl.h (revision 2608) > +++ dat-provider/dapl.h (working copy) > @@ -83,9 +83,6 @@ > > typedef void (*ib_async_handler_t) (struct ib_event *, void *); > > -typedef u64 DAPL_HASH_KEY; > -typedef void *DAPL_HASH_DATA; > - > /********************************************************************* > * * > * Structures * > @@ -124,8 +121,6 @@ > u8 port_num; > struct ib_device *ib_hca_handle; > struct ib_cq *null_cq; /* CQ with 0 entries */ > - /* Memory Subsystem Support */ > - struct dapl_hash_table *lmr_hash_table; > /* Limits & useful HCA attributes */ > struct dat_ia_attr ia_attr; > struct dat_ep_attr ep_attr; > @@ -542,6 +537,7 @@ > struct dat_rmr_param *rmr_args); > > extern u32 dapl_rmr_bind(struct dat_rmr *rmr, > + struct dat_lmr *lmr, > const struct dat_lmr_triplet *lmr_triplet, > enum dat_mem_priv_flags mem_priv, > struct dat_ep *ep, > Index: dat-provider/dapl_hash.h > =================================================================== > --- dat-provider/dapl_hash.h (revision 2608) > +++ dat-provider/dapl_hash.h (working copy) > @@ -1,80 +0,0 @@ > -/* > - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > - * > - * This Software is licensed under one of the following licenses: > - * > - * 1) under the terms of the "Common Public License 1.0" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/cpl.php. > - * > - * 2) under the terms of the "The BSD License" a copy of which is > - * available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/bsd-license.php. > - * > - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > - * copy of which is available from the Open Source Initiative, see > - * http://www.opensource.org/licenses/gpl-license.php. > - * > - * Licensee has the right to choose one of the above licenses. > - * > - * Redistributions of source code must retain the above copyright > - * notice and one of the license notices. > - * > - * Redistributions in binary form must reproduce both the above copyright > - * notice, one of the license notices in the documentation > - * and/or other materials provided with the distribution. > - */ > - > -/* > - * $Id$ > - */ > - > -#ifndef DAPL_HASH_H > -#define DAPL_HASH_H > - > -#include "dapl.h" > - > -/********************************************************************* > - * * > - * Defines * > - * * > - *********************************************************************/ > - > -/* > - * Hash table size. > - * > - * Default is small; use the larger sample values for hash tables > - * known to be heavily used. The sample values chosen are the > - * largest primes below 2^8, 2^9, and 2^10. > - */ > -#define DAPL_DEF_HASHSIZE 251 > -#define DAPL_MED_HASHSIZE 509 > -#define DAPL_LRG_HASHSIZE 1021 > - > -#define DAPL_HASH_TABLE_DEFAULT_CAPACITY DAPL_DEF_HASHSIZE > - > -/* The hash function */ > -#define DAPL_DOHASH(key,hashsize) dapl_os_mod64(key,hashsize) > - > -/********************************************************************* > - * * > - * Function Prototypes * > - * * > - *********************************************************************/ > - > -extern u32 dapl_hash_create(int capacity, struct dapl_hash_table **pp_table); > - > -extern u32 dapl_hash_free(struct dapl_hash_table *p_table); > - > -extern u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size); > - > -extern u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, > - DAPL_HASH_DATA data); > - > -extern u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, > - DAPL_HASH_DATA *p_data); > - > -extern u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, > - DAPL_HASH_DATA *p_data); > - > -#endif /* DAPL_HASH_H */ > Index: dat-provider/dapl_provider.c > =================================================================== > --- dat-provider/dapl_provider.c (revision 2608) > +++ dat-provider/dapl_provider.c (working copy) > @@ -34,7 +34,6 @@ > #include > > #include "dapl.h" > -#include "dapl_hca_util.h" > #include "dapl_provider.h" > #include "dapl_util.h" > #include "dapl_openib_util.h" > @@ -247,6 +246,23 @@ > provider_info->is_thread_safe = DAPL_THREADSAFE; > } > > +struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) > +{ > + struct dapl_hca *hca; > + int malloc_size = sizeof *hca + strlen(name) + 1; > + hca = kmalloc(malloc_size, GFP_ATOMIC); > + if (hca) { > + memset(hca, 0, malloc_size); > + spin_lock_init(&hca->lock); > + INIT_LIST_HEAD(&hca->ia_list); > + hca->name = (char *)hca + sizeof *hca; > + strcpy(hca->name, name); > + hca->ib_hca_handle = device; > + hca->port_num = port; > + } > + return hca; > +} > + > static void dapl_add_port(struct ib_device *device, u8 port) > { > struct dat_provider_info provider_info; > @@ -306,7 +322,7 @@ > (void)dapl_provider_list_remove(provider_info.ia_name); > > if (NULL != hca) > - dapl_hca_free(hca); > + kfree(hca); > } > } > > @@ -339,7 +355,7 @@ > provider_info.ia_name); > } > > - dapl_hca_free(provider->extension); > + kfree(provider->extension); > > dapl_provider_list_remove(provider_info.ia_name); > } > Index: dat/dat.h > =================================================================== > --- dat/dat.h (revision 2608) > +++ dat/dat.h (working copy) > @@ -1153,7 +1153,8 @@ > > typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); > > -typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, > +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, struct dat_lmr *, > + const struct dat_lmr_triplet *, > enum dat_mem_priv_flags, struct dat_ep *, > DAT_RMR_COOKIE, enum dat_completion_flags, > DAT_RMR_CONTEXT *); > @@ -1638,14 +1639,15 @@ > } > > static inline u32 dat_rmr_bind(struct dat_rmr * rmr, > - const struct dat_lmr_triplet *iov, > + struct dat_lmr *lmr, > + const struct dat_lmr_triplet *iov, > enum dat_mem_priv_flags mem_flags, > struct dat_ep *ep, DAT_RMR_COOKIE cookie, > enum dat_completion_flags comp_flags, > DAT_RMR_CONTEXT *context) > { > - return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, iov, mem_flags, ep, > - cookie, comp_flags, context); > + return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, lmr, iov, mem_flags, > + ep, cookie, comp_flags, context); > } > > static inline u32 dat_rmr_free(struct dat_rmr * rmr) > > -- > Itamar > From roland at topspin.com Thu Jun 16 08:58:42 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 08:58:42 -0700 Subject: [openib-general] Re: [IPoIB] Add support for MTU module parameter In-Reply-To: <1118925570.4506.1052.camel@localhost.localdomain> (Hal Rosenstock's message of "16 Jun 2005 08:40:19 -0400") References: <1118925570.4506.1052.camel@localhost.localdomain> Message-ID: <52br66cmz1.fsf@topspin.com> Hal> [IPoIB] Add support for MTU module parameter. This is so Hal> there can be a non default MTU at boot up if the Hal> administrator so desires (prior to being able to invoke Hal> ifconfig). Why is it useful to set the MTU before being able to invoke ifconfig? The network interface can't do anything useful until it is brought up. - R. From roland at topspin.com Thu Jun 16 09:02:49 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 09:02:49 -0700 Subject: [openib-general] Re: [IPoIB] Add support for MTU module parameter In-Reply-To: <52br66cmz1.fsf@topspin.com> (Roland Dreier's message of "Thu, 16 Jun 2005 08:58:42 -0700") References: <1118925570.4506.1052.camel@localhost.localdomain> <52br66cmz1.fsf@topspin.com> Message-ID: <5264wecms6.fsf@topspin.com> Roland> Why is it useful to set the MTU before being able to Roland> invoke ifconfig? The network interface can't do anything Roland> useful until it is brought up. Also, if this is useful for some application (I guess I can imagine booting with NFS root), shouldn't the MTU setting be part of the core network, rather than an IPoIB-only option? - R. From jlentini at netapp.com Thu Jun 16 09:15:14 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 16 Jun 2005 12:15:14 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: anal lawyer update In-Reply-To: <1118855512.29970.6.camel@duffman> References: <1118855512.29970.6.camel@duffman> Message-ID: Committed in 2640. On Wed, 15 Jun 2005, Tom Duffy wrote: tduffy> James, the Sun lawyers are getting anal and want any code that I have tduffy> modified substantially on OpenIB to have a Sun copyright on it. I have tduffy> gone through the DAPL stuff and grepped the svn logs for sun.com and tduffy> updated those files with a Sun copyright. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/test/dapltest/test/dapl_performance_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_performance_client.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_performance_client.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_performance_client.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_transaction_stats.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_transaction_stats.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_transaction_stats.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_cnxn.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_cnxn.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_cnxn.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_server.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_server.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_server.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_thread.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_thread.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_thread.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_test_data.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_test_data.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_test_data.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_mem.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_mem.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_mem.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_limit.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_server_info.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_server_info.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_server_info.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_performance_server.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_performance_server.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_performance_server.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_connmgt.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_connmgt.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_memlist.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_memlist.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_memlist.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_bpool.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_test_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_util.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_client.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_endpoint.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_transaction_util.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_pz.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_hwconn.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_client_info.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_client_info.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_client_info.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_server_cmd.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_server_cmd.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_server_cmd.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_tdep.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_tdep.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_tdep.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_performance_test.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_global.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_global.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_global.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_limit_cmd.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_limit_cmd.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_limit_cmd.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_fft_cmd.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_fft_cmd.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_fft_cmd.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_proto.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_quit_cmd.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_quit_cmd.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_quit_cmd.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_test_data.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_test_data.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_test_data.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_transaction_cmd.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_transaction_cmd.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_transaction_cmd.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_bpool.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_fft_util.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_performance_stats.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_performance_stats.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_performance_stats.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_server_info.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_server_info.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_server_info.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_params.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_params.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_params.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_performance_cmd.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_performance_cmd.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_performance_cmd.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/include/dapl_common.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_common.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/include/dapl_common.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_qos_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_qos_util.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_qos_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_netaddr.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_netaddr.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_netaddr.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_params.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_params.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_params.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_test_data.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_test_data.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_test_data.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_server_cmd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_server_cmd.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_server_cmd.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h (revision 2622) tduffy> +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/common/dapl_endian.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/common/dapl_endian.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/common/dapl_endian.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/common/dapl_global.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/common/dapl_global.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/common/dapl_global.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/kdapl/kdapl_module.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/kdapl/kdapl_module.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/kdapl/kdapl_module.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2622) tduffy> +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) tduffy> @@ -1,4 +1,7 @@ tduffy> -/* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. * tduffy> +/* tduffy> + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> + * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> Index: linux-kernel/dat-provider/dapl_provider.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_provider.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_provider.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_openib_cm.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_cm.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_openib_cm.h (working copy) tduffy> @@ -3,6 +3,7 @@ tduffy> * Copyright (c) 2005 Intel Corporation. All rights reserved. tduffy> * Copyright (c) 2004-2005, Mellanox Technologies, Inc. All rights reserved. tduffy> * Copyright (c) 2003 Topspin Corporation. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_srq.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_srq.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_srq.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_cookie.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cookie.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -1,6 +1,7 @@ tduffy> /* tduffy> * Copyright (c) 1999-2005, Mellanox Technologies, Inc. All rights reserved. tduffy> * Copyright (c) 2005 Voltaire Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_openib_qp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) tduffy> @@ -1,6 +1,7 @@ tduffy> /* tduffy> * Copyright (c) 1999-2005, Mellanox Technologies, Inc. All rights reserved. tduffy> * Copyright (c) 2005 Voltaire Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_cookie.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cookie.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_cookie.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_openib_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_util.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_openib_util.h (working copy) tduffy> @@ -1,6 +1,7 @@ tduffy> /* tduffy> * Copyright (c) 1999-2005, Mellanox Technologies, Inc. All rights reserved. tduffy> * Copyright (c) 2005 Voltaire Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_ring_buffer_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ring_buffer_util.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_ring_buffer_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_openib_dto.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_dto.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_openib_dto.h (working copy) tduffy> @@ -1,6 +1,7 @@ tduffy> /* tduffy> * Copyright (c) 1999-2005, Mellanox Technologies, Inc. All rights reserved. tduffy> * Copyright (c) 2005 Voltaire Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_hca_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_hca_util.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_hca_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_pz.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_pz.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_pz.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_ia.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_ia.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_lmr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_lmr.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_lmr.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_ring_buffer_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ring_buffer_util.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_ring_buffer_util.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_hca_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_hca_util.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_hca_util.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_ia.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_ia.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_rmr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_rmr.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_ep.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_ep.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_cr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cr.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_cr.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_ep.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_ep.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_cr.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cr.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_cr.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_util.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_evd.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_util.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_sp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_sp.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_hash.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_hash.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_hash.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_evd.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_evd.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_sp.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_sp.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_hash.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_hash.h (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_hash.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_provider.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_provider.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_provider.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat-provider/dapl_openib_cm.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2622) tduffy> +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) tduffy> @@ -3,6 +3,7 @@ tduffy> * Copyright (c) 2005 Intel Corporation. All rights reserved. tduffy> * Copyright (c) 2004-2005, Mellanox Technologies, Inc. All rights reserved. tduffy> * Copyright (c) 2003 Topspin Corporation. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat/dictionary.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/dictionary.c (revision 2622) tduffy> +++ linux-kernel/dat/dictionary.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat/dr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/dr.c (revision 2622) tduffy> +++ linux-kernel/dat/dr.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat/core.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/core.c (revision 2622) tduffy> +++ linux-kernel/dat/core.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat/api.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/api.c (revision 2622) tduffy> +++ linux-kernel/dat/api.c (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat/dr.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/dr.h (revision 2622) tduffy> +++ linux-kernel/dat/dr.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat/dictionary.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/dictionary.h (revision 2622) tduffy> +++ linux-kernel/dat/dictionary.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat/dat.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/dat.h (revision 2622) tduffy> +++ linux-kernel/dat/dat.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> Index: linux-kernel/dat/core.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/core.h (revision 2622) tduffy> +++ linux-kernel/dat/core.h (working copy) tduffy> @@ -1,5 +1,6 @@ tduffy> /* tduffy> * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. tduffy> * tduffy> * This Software is licensed under one of the following licenses: tduffy> * tduffy> From halr at voltaire.com Thu Jun 16 09:58:36 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 Jun 2005 12:58:36 -0400 Subject: [openib-general] kdapl locking problem In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35C9C@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35C9C@mtvex01.mtv.mtl.com> Message-ID: <1118941115.4506.21.camel@localhost.localdomain> On Thu, 2005-06-16 at 04:39, Itamar Rabenstein wrote: > Hi Hal, > I am trying to understand what is going here and i still dont see how this > happan . > > This prints are only set in UP mode .(is this your system UP?) Yes. > the code is (function: dapl_evd_connection_callback): > spin_lock_irqsave(&ep->common.lock, ep->common.flags); > case on the event type > disconnect: dapl_ib_disconnect_clean(ep, TRUE); > spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); > > from some reason in the middle between the lock and the unlock there is a > call to consumer > function (dat_ep_disconnetc) that try to disconnect the same ep and the lock > fail. Could this be a "local" disconnect race of some sort ? > the evd_cb function is either an interupt from the CM so i dont see how the > consumer can call > dat_ib_disconnect in the middle > or the user called twice to dat_ib_disconnect on the same ep and youe kernel > give preemption CONFIG_PREEMPT is not set in my kernel config. > i dont understand both (;-) > > can you try to run it with some debug? > at least ot know who called to dapl_evd_connection_callback ? All calls to dapl_evd_connection_callback are out of the CM except one case in dapl_ep_disconnect. In the case of dapl_ep_disconnect, the lock is obtained in dapl_ep_disconnect before the connection callback routine would/might have been called. One instance: Jun 16 12:47:33 localhost kernel: dapl_ep_disconnect: dapl_evd_connection_callback EP 0xc91e5bf8 CM ID 0x00000000 EP common lock 0xc91e5c08 Jun 16 12:47:34 localhost kernel: dapl_ep_disconnect: dapl_evd_connection_callback EP 0xc987ebf8 CM ID 0x00000000 EP common lock 0xc987ec08 Jun 16 12:47:34 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_ep.c:1110: spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:ce609c08) already locked by drivers/infiniband/ulp/dat-provider/dapl_cr.c/501 Jun 16 12:47:34 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_cr.c:512: spin_unlock Another instance: Jun 16 12:55:11 localhost kernel: dapl_ep_disconnect: dapl_evd_connection_callback EP 0xc64b1bf8 CM ID 0x00000000 EP common lock 0xc64b1c08 Jun 16 12:55:12 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_ep.c:1110: spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:ce5a6c08) already locked by drivers/infiniband/ulp/dat-provider/dapl_cr.c/501 Jun 16 12:55:12 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_cr.c:512: spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:ce5a6c08) not locked Jun 16 12:55:12 localhost kernel: dapl_ep_disconnect: dapl_evd_connection_callback EP 0xc07debf8 CM ID 0x00000000 EP common lock 0xc07dec08 Yet another instance: Jun 16 12:55:12 localhost kernel: dapl_cm_active_cb_handler: TIMEWAIT EXIT dapl_evd_connection_callback EP 0xce609bf8 CM ID 0xc82dcdf8 EP common lock 0xce609c08 Jun 16 12:55:12 localhost kernel: dapl_ep_disconnect: dapl_evd_connection_callback EP 0xcbeb8bf8 CM ID 0x00000000 EP common lock 0xcbeb8c08 Jun 16 12:55:12 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_ep.c:1110: spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:cf35bc08) already locked by drivers/infiniband/ulp/dat-provider/dapl_cr.c/501 Jun 16 12:55:12 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_cr.c:512: spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:cf35bc08) not locked -- Hal > Itamar > > > > --Original Message-- > > From: Hal Rosenstock [mailto:halr at voltaire.com] > > Sent: Tuesday, June 14, 2005 8:37 PM > > To: James Lentini > > Cc: openib-general at openib.org > > Subject: [openib-general] kdapl locking problem > > > > > > Hi, > > > > When running in loopback mode (client and server on same > > machine (x86)): > > kdapltest -T T -s -D mthca0a -d -t 2 -w 8 -i 20 > > client SR server SR > > I see the following locking problem: > > > > Jun 14 13:30:08 localhost kernel: > > drivers/infiniband/ulp/dat-provider/dapl_ep.c:1111: > > spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c44b1c > > 18) already locked by > > drivers/infiniband/ulp/dat-provider/dapl_evd.c/756 > > Jun 14 13:30:08 localhost kernel: > > drivers/infiniband/ulp/dat-provider/dapl_evd.c:797: > > spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c44b > > 1c18) not locked > > > > -- Hal > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general From libor at topspin.com Thu Jun 16 11:12:55 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 16 Jun 2005 11:12:55 -0700 Subject: [openib-general] SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050615213047.GM577@kalmia.hozed.org>; from hozer@hozed.org on Wed, Jun 15, 2005 at 04:30:47PM -0500 References: <20050615213047.GM577@kalmia.hozed.org> Message-ID: <20050616111255.A8268@topspin.com> On Wed, Jun 15, 2005 at 04:30:47PM -0500, Troy Benjegerdes wrote: > I'm getting a 'SDP: device mthca0 does nto support fast memory regions' > error on PPC64 systems.. Is there something that needs to be done for > PPC, or could I have an older version of the mthca module hanging > around? I'm not sure about the state or plans for FMR support on PPC64, Roland will have a better idea. I have not added support for AIO when FMRs are disabled, but if you are not using AIO, then the following patch should be sufficient. Currently the AIO code path assumes that there is a valid fmr_pool, I'll need to fix this so that we gracefully fall back to copy mode when fmr registration fails. -Libor Index: sdp_conn.c =================================================================== --- sdp_conn.c (revision 2628) +++ sdp_conn.c (working copy) @@ -1857,9 +1857,8 @@ */ hca->fmr_pool = ib_create_fmr_pool(hca->pd, &fmr_param_s); if (IS_ERR(hca->fmr_pool)) { - sdp_warn("Error <%ld> creating HCA <%s> fast memory pool", - PTR_ERR(hca->fmr_pool), device->name); - goto error; + sdp_warn("Warning, could not creating HCA <%s> FMR pool <%ld>", + device->name, PTR_ERR(hca->fmr_pool)); } /* * port allocation From tduffy at sun.com Thu Jun 16 11:35:57 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 16 Jun 2005 11:35:57 -0700 Subject: [openib-general] [PATCH] kDAPL: remove the remaining typedef enums Message-ID: <1118946957.21846.14.camel@duffman> This patch removes the last remaining typedef enums (minus bool which /might/ justify it). Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel/dat-provider/dapl_cookie.c (revision 2640) +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) @@ -262,7 +262,8 @@ bail: * DAT_INSUFFICIENT_EMPTY * */ -u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, DAPL_DTO_TYPE type, +u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, + enum dapl_dto_type type, DAT_DTO_COOKIE user_cookie, struct dapl_cookie **cookie_ptr) { Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2640) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -523,7 +523,7 @@ u32 dapl_ib_mw_unbind(struct dapl_rmr *r * */ u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, - DAPL_ASYNC_HANDLER_TYPE handler_type, + enum async_handler_type handler_type, ib_async_handler_t callback, void *context) { int ib_status = 0; Index: linux-kernel/dat-provider/dapl_cookie.h =================================================================== --- linux-kernel/dat-provider/dapl_cookie.h (revision 2640) +++ linux-kernel/dat-provider/dapl_cookie.h (working copy) @@ -48,7 +48,8 @@ extern u32 dapl_rmr_cookie_alloc(struct struct dapl_cookie **cookie_ptr); extern u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, - DAPL_DTO_TYPE type, DAT_DTO_COOKIE user_cookie, + enum dapl_dto_type type, + DAT_DTO_COOKIE user_cookie, struct dapl_cookie **cookie_ptr); extern void dapl_cookie_dealloc(struct dapl_cookie_buffer *buffer, Index: linux-kernel/dat-provider/dapl_openib_util.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.h (revision 2640) +++ linux-kernel/dat-provider/dapl_openib_util.h (working copy) @@ -36,11 +36,11 @@ #include "ib_verbs.h" #include "ib_cm.h" -typedef enum async_handler_type { +enum async_handler_type { DAPL_ASYNC_UNAFILIATED, DAPL_ASYNC_CQ_ERROR, DAPL_ASYNC_QP_ERROR -} DAPL_ASYNC_HANDLER_TYPE; +}; /* alignment for the bus transfer from the HCA/IB chip to the main memory */ #define DAPL_OPTIMAL_ALIGNMENT 256 @@ -76,7 +76,7 @@ u32 dapl_ib_accept_connection(struct dap u32 dapl_ib_reject_connection(struct dapl_cm_ctx *cm_ctx); u32 dapl_ib_setup_async_callback(struct dapl_ia *ia, - DAPL_ASYNC_HANDLER_TYPE handler_type, + enum async_handler_type handler_type, ib_async_handler_t callback, void *context); u32 dapl_ib_cq_alloc(struct dapl_ia *ia, struct dapl_evd *evd, int *cqlen, Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2640) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -168,7 +168,7 @@ static u32 dapl_ep_post_send_req(struct DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags, - DAPL_DTO_TYPE dto_type, + enum dapl_dto_type dto_type, enum ib_wr_opcode op_type) { struct dapl_ep *ep_ptr; Index: linux-kernel/dat-provider/dapl_util.c =================================================================== --- linux-kernel/dat-provider/dapl_util.c (revision 2640) +++ linux-kernel/dat-provider/dapl_util.c (working copy) @@ -36,7 +36,7 @@ #ifdef DAPL_DBG -void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) +void dapl_dbg_log(enum dapl_dbg_type type, const char *fmt, ...) { char buf[1024]; va_list args; Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2640) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -557,7 +557,7 @@ static void dapl_evd_wc_to_event(struct /* We can not check the following assert for now */ if (dto_status == DAT_DTO_SUCCESS) { enum ib_wc_opcode ib_opcode = wc->opcode; - DAPL_DTO_TYPE dto_type = cookie->val.dto.type; + enum dapl_dto_type dto_type = cookie->val.dto.type; dapl_os_assert((ib_opcode == IB_WC_SEND && dto_type == DAPL_DTO_TYPE_SEND) || (ib_opcode == IB_WC_RECV && @@ -877,7 +877,7 @@ void dapl_evd_connection_callback(struct static void dapl_evd_dto_callback(struct ib_cq *cq, void *user_context) { struct dapl_evd *evd; - DAPL_EVD_STATE state; + enum dapl_evd_state state; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, "dapl_evd_dto_callback(%p, %p)\n", @@ -888,7 +888,7 @@ static void dapl_evd_dto_callback(struct dapl_os_assert(evd->cq == cq); /* Read once. */ - state = *(volatile DAPL_EVD_STATE *)&evd->evd_state; + state = *(volatile enum dapl_evd_state *)&evd->evd_state; dapl_dbg_log(DAPL_DBG_TYPE_EVD, "dapl_evd_dto_callback: CQ %p, state %x\n", Index: linux-kernel/dat-provider/dapl_sp.c =================================================================== --- linux-kernel/dat-provider/dapl_sp.c (revision 2640) +++ linux-kernel/dat-provider/dapl_sp.c (working copy) @@ -624,7 +624,7 @@ u32 dapl_psp_free(struct dat_sp *dat_psp { struct dapl_ia *ia; struct dapl_sp *sp; - DAPL_SP_STATE save_state; + enum dapl_sp_state save_state; u32 status = DAT_SUCCESS; sp = (struct dapl_sp *)dat_psp; Index: linux-kernel/dat-provider/dapl_util.h =================================================================== --- linux-kernel/dat-provider/dapl_util.h (revision 2640) +++ linux-kernel/dat-provider/dapl_util.h (working copy) @@ -167,7 +167,7 @@ static inline long dapl_os_mod64(uint64_ * Use these bits to enable various tracing/debug options. Each bit * represents debugging in a particular subsystem or area of the code. */ -typedef enum { +enum dapl_dbg_type { DAPL_DBG_TYPE_ERR = 0x0001, DAPL_DBG_TYPE_WARN = 0x0002, DAPL_DBG_TYPE_EVD = 0x0004, @@ -180,12 +180,12 @@ typedef enum { DAPL_DBG_TYPE_RTN = 0x0200, DAPL_DBG_TYPE_EXCEPTION = 0x0400, DAPL_DBG_TYPE_SRQ = 0x0800 -} DAPL_DBG_TYPE; +}; typedef int DAPL_DBG_MASK; #if defined(DAPL_DBG) -extern void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...); +extern void dapl_dbg_log(enum dapl_dbg_type type, const char *fmt, ...); #else /* !DAPL_DBG */ #define dapl_dbg_log(...) #endif /* DAPL_DBG */ Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2640) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -49,17 +49,17 @@ * * *********************************************************************/ -typedef enum dapl_evd_state { +enum dapl_evd_state { DAPL_EVD_STATE_OPEN, DAPL_EVD_STATE_DEAD = 0xDEAD -} DAPL_EVD_STATE; +}; -typedef enum dapl_evd_completion { +enum dapl_evd_completion { DAPL_EVD_STATE_INIT, DAPL_EVD_STATE_SOLICITED_WAIT, DAPL_EVD_STATE_THRESHOLD, DAPL_EVD_STATE_UNSIGNALLED -} DAPL_EVD_COMPLETION; +}; /********************************************************************* * * @@ -161,7 +161,7 @@ struct dapl_evd { struct dat_evd evd; struct dapl_common common; struct list_head list; - DAPL_EVD_STATE evd_state; + enum dapl_evd_state evd_state; enum dat_evd_flags evd_flags; /* Derived from evd_flags; see dapls_evd_internal_create. */ @@ -265,19 +265,19 @@ struct dapl_rmr { }; /* SP types, indicating the state and queue */ -typedef enum dapl_sp_state { +enum dapl_sp_state { DAPL_SP_STATE_FREE, DAPL_SP_STATE_PSP_LISTENING, DAPL_SP_STATE_PSP_PENDING, DAPL_SP_STATE_RSP_LISTENING, DAPL_SP_STATE_RSP_PENDING -} DAPL_SP_STATE; +}; struct dapl_sp { struct dat_sp sp; struct dapl_common common; struct list_head list; - DAPL_SP_STATE state; /* type and queue of the SP */ + enum dapl_sp_state state; /* type and queue of the SP */ /* PSP/RSP PARAM fields */ struct dat_ia *ia; @@ -313,22 +313,22 @@ struct dapl_cr { struct dapl_sp *sp; }; -typedef enum dapl_dto_type { +enum dapl_dto_type { DAPL_DTO_TYPE_SEND, DAPL_DTO_TYPE_RECV, DAPL_DTO_TYPE_RDMA_WRITE, DAPL_DTO_TYPE_RDMA_READ, -} DAPL_DTO_TYPE; +}; -typedef enum dapl_cookie_type { +enum dapl_cookie_type { DAPL_COOKIE_TYPE_NULL, DAPL_COOKIE_TYPE_DTO, DAPL_COOKIE_TYPE_RMR, -} DAPL_COOKIE_TYPE; +}; /* struct dapl_dto_cookie used as context for DTO WQEs */ struct dapl_dto_cookie { - DAPL_DTO_TYPE type; + enum dapl_dto_type type; DAT_DTO_COOKIE cookie; int size; /* used for SEND and RDMA write */ }; @@ -341,7 +341,7 @@ struct dapl_rmr_cookie { /* struct dapl_cookie used as context for WQEs */ struct dapl_cookie { - DAPL_COOKIE_TYPE type; /* Must be first, to define struct. */ + enum dapl_cookie_type type; /* Must be first, to define struct. */ struct dapl_ep *ep; int index; union { Index: linux-kernel/dat/core.c =================================================================== --- linux-kernel/dat/core.c (revision 2640) +++ linux-kernel/dat/core.c (working copy) @@ -43,7 +43,7 @@ static DAT_DBG_MASK g_dbg_mask = DAT_DBG module_param_named(dbg_mask, g_dbg_mask, int, 0644); MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug message types."); -void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...) +void dat_dbg_print(enum dat_dbg_type type, const char *fmt, ...) { static char buf[1024]; Index: linux-kernel/dat/core.h =================================================================== --- linux-kernel/dat/core.h (revision 2640) +++ linux-kernel/dat/core.h (working copy) @@ -45,17 +45,17 @@ #define DAT_IS_BAD_POINTER(p) ( NULL == (p) ) -typedef enum { +enum dat_dbg_type { DAT_DBG_TYPE_ERROR = (1 << 0), DAT_DBG_TYPE_GENERIC = (1 << 1), DAT_DBG_TYPE_DR = (1 << 2), DAT_DBG_TYPE_PROVIDER_API = (1 << 3), DAT_DBG_TYPE_CONSUMER_API = (1 << 4), DAT_DBG_TYPE_ALL = 0x1f -} DAT_DBG_TYPE; +}; typedef int DAT_DBG_MASK; -extern void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...); +extern void dat_dbg_print(enum dat_dbg_type type, const char *fmt, ...); #endif /* CORE_H */ From halr at voltaire.com Thu Jun 16 11:57:23 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 Jun 2005 14:57:23 -0400 Subject: [openib-general] SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050615213047.GM577@kalmia.hozed.org> References: <20050615213047.GM577@kalmia.hozed.org> Message-ID: <1118948242.4506.34.camel@localhost.localdomain> On Wed, 2005-06-15 at 17:30, Troy Benjegerdes wrote: > I'm getting a 'SDP: device mthca0 does nto support fast memory regions' > error on PPC64 systems.. Is there something that needs to be done for > PPC, or could I have an older version of the mthca module hanging > around? Not sure about any PPC64 issues with FMR but is the DDR hidden ? -- Hal From roland at topspin.com Thu Jun 16 12:01:26 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 12:01:26 -0700 Subject: [openib-general] SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050616111255.A8268@topspin.com> (Libor Michalek's message of "Thu, 16 Jun 2005 11:12:55 -0700") References: <20050615213047.GM577@kalmia.hozed.org> <20050616111255.A8268@topspin.com> Message-ID: <52ekb2azy1.fsf@topspin.com> Libor> I'm not sure about the state or plans for FMR support on Libor> PPC64, Roland will have a better idea. There's no reason it shouldn't work with the latest code. I've not tried it, but the only reason that mthca would not provide the FMR functions would be if the FW is burned with "DDR hidden" (ie no BAR for the HCA-attached memory -- you won't see the third 128MB BAR in lspci). Otherwise, mthca will try to provide FMRs -- ie if they don't work on ppc64, you'll get a crash rather than a nice error. - R. From mst at mellanox.co.il Thu Jun 16 12:03:55 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 16 Jun 2005 22:03:55 +0300 Subject: [openib-general] Re: SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050616111255.A8268@topspin.com> References: <20050615213047.GM577@kalmia.hozed.org> <20050616111255.A8268@topspin.com> Message-ID: <20050616190355.GB32088@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: SDP: device mthca0 does not support fast memory regions > > On Wed, Jun 15, 2005 at 04:30:47PM -0500, Troy Benjegerdes wrote: > > I'm getting a 'SDP: device mthca0 does nto support fast memory regions' > > error on PPC64 systems.. Is there something that needs to be done for > > PPC, or could I have an older version of the mthca module hanging > > around? > > I'm not sure about the state or plans for FMR support on PPC64, Roland > will have a better idea. Should just work AFAIK. But maybe your card hides the onboard memory? Try lspci -vv to see if the card has the huge bar (100Mbyte or more) corresponding to that memory. > I have not added support for AIO when FMRs are disabled, but if you are > not using AIO, then the following patch should be sufficient. Currently > the AIO code path assumes that there is a valid fmr_pool, I'll need to > fix this so that we gracefully fall back to copy mode when fmr registration > fails. > > -Libor Thats a good idea anyway, since there may be other reasons for failure: for example we can run out of locked memory if the mlock rlimit is low. Should fall back on bcopy in that case, as well. > Index: sdp_conn.c > =================================================================== > --- sdp_conn.c (revision 2628) > +++ sdp_conn.c (working copy) > @@ -1857,9 +1857,8 @@ > */ > hca->fmr_pool = ib_create_fmr_pool(hca->pd, &fmr_param_s); > if (IS_ERR(hca->fmr_pool)) { > - sdp_warn("Error <%ld> creating HCA <%s> fast memory pool", > - PTR_ERR(hca->fmr_pool), device->name); > - goto error; > + sdp_warn("Warning, could not creating HCA <%s> FMR pool <%ld>", > + device->name, PTR_ERR(hca->fmr_pool)); could not creating -> unable to create > } > /* > * port allocation -- MST From rabenstein.si at gmail.com Thu Jun 16 12:20:27 2005 From: rabenstein.si at gmail.com (Itamar & Shira Rabenstein) Date: Thu, 16 Jun 2005 21:20:27 +0200 Subject: [openib-general] Re: [PATCH][kdapl]update dat_rmr_bind API & delete dapl_hash Message-ID: <7414e54805061612203fc942cb@mail.gmail.com> >Itamar, > >Can you seperate the RMR api change and hash table removal into a >seperate patch? I dont see why, the new input param to dat_rmr_bind (lmr_handle) come in order that we will be able to remove the hash table so this is one patch. >Merging the HCA and IA code into the dapl_ia.[ch] files doesn't feel >right to me. I think of the HCA as a DAT object just like IAs, EPs, >LMRs, etc, and hence it should have its own file. > >james There are 4 function in this files: 2 were merged to provider code: dapl_hca_alloc(13 lines) :which i think is part of the provider code . dapl_hca_free (1 line) == kfree , no need a function for this. 2 were merged to ia code: dapl_hca_link_ia (3 lines ) : no need a function to call list add dapl_hca_unlink_ia(3 lines) : no need a function to call list_del I belive that we need to make the code more readable and to have this functions is not a good idea and to add 2 files for this is for sure a bad idea ... So Please commit as is. Itamar >On Wed, 15 Jun 2005, Rabenstein Itamar wrote: > >> Updated dat_rmr_bind API (added lmr_handle as input param) >> delete dapl_hash.[h|c] files (not needed any more) >> Integrate dapl_hca_util functions in ia/provider code >> delete dapl_hca_util.[h|c] files >> >>Signed-off-by: Itamar Rabenstein >> From roland at topspin.com Thu Jun 16 12:59:32 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 12:59:32 -0700 Subject: [openib-general] [PATCH] Ref count user doorbell pages in mthca Message-ID: <524qbyax97.fsf@topspin.com> This patch keeps a reference count on the doorbell pages that are mapped into userspace for direct verbs access. This prevents a buggy/malicious app from closing its context but keeping a doorbell page mapped, and then improperly accessing the doorbell page after it gets allocated again to a new process. Anyone see any issues with this? - R. --- infiniband/hw/mthca/mthca_dev.h (revision 2631) +++ infiniband/hw/mthca/mthca_dev.h (working copy) @@ -301,7 +301,7 @@ struct mthca_dev { struct mthca_av_table av_table; struct mthca_mcg_table mcg_table; - struct mthca_uar driver_uar; + struct mthca_uar *driver_uar; struct mthca_db_table *db_tab; struct mthca_pd driver_pd; struct mthca_mr driver_mr; @@ -382,8 +382,8 @@ void mthca_cleanup_mcg_table(struct mthc int mthca_register_device(struct mthca_dev *dev); void mthca_unregister_device(struct mthca_dev *dev); -int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar); -void mthca_uar_free(struct mthca_dev *dev, struct mthca_uar *uar); +struct mthca_uar *mthca_uar_alloc(struct mthca_dev *dev); +void mthca_uar_put(struct mthca_uar *uar); int mthca_pd_alloc(struct mthca_dev *dev, int privileged, struct mthca_pd *pd); void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd); --- infiniband/hw/mthca/mthca_main.c (revision 2631) +++ infiniband/hw/mthca/mthca_main.c (working copy) @@ -636,14 +636,15 @@ static int __devinit mthca_setup_hca(str return err; } - err = mthca_uar_alloc(dev, &dev->driver_uar); - if (err) { + dev->driver_uar = mthca_uar_alloc(dev); + if (IS_ERR(dev->driver_uar)) { mthca_err(dev, "Failed to allocate driver access region, " "aborting.\n"); + err = PTR_ERR(dev->driver_uar); goto err_uar_table_free; } - dev->kar = ioremap(dev->driver_uar.pfn << PAGE_SHIFT, PAGE_SIZE); + dev->kar = ioremap(dev->driver_uar->pfn << PAGE_SHIFT, PAGE_SIZE); if (!dev->kar) { mthca_err(dev, "Couldn't map kernel access region, " "aborting.\n"); @@ -760,7 +761,7 @@ err_kar_unmap: iounmap(dev->kar); err_uar_free: - mthca_uar_free(dev, &dev->driver_uar); + mthca_uar_put(dev->driver_uar); err_uar_table_free: mthca_cleanup_uar_table(dev); @@ -1117,7 +1118,7 @@ static void __devexit mthca_remove_one(s mthca_cleanup_pd_table(mdev); iounmap(mdev->kar); - mthca_uar_free(mdev, &mdev->driver_uar); + mthca_uar_put(mdev->driver_uar); mthca_cleanup_uar_table(mdev); mthca_close_hca(mdev); --- infiniband/hw/mthca/mthca_uar.c (revision 2631) +++ infiniband/hw/mthca/mthca_uar.c (working copy) @@ -35,20 +35,35 @@ #include "mthca_dev.h" #include "mthca_memfree.h" -int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar) +struct mthca_uar *mthca_uar_alloc(struct mthca_dev *dev) { + struct mthca_uar *uar = kmalloc(sizeof *uar, GFP_KERNEL); + if (!uar) + return ERR_PTR(-ENOMEM); + + kref_init(&uar->ref); + uar->index = mthca_alloc(&dev->uar_table.alloc); if (uar->index == -1) - return -ENOMEM; + return ERR_PTR(-ENOMEM); uar->pfn = (pci_resource_start(dev->pdev, 2) >> PAGE_SHIFT) + uar->index; + uar->dev = dev; + + return uar; +} + +static void mthca_uar_free(struct kref *ref) +{ + struct mthca_uar *uar = container_of(ref, struct mthca_uar, ref); - return 0; + mthca_free(&uar->dev->uar_table.alloc, uar->index); + kfree(uar); } -void mthca_uar_free(struct mthca_dev *dev, struct mthca_uar *uar) +void mthca_uar_put(struct mthca_uar *uar) { - mthca_free(&dev->uar_table.alloc, uar->index); + kref_put(&uar->ref, mthca_uar_free); } int mthca_init_uar_table(struct mthca_dev *dev) --- infiniband/hw/mthca/mthca_provider.c (revision 2631) +++ infiniband/hw/mthca/mthca_provider.c (working copy) @@ -309,8 +309,9 @@ static struct ib_ucontext *mthca_alloc_u if (!context) return ERR_PTR(-ENOMEM); - err = mthca_uar_alloc(to_mdev(ibdev), &context->uar); - if (err) { + context->uar = mthca_uar_alloc(to_mdev(ibdev)); + if (IS_ERR(context->uar)) { + err = PTR_ERR(context->uar); kfree(context); return ERR_PTR(err); } @@ -318,15 +319,15 @@ static struct ib_ucontext *mthca_alloc_u context->db_tab = mthca_init_user_db_tab(to_mdev(ibdev)); if (IS_ERR(context->db_tab)) { err = PTR_ERR(context->db_tab); - mthca_uar_free(to_mdev(ibdev), &context->uar); + mthca_uar_put(context->uar); kfree(context); return ERR_PTR(err); } if (copy_to_user((void __user *) (unsigned long) ucmd.respbuf, &uresp, sizeof uresp)) { - mthca_cleanup_user_db_tab(to_mdev(ibdev), &context->uar, context->db_tab); - mthca_uar_free(to_mdev(ibdev), &context->uar); + mthca_cleanup_user_db_tab(to_mdev(ibdev), context->uar, context->db_tab); + mthca_uar_put(context->uar); kfree(context); return ERR_PTR(-EFAULT); } @@ -336,14 +337,30 @@ static struct ib_ucontext *mthca_alloc_u static int mthca_dealloc_ucontext(struct ib_ucontext *context) { - mthca_cleanup_user_db_tab(to_mdev(context->device), &to_mucontext(context)->uar, + mthca_cleanup_user_db_tab(to_mdev(context->device), to_mucontext(context)->uar, to_mucontext(context)->db_tab); - mthca_uar_free(to_mdev(context->device), &to_mucontext(context)->uar); + mthca_uar_put(to_mucontext(context)->uar); kfree(to_mucontext(context)); return 0; } +static void mthca_open_vma(struct vm_area_struct *vma) +{ + struct mthca_uar *uar = vma->vm_private_data; + kref_get(&uar->ref); +} + +static void mthca_close_vma(struct vm_area_struct *vma) +{ + mthca_uar_put(vma->vm_private_data); +} + +static struct vm_operations_struct mthca_vm_ops = { + .open = mthca_open_vma, + .close = mthca_close_vma +}; + static int mthca_mmap_uar(struct ib_ucontext *context, struct vm_area_struct *vma) { @@ -353,10 +370,15 @@ static int mthca_mmap_uar(struct ib_ucon vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); if (remap_pfn_range(vma, vma->vm_start, - to_mucontext(context)->uar.pfn, + to_mucontext(context)->uar->pfn, PAGE_SIZE, vma->vm_page_prot)) return -EAGAIN; + kref_get(&to_mucontext(context)->uar->ref); + + vma->vm_private_data = to_mucontext(context)->uar; + vma->vm_ops = &mthca_vm_ops; + return 0; } @@ -460,7 +482,7 @@ static struct ib_qp *mthca_create_qp(str if (copy_from_user(&ucmd, udata, sizeof ucmd)) return ERR_PTR(-EFAULT); - err = mthca_map_user_db(to_mdev(pd->device), &context->uar, + err = mthca_map_user_db(to_mdev(pd->device), context->uar, context->db_tab, ucmd.sq_db_index, ucmd.sq_db_page); if (err) { @@ -468,12 +490,12 @@ static struct ib_qp *mthca_create_qp(str return ERR_PTR(err); } - err = mthca_map_user_db(to_mdev(pd->device), &context->uar, + err = mthca_map_user_db(to_mdev(pd->device), context->uar, context->db_tab, ucmd.rq_db_index, ucmd.rq_db_page); if (err) { mthca_unmap_user_db(to_mdev(pd->device), - &context->uar, + context->uar, context->db_tab, ucmd.sq_db_index); kfree(qp); @@ -495,11 +517,11 @@ static struct ib_qp *mthca_create_qp(str context = to_mucontext(pd->uobject->context); mthca_unmap_user_db(to_mdev(pd->device), - &context->uar, + context->uar, context->db_tab, ucmd.sq_db_index); mthca_unmap_user_db(to_mdev(pd->device), - &context->uar, + context->uar, context->db_tab, ucmd.rq_db_index); } @@ -551,11 +573,11 @@ static int mthca_destroy_qp(struct ib_qp { if (qp->uobject) { mthca_unmap_user_db(to_mdev(qp->device), - &to_mucontext(qp->uobject->context)->uar, + to_mucontext(qp->uobject->context)->uar, to_mucontext(qp->uobject->context)->db_tab, to_mqp(qp)->sq.db_index); mthca_unmap_user_db(to_mdev(qp->device), - &to_mucontext(qp->uobject->context)->uar, + to_mucontext(qp->uobject->context)->uar, to_mucontext(qp->uobject->context)->db_tab, to_mqp(qp)->rq.db_index); } @@ -580,13 +602,13 @@ static struct ib_cq *mthca_create_cq(str if (copy_from_user(&ucmd, udata, sizeof ucmd)) return ERR_PTR(-EFAULT); - err = mthca_map_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, + err = mthca_map_user_db(to_mdev(ibdev), to_mucontext(context)->uar, to_mucontext(context)->db_tab, ucmd.set_db_index, ucmd.set_db_page); if (err) return ERR_PTR(err); - err = mthca_map_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, + err = mthca_map_user_db(to_mdev(ibdev), to_mucontext(context)->uar, to_mucontext(context)->db_tab, ucmd.arm_db_index, ucmd.arm_db_page); if (err) @@ -627,12 +649,12 @@ err_free: err_unmap_arm: if (context) - mthca_unmap_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, + mthca_unmap_user_db(to_mdev(ibdev), to_mucontext(context)->uar, to_mucontext(context)->db_tab, ucmd.arm_db_index); err_unmap_set: if (context) - mthca_unmap_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, + mthca_unmap_user_db(to_mdev(ibdev), to_mucontext(context)->uar, to_mucontext(context)->db_tab, ucmd.set_db_index); return ERR_PTR(err); @@ -642,11 +664,11 @@ static int mthca_destroy_cq(struct ib_cq { if (cq->uobject) { mthca_unmap_user_db(to_mdev(cq->device), - &to_mucontext(cq->uobject->context)->uar, + to_mucontext(cq->uobject->context)->uar, to_mucontext(cq->uobject->context)->db_tab, to_mcq(cq)->arm_db_index); mthca_unmap_user_db(to_mdev(cq->device), - &to_mucontext(cq->uobject->context)->uar, + to_mucontext(cq->uobject->context)->uar, to_mucontext(cq->uobject->context)->db_tab, to_mcq(cq)->set_ci_db_index); } --- infiniband/hw/mthca/mthca_provider.h (revision 2631) +++ infiniband/hw/mthca/mthca_provider.h (working copy) @@ -50,15 +50,17 @@ struct mthca_buf_list { }; struct mthca_uar { - unsigned long pfn; - int index; + struct mthca_dev *dev; + unsigned long pfn; + int index; + struct kref ref; }; struct mthca_user_db_table; struct mthca_ucontext { struct ib_ucontext ibucontext; - struct mthca_uar uar; + struct mthca_uar *uar; struct mthca_user_db_table *db_tab; }; --- infiniband/hw/mthca/mthca_cq.c (revision 2631) +++ infiniband/hw/mthca/mthca_cq.c (working copy) @@ -809,9 +809,9 @@ int mthca_init_cq(struct mthca_dev *dev, cq_context->start = cpu_to_be64(0); cq_context->logsize_usrpage = cpu_to_be32((ffs(nent) - 1) << 24); if (ctx) - cq_context->logsize_usrpage |= cpu_to_be32(ctx->uar.index); + cq_context->logsize_usrpage |= cpu_to_be32(ctx->uar->index); else - cq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar.index); + cq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar->index); cq_context->error_eqn = cpu_to_be32(dev->eq_table.eq[MTHCA_EQ_ASYNC].eqn); cq_context->comp_eqn = cpu_to_be32(dev->eq_table.eq[MTHCA_EQ_COMP].eqn); cq_context->pd = cpu_to_be32(pdn); --- infiniband/hw/mthca/mthca_eq.c (revision 2631) +++ infiniband/hw/mthca/mthca_eq.c (working copy) @@ -542,7 +542,7 @@ static int __devinit mthca_create_eq(str if (mthca_is_memfree(dev)) { eq_context->arbel_pd = cpu_to_be32(dev->driver_pd.pd_num); } else { - eq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar.index); + eq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar->index); eq_context->tavor_pd = cpu_to_be32(dev->driver_pd.pd_num); } eq_context->intr = intr; --- infiniband/hw/mthca/mthca_qp.c (revision 2631) +++ infiniband/hw/mthca/mthca_qp.c (working copy) @@ -693,9 +693,9 @@ int mthca_modify_qp(struct ib_qp *ibqp, if (qp->ibqp.uobject) qp_context->usr_page = - cpu_to_be32(to_mucontext(qp->ibqp.uobject->context)->uar.index); + cpu_to_be32(to_mucontext(qp->ibqp.uobject->context)->uar->index); else - qp_context->usr_page = cpu_to_be32(dev->driver_uar.index); + qp_context->usr_page = cpu_to_be32(dev->driver_uar->index); qp_context->local_qpn = cpu_to_be32(qp->qpn); if (attr_mask & IB_QP_DEST_QPN) { qp_context->remote_qpn = cpu_to_be32(attr->dest_qp_num); --- infiniband/hw/mthca/mthca_memfree.c (revision 2631) +++ infiniband/hw/mthca/mthca_memfree.c (working copy) @@ -527,7 +527,7 @@ int mthca_alloc_db(struct mthca_dev *dev memset(page->db_rec, 0, 4096); ret = mthca_MAP_ICM_page(dev, page->mapping, - mthca_uarc_virt(dev, &dev->driver_uar, i), &status); + mthca_uarc_virt(dev, dev->driver_uar, i), &status); if (!ret && status) ret = -EINVAL; if (ret) { @@ -581,7 +581,7 @@ void mthca_free_db(struct mthca_dev *dev if (bitmap_empty(page->used, MTHCA_DB_REC_PER_PAGE) && i >= dev->db_tab->max_group1 - 1) { - mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, &dev->driver_uar, i), 1, &status); + mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, dev->driver_uar, i), 1, &status); dma_free_coherent(&dev->pdev->dev, 4096, page->db_rec, page->mapping); @@ -650,7 +650,7 @@ void mthca_cleanup_db_tab(struct mthca_d if (!bitmap_empty(dev->db_tab->page[i].used, MTHCA_DB_REC_PER_PAGE)) mthca_warn(dev, "Kernel UARC page %d not empty\n", i); - mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, &dev->driver_uar, i), 1, &status); + mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, dev->driver_uar, i), 1, &status); dma_free_coherent(&dev->pdev->dev, 4096, dev->db_tab->page[i].db_rec, From mst at mellanox.co.il Thu Jun 16 13:43:52 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 16 Jun 2005 23:43:52 +0300 Subject: [openib-general] Re: [PATCH] Ref count user doorbell pages in mthca In-Reply-To: <524qbyax97.fsf@topspin.com> References: <524qbyax97.fsf@topspin.com> Message-ID: <20050616204352.GA32394@mellanox.co.il> Quoting r. Roland Dreier : > Subject: [PATCH] Ref count user doorbell pages in mthca > > This patch keeps a reference count on the doorbell pages that are > mapped into userspace for direct verbs access. This prevents a > buggy/malicious app from closing its context but keeping a doorbell > page mapped, and then improperly accessing the doorbell page after it > gets allocated again to a new process. How does one close the context? I thought thats done by closing the file descriptor, isnt that right? And if not - why not? I think closing the file descriptor will kill the vmas without work in mthca, avoiding this issue altogether. > Anyone see any issues with this? > > - R. For hotswap we'll need a way to find and unmap all uars, anyway, and I think the same shall be done when context is closed by the application. IMO, when the app closes the context, all resources shall go. > +void mthca_uar_put(struct mthca_uar *uar) > { > - mthca_free(&dev->uar_table.alloc, uar->index); > + kref_put(&uar->ref, mthca_uar_free); > } What lock prevents the reference count from going to 0, and driver deciding to kill the uar object, when at the same time vma (by vm_ops) has a pointer to this object and is calling kref_get? MST > --- infiniband/hw/mthca/mthca_dev.h (revision 2631) > +++ infiniband/hw/mthca/mthca_dev.h (working copy) > @@ -301,7 +301,7 @@ struct mthca_dev { > struct mthca_av_table av_table; > struct mthca_mcg_table mcg_table; > > - struct mthca_uar driver_uar; > + struct mthca_uar *driver_uar; > struct mthca_db_table *db_tab; > struct mthca_pd driver_pd; > struct mthca_mr driver_mr; > @@ -382,8 +382,8 @@ void mthca_cleanup_mcg_table(struct mthc > int mthca_register_device(struct mthca_dev *dev); > void mthca_unregister_device(struct mthca_dev *dev); > > -int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar); > -void mthca_uar_free(struct mthca_dev *dev, struct mthca_uar *uar); > +struct mthca_uar *mthca_uar_alloc(struct mthca_dev *dev); > +void mthca_uar_put(struct mthca_uar *uar); > > int mthca_pd_alloc(struct mthca_dev *dev, int privileged, struct mthca_pd *pd); > void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd); > --- infiniband/hw/mthca/mthca_main.c (revision 2631) > +++ infiniband/hw/mthca/mthca_main.c (working copy) > @@ -636,14 +636,15 @@ static int __devinit mthca_setup_hca(str > return err; > } > > - err = mthca_uar_alloc(dev, &dev->driver_uar); > - if (err) { > + dev->driver_uar = mthca_uar_alloc(dev); > + if (IS_ERR(dev->driver_uar)) { > mthca_err(dev, "Failed to allocate driver access region, " > "aborting.\n"); > + err = PTR_ERR(dev->driver_uar); > goto err_uar_table_free; > } > > - dev->kar = ioremap(dev->driver_uar.pfn << PAGE_SHIFT, PAGE_SIZE); > + dev->kar = ioremap(dev->driver_uar->pfn << PAGE_SHIFT, PAGE_SIZE); > if (!dev->kar) { > mthca_err(dev, "Couldn't map kernel access region, " > "aborting.\n"); > @@ -760,7 +761,7 @@ err_kar_unmap: > iounmap(dev->kar); > > err_uar_free: > - mthca_uar_free(dev, &dev->driver_uar); > + mthca_uar_put(dev->driver_uar); > > err_uar_table_free: > mthca_cleanup_uar_table(dev); > @@ -1117,7 +1118,7 @@ static void __devexit mthca_remove_one(s > mthca_cleanup_pd_table(mdev); > > iounmap(mdev->kar); > - mthca_uar_free(mdev, &mdev->driver_uar); > + mthca_uar_put(mdev->driver_uar); > mthca_cleanup_uar_table(mdev); > > mthca_close_hca(mdev); > --- infiniband/hw/mthca/mthca_uar.c (revision 2631) > +++ infiniband/hw/mthca/mthca_uar.c (working copy) > @@ -35,20 +35,35 @@ > #include "mthca_dev.h" > #include "mthca_memfree.h" > > -int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar) > +struct mthca_uar *mthca_uar_alloc(struct mthca_dev *dev) > { > + struct mthca_uar *uar = kmalloc(sizeof *uar, GFP_KERNEL); > + if (!uar) > + return ERR_PTR(-ENOMEM); > + > + kref_init(&uar->ref); > + > uar->index = mthca_alloc(&dev->uar_table.alloc); > if (uar->index == -1) > - return -ENOMEM; > + return ERR_PTR(-ENOMEM); > > uar->pfn = (pci_resource_start(dev->pdev, 2) >> PAGE_SHIFT) + uar->index; > + uar->dev = dev; > + > + return uar; > +} > + > +static void mthca_uar_free(struct kref *ref) > +{ > + struct mthca_uar *uar = container_of(ref, struct mthca_uar, ref); > > - return 0; > + mthca_free(&uar->dev->uar_table.alloc, uar->index); > + kfree(uar); > } > > -void mthca_uar_free(struct mthca_dev *dev, struct mthca_uar *uar) > +void mthca_uar_put(struct mthca_uar *uar) > { > - mthca_free(&dev->uar_table.alloc, uar->index); > + kref_put(&uar->ref, mthca_uar_free); > } > > int mthca_init_uar_table(struct mthca_dev *dev) > --- infiniband/hw/mthca/mthca_provider.c (revision 2631) > +++ infiniband/hw/mthca/mthca_provider.c (working copy) > @@ -309,8 +309,9 @@ static struct ib_ucontext *mthca_alloc_u > if (!context) > return ERR_PTR(-ENOMEM); > > - err = mthca_uar_alloc(to_mdev(ibdev), &context->uar); > - if (err) { > + context->uar = mthca_uar_alloc(to_mdev(ibdev)); > + if (IS_ERR(context->uar)) { > + err = PTR_ERR(context->uar); > kfree(context); > return ERR_PTR(err); > } > @@ -318,15 +319,15 @@ static struct ib_ucontext *mthca_alloc_u > context->db_tab = mthca_init_user_db_tab(to_mdev(ibdev)); > if (IS_ERR(context->db_tab)) { > err = PTR_ERR(context->db_tab); > - mthca_uar_free(to_mdev(ibdev), &context->uar); > + mthca_uar_put(context->uar); > kfree(context); > return ERR_PTR(err); > } > > if (copy_to_user((void __user *) (unsigned long) ucmd.respbuf, > &uresp, sizeof uresp)) { > - mthca_cleanup_user_db_tab(to_mdev(ibdev), &context->uar, context->db_tab); > - mthca_uar_free(to_mdev(ibdev), &context->uar); > + mthca_cleanup_user_db_tab(to_mdev(ibdev), context->uar, context->db_tab); > + mthca_uar_put(context->uar); > kfree(context); > return ERR_PTR(-EFAULT); > } > @@ -336,14 +337,30 @@ static struct ib_ucontext *mthca_alloc_u > > static int mthca_dealloc_ucontext(struct ib_ucontext *context) > { > - mthca_cleanup_user_db_tab(to_mdev(context->device), &to_mucontext(context)->uar, > + mthca_cleanup_user_db_tab(to_mdev(context->device), to_mucontext(context)->uar, > to_mucontext(context)->db_tab); > - mthca_uar_free(to_mdev(context->device), &to_mucontext(context)->uar); > + mthca_uar_put(to_mucontext(context)->uar); > kfree(to_mucontext(context)); > > return 0; > } > > +static void mthca_open_vma(struct vm_area_struct *vma) > +{ > + struct mthca_uar *uar = vma->vm_private_data; > + kref_get(&uar->ref); > +} > + > +static void mthca_close_vma(struct vm_area_struct *vma) > +{ > + mthca_uar_put(vma->vm_private_data); > +} > + > +static struct vm_operations_struct mthca_vm_ops = { > + .open = mthca_open_vma, > + .close = mthca_close_vma > +}; > + > static int mthca_mmap_uar(struct ib_ucontext *context, > struct vm_area_struct *vma) > { > @@ -353,10 +370,15 @@ static int mthca_mmap_uar(struct ib_ucon > vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > > if (remap_pfn_range(vma, vma->vm_start, > - to_mucontext(context)->uar.pfn, > + to_mucontext(context)->uar->pfn, > PAGE_SIZE, vma->vm_page_prot)) > return -EAGAIN; > > + kref_get(&to_mucontext(context)->uar->ref); > + > + vma->vm_private_data = to_mucontext(context)->uar; > + vma->vm_ops = &mthca_vm_ops; > + > return 0; > } > > @@ -460,7 +482,7 @@ static struct ib_qp *mthca_create_qp(str > if (copy_from_user(&ucmd, udata, sizeof ucmd)) > return ERR_PTR(-EFAULT); > > - err = mthca_map_user_db(to_mdev(pd->device), &context->uar, > + err = mthca_map_user_db(to_mdev(pd->device), context->uar, > context->db_tab, > ucmd.sq_db_index, ucmd.sq_db_page); > if (err) { > @@ -468,12 +490,12 @@ static struct ib_qp *mthca_create_qp(str > return ERR_PTR(err); > } > > - err = mthca_map_user_db(to_mdev(pd->device), &context->uar, > + err = mthca_map_user_db(to_mdev(pd->device), context->uar, > context->db_tab, > ucmd.rq_db_index, ucmd.rq_db_page); > if (err) { > mthca_unmap_user_db(to_mdev(pd->device), > - &context->uar, > + context->uar, > context->db_tab, > ucmd.sq_db_index); > kfree(qp); > @@ -495,11 +517,11 @@ static struct ib_qp *mthca_create_qp(str > context = to_mucontext(pd->uobject->context); > > mthca_unmap_user_db(to_mdev(pd->device), > - &context->uar, > + context->uar, > context->db_tab, > ucmd.sq_db_index); > mthca_unmap_user_db(to_mdev(pd->device), > - &context->uar, > + context->uar, > context->db_tab, > ucmd.rq_db_index); > } > @@ -551,11 +573,11 @@ static int mthca_destroy_qp(struct ib_qp > { > if (qp->uobject) { > mthca_unmap_user_db(to_mdev(qp->device), > - &to_mucontext(qp->uobject->context)->uar, > + to_mucontext(qp->uobject->context)->uar, > to_mucontext(qp->uobject->context)->db_tab, > to_mqp(qp)->sq.db_index); > mthca_unmap_user_db(to_mdev(qp->device), > - &to_mucontext(qp->uobject->context)->uar, > + to_mucontext(qp->uobject->context)->uar, > to_mucontext(qp->uobject->context)->db_tab, > to_mqp(qp)->rq.db_index); > } > @@ -580,13 +602,13 @@ static struct ib_cq *mthca_create_cq(str > if (copy_from_user(&ucmd, udata, sizeof ucmd)) > return ERR_PTR(-EFAULT); > > - err = mthca_map_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, > + err = mthca_map_user_db(to_mdev(ibdev), to_mucontext(context)->uar, > to_mucontext(context)->db_tab, > ucmd.set_db_index, ucmd.set_db_page); > if (err) > return ERR_PTR(err); > > - err = mthca_map_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, > + err = mthca_map_user_db(to_mdev(ibdev), to_mucontext(context)->uar, > to_mucontext(context)->db_tab, > ucmd.arm_db_index, ucmd.arm_db_page); > if (err) > @@ -627,12 +649,12 @@ err_free: > > err_unmap_arm: > if (context) > - mthca_unmap_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, > + mthca_unmap_user_db(to_mdev(ibdev), to_mucontext(context)->uar, > to_mucontext(context)->db_tab, ucmd.arm_db_index); > > err_unmap_set: > if (context) > - mthca_unmap_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, > + mthca_unmap_user_db(to_mdev(ibdev), to_mucontext(context)->uar, > to_mucontext(context)->db_tab, ucmd.set_db_index); > > return ERR_PTR(err); > @@ -642,11 +664,11 @@ static int mthca_destroy_cq(struct ib_cq > { > if (cq->uobject) { > mthca_unmap_user_db(to_mdev(cq->device), > - &to_mucontext(cq->uobject->context)->uar, > + to_mucontext(cq->uobject->context)->uar, > to_mucontext(cq->uobject->context)->db_tab, > to_mcq(cq)->arm_db_index); > mthca_unmap_user_db(to_mdev(cq->device), > - &to_mucontext(cq->uobject->context)->uar, > + to_mucontext(cq->uobject->context)->uar, > to_mucontext(cq->uobject->context)->db_tab, > to_mcq(cq)->set_ci_db_index); > } > --- infiniband/hw/mthca/mthca_provider.h (revision 2631) > +++ infiniband/hw/mthca/mthca_provider.h (working copy) > @@ -50,15 +50,17 @@ struct mthca_buf_list { > }; > > struct mthca_uar { > - unsigned long pfn; > - int index; > + struct mthca_dev *dev; > + unsigned long pfn; > + int index; > + struct kref ref; > }; > > struct mthca_user_db_table; > > struct mthca_ucontext { > struct ib_ucontext ibucontext; > - struct mthca_uar uar; > + struct mthca_uar *uar; > struct mthca_user_db_table *db_tab; > }; > > --- infiniband/hw/mthca/mthca_cq.c (revision 2631) > +++ infiniband/hw/mthca/mthca_cq.c (working copy) > @@ -809,9 +809,9 @@ int mthca_init_cq(struct mthca_dev *dev, > cq_context->start = cpu_to_be64(0); > cq_context->logsize_usrpage = cpu_to_be32((ffs(nent) - 1) << 24); > if (ctx) > - cq_context->logsize_usrpage |= cpu_to_be32(ctx->uar.index); > + cq_context->logsize_usrpage |= cpu_to_be32(ctx->uar->index); > else > - cq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar.index); > + cq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar->index); > cq_context->error_eqn = cpu_to_be32(dev->eq_table.eq[MTHCA_EQ_ASYNC].eqn); > cq_context->comp_eqn = cpu_to_be32(dev->eq_table.eq[MTHCA_EQ_COMP].eqn); > cq_context->pd = cpu_to_be32(pdn); > --- infiniband/hw/mthca/mthca_eq.c (revision 2631) > +++ infiniband/hw/mthca/mthca_eq.c (working copy) > @@ -542,7 +542,7 @@ static int __devinit mthca_create_eq(str > if (mthca_is_memfree(dev)) { > eq_context->arbel_pd = cpu_to_be32(dev->driver_pd.pd_num); > } else { > - eq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar.index); > + eq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar->index); > eq_context->tavor_pd = cpu_to_be32(dev->driver_pd.pd_num); > } > eq_context->intr = intr; > --- infiniband/hw/mthca/mthca_qp.c (revision 2631) > +++ infiniband/hw/mthca/mthca_qp.c (working copy) > @@ -693,9 +693,9 @@ int mthca_modify_qp(struct ib_qp *ibqp, > > if (qp->ibqp.uobject) > qp_context->usr_page = > - cpu_to_be32(to_mucontext(qp->ibqp.uobject->context)->uar.index); > + cpu_to_be32(to_mucontext(qp->ibqp.uobject->context)->uar->index); > else > - qp_context->usr_page = cpu_to_be32(dev->driver_uar.index); > + qp_context->usr_page = cpu_to_be32(dev->driver_uar->index); > qp_context->local_qpn = cpu_to_be32(qp->qpn); > if (attr_mask & IB_QP_DEST_QPN) { > qp_context->remote_qpn = cpu_to_be32(attr->dest_qp_num); > --- infiniband/hw/mthca/mthca_memfree.c (revision 2631) > +++ infiniband/hw/mthca/mthca_memfree.c (working copy) > @@ -527,7 +527,7 @@ int mthca_alloc_db(struct mthca_dev *dev > memset(page->db_rec, 0, 4096); > > ret = mthca_MAP_ICM_page(dev, page->mapping, > - mthca_uarc_virt(dev, &dev->driver_uar, i), &status); > + mthca_uarc_virt(dev, dev->driver_uar, i), &status); > if (!ret && status) > ret = -EINVAL; > if (ret) { > @@ -581,7 +581,7 @@ void mthca_free_db(struct mthca_dev *dev > > if (bitmap_empty(page->used, MTHCA_DB_REC_PER_PAGE) && > i >= dev->db_tab->max_group1 - 1) { > - mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, &dev->driver_uar, i), 1, &status); > + mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, dev->driver_uar, i), 1, &status); > > dma_free_coherent(&dev->pdev->dev, 4096, > page->db_rec, page->mapping); > @@ -650,7 +650,7 @@ void mthca_cleanup_db_tab(struct mthca_d > if (!bitmap_empty(dev->db_tab->page[i].used, MTHCA_DB_REC_PER_PAGE)) > mthca_warn(dev, "Kernel UARC page %d not empty\n", i); > > - mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, &dev->driver_uar, i), 1, &status); > + mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, dev->driver_uar, i), 1, &status); > > dma_free_coherent(&dev->pdev->dev, 4096, > dev->db_tab->page[i].db_rec, -- MST From bjordan at silverstorm.com Thu Jun 16 13:49:59 2005 From: bjordan at silverstorm.com (Jordan, Bill) Date: Thu, 16 Jun 2005 16:49:59 -0400 Subject: [openib-general] [PATCH] cm: fix for service_id/service_mask hole Message-ID: <5D78D28F88822E4D8702BB9EEF1A4367140273@mercury.infiniconsys.com> Sean, Here is a to fix a bug in service_id/service_mask handling in CM. Previously, if you tried to listen on service_id: service_mask of 0x00:0x0F, followed by a listen on 0xF0:0xF0, the second listen would fail because the two ranges overlap (0xF0 appears in both ranges). But, if you tried the listens in the reverse order, both would succeed leading to an ambigious listen. This patch forces both service_ids in the compare to be masked against both mask values first. Signed-off-by: Bill Jordan Index: src/linux-kernel/infiniband/core/cm.c =================================================================== --- src/linux-kernel/infiniband/core/cm.c (revision 2636) +++ src/linux-kernel/infiniband/core/cm.c (working copy) @@ -354,14 +354,14 @@ static struct cm_id_private * cm_insert_ struct rb_node *parent = NULL; struct cm_id_private *cur_cm_id_priv; u64 service_id = cm_id_priv->id.service_id; + u64 service_mask = cm_id_priv->id.service_mask; while (*link) { parent = *link; cur_cm_id_priv = rb_entry(parent, struct cm_id_private, service_node); if ((cur_cm_id_priv->id.service_mask & service_id) == - (cur_cm_id_priv->id.service_mask & - cur_cm_id_priv->id.service_id)) + (service_mask & cur_cm_id_priv->id.service_id)) return cm_id_priv; if (service_id < cur_cm_id_priv->id.service_id) link = &(*link)->rb_left; @@ -719,6 +719,8 @@ int ib_cm_listen(struct ib_cm_id *cm_id, unsigned long flags; int ret = 0; + service_mask = service_mask ? service_mask : ~0ULL; + service_id &= service_mask; if ((service_id & IB_SERVICE_ID_AGN_MASK) == IB_CM_ASSIGN_SERVICE_ID && (service_id != IB_CM_ASSIGN_SERVICE_ID)) return -EINVAL; @@ -734,7 +736,7 @@ int ib_cm_listen(struct ib_cm_id *cm_id, cm_id->service_mask = ~0ULL; } else { cm_id->service_id = service_id; - cm_id->service_mask = service_mask ? service_mask : ~0ULL; + cm_id->service_mask = service_mask; } cur_cm_id_priv = cm_insert_listen(cm_id_priv); spin_unlock_irqrestore(&cm.lock, flags); From roland at topspin.com Thu Jun 16 13:54:54 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 13:54:54 -0700 Subject: [openib-general] Re: [PATCH] Ref count user doorbell pages in mthca In-Reply-To: <20050616204352.GA32394@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 16 Jun 2005 23:43:52 +0300") References: <524qbyax97.fsf@topspin.com> <20050616204352.GA32394@mellanox.co.il> Message-ID: <52r7f29g4h.fsf@topspin.com> Michael> How does one close the context? I thought thats done by Michael> closing the file descriptor, isnt that right? And if not Michael> - why not? I think closing the file descriptor will kill Michael> the vmas without work in mthca, avoiding this issue Michael> altogether. Closing a file descriptor does not unmap any mmaps done on that FD. Michael> For hotswap we'll need a way to find and unmap all uars, Michael> anyway, and I think the same shall be done when context Michael> is closed by the application. IMO, when the app closes Michael> the context, all resources shall go. That's not the usual semantics of mmap() vs. closing a file. Michael> What lock prevents the reference count from going to 0, Michael> and driver deciding to kill the uar object, when at the Michael> same time vma (by vm_ops) has a pointer to this object Michael> and is calling kref_get? If the reference count is going to 0, then either: - there are no vmas left and the file is already closed so we don't have to wory about racing with mmap() or - the context's file descriptor is already be closed, so mmap() can't happen. We're going through vm_ops->close(), which is protected by the usual mm locking (page_table_lock?). - R. From tduffy at sun.com Thu Jun 16 14:04:11 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 16 Jun 2005 14:04:11 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <1118775221.24424.21.camel@localhost> References: <1118774337.22484.16.camel@duffman> <1118775221.24424.21.camel@localhost> Message-ID: <1118955851.21846.21.camel@duffman> On Tue, 2005-06-14 at 11:53 -0700, Matt Leininger wrote: > On Tue, 2005-06-14 at 11:38 -0700, Tom Duffy wrote: > > Can the folks at Sandia please update the http://www.openib.org/doc.html > > page to point to the wiki site? > > > > Also, we should move the FAQ's to WIKI. I can do this if you want. > > > I'll add a link to doc.html pointing to the wiki. Once the FAQ's are > moved to the wiki I can just have doc.html point directly to the wiki. I have moved the main FAQ to WIKI and updated it. https://openib.org/tiki/tiki-index.php?page=OpenIBFAQ You can go ahead and remove all the rest of the *links* on the main doc.html page. The rest is are pointed to within the Wiki. Also, the mime type on the architecture pdf seems to be set wrong as my browser displays it as plain text (I see the pdf sources). -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mst at mellanox.co.il Thu Jun 16 14:21:19 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 17 Jun 2005 00:21:19 +0300 Subject: [openib-general] Re: [PATCH] Ref count user doorbell pages in mthca In-Reply-To: <52r7f29g4h.fsf@topspin.com> References: <524qbyax97.fsf@topspin.com> <20050616204352.GA32394@mellanox.co.il> <52r7f29g4h.fsf@topspin.com> Message-ID: <20050616212119.GB32394@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] Ref count user doorbell pages in mthca > > Roland> This patch keeps a reference count on the doorbell pages that are > Roland> mapped into userspace for direct verbs access. This prevents a > Roland> buggy/malicious app from closing its context but keeping a > Roland> doorbell page mapped, and then improperly accessing the doorbell > Roland> page after it gets allocated again to a new process. Is this something you've seen in practice? Looking at code under mm/, it seems that when you mmap something, get_file is called on the descriptor (and fput on close). So it seems to me the driver's release wont get called until the app actually unmaps the memory. If thats right, there's no issue since the context wont get closed. It seems to me if there are active vmas after the file is closed, we have other issues like the driver may get unloaded while vmas have pointer to mthca_open_vma. -- MST From roland at topspin.com Thu Jun 16 14:28:46 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 14:28:46 -0700 Subject: [openib-general] Re: [PATCH] Ref count user doorbell pages in mthca In-Reply-To: <20050616212119.GB32394@mellanox.co.il> (Michael S. Tsirkin's message of "Fri, 17 Jun 2005 00:21:19 +0300") References: <524qbyax97.fsf@topspin.com> <20050616204352.GA32394@mellanox.co.il> <52r7f29g4h.fsf@topspin.com> <20050616212119.GB32394@mellanox.co.il> Message-ID: <52aclq9ek1.fsf@topspin.com> Michael> Looking at code under mm/, it seems that when you mmap Michael> something, get_file is called on the descriptor (and fput Michael> on close). So it seems to me the driver's release wont Michael> get called until the app actually unmaps the memory. If Michael> thats right, there's no issue since the context wont get Michael> closed. You may be right. I need to investigate what happens if a process with a UAR mapped forks though. - R. From ftillier at silverstorm.com Thu Jun 16 14:39:33 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 16 Jun 2005 14:39:33 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <1118955851.21846.21.camel@duffman> Message-ID: <000401c572bb$e30f78e0$9c5aa8c0@infiniconsys.com> > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Thursday, June 16, 2005 2:04 PM > > On Tue, 2005-06-14 at 11:53 -0700, Matt Leininger wrote: > > On Tue, 2005-06-14 at 11:38 -0700, Tom Duffy wrote: > > > Can the folks at Sandia please update the http://www.openib.org/doc.html > > > page to point to the wiki site? > > > > > > Also, we should move the FAQ's to WIKI. I can do this if you want. > > > > > I'll add a link to doc.html pointing to the wiki. Once the FAQ's are > > moved to the wiki I can just have doc.html point directly to the wiki. > > I have moved the main FAQ to WIKI and updated it. > > https://openib.org/tiki/tiki-index.php?page=OpenIBFAQ This looks good. I'd like to setup a page for the windows project, rather than duplicating all of this infrastructure. How would I go about this? Thanks, - Fab From mst at mellanox.co.il Thu Jun 16 14:50:35 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 17 Jun 2005 00:50:35 +0300 Subject: [openib-general] Re: [PATCH] Ref count user doorbell pages in mthca In-Reply-To: <52aclq9ek1.fsf@topspin.com> References: <524qbyax97.fsf@topspin.com> <20050616204352.GA32394@mellanox.co.il> <52r7f29g4h.fsf@topspin.com> <20050616212119.GB32394@mellanox.co.il> <52aclq9ek1.fsf@topspin.com> Message-ID: <20050616215035.GC32394@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] Ref count user doorbell pages in mthca > > Michael> Looking at code under mm/, it seems that when you mmap > Michael> something, get_file is called on the descriptor (and fput > Michael> on close). So it seems to me the driver's release wont > Michael> get called until the app actually unmaps the memory. If > Michael> thats right, there's no issue since the context wont get > Michael> closed. > > You may be right. I need to investigate what happens if a process > with a UAR mapped forks though. > > - R. > Why dont we just set VM_DONTCOPY? -- MST From roland at topspin.com Thu Jun 16 14:54:47 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 14:54:47 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <000401c572bb$e30f78e0$9c5aa8c0@infiniconsys.com> (Fab Tillier's message of "Thu, 16 Jun 2005 14:39:33 -0700") References: <000401c572bb$e30f78e0$9c5aa8c0@infiniconsys.com> Message-ID: <524qby9dco.fsf@topspin.com> Fab> This looks good. I'd like to setup a page for the windows Fab> project, rather than duplicating all of this infrastructure. Fab> How would I go about this? Just create an account on the wiki and then create whatever pages you want (and edit existing pages to add links as necessary). I thought we wanted to keep the Windows project separate from the Linux project and not share this short of thing though... - R. From roland at topspin.com Thu Jun 16 14:55:16 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 14:55:16 -0700 Subject: [openib-general] Re: [PATCH] Ref count user doorbell pages in mthca In-Reply-To: <20050616215035.GC32394@mellanox.co.il> (Michael S. Tsirkin's message of "Fri, 17 Jun 2005 00:50:35 +0300") References: <524qbyax97.fsf@topspin.com> <20050616204352.GA32394@mellanox.co.il> <52r7f29g4h.fsf@topspin.com> <20050616212119.GB32394@mellanox.co.il> <52aclq9ek1.fsf@topspin.com> <20050616215035.GC32394@mellanox.co.il> Message-ID: <52zmtq7yrf.fsf@topspin.com> Michael> Why dont we just set VM_DONTCOPY? Hmm, that's a good point. I think that solves the problem in a much simpler way. - R. From ftillier at silverstorm.com Thu Jun 16 15:14:54 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 16 Jun 2005 15:14:54 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <524qby9dco.fsf@topspin.com> Message-ID: <000501c572c0$d372a380$9c5aa8c0@infiniconsys.com> > From: Roland Dreier [mailto:roland at topspin.com] > Sent: Thursday, June 16, 2005 2:55 PM > > Fab> This looks good. I'd like to setup a page for the windows > Fab> project, rather than duplicating all of this infrastructure. > Fab> How would I go about this? > > Just create an account on the wiki and then create whatever pages you > want (and edit existing pages to add links as necessary). I can see how to edit pages, but don't see any links or buttons to create new pages... I'm likely just missing it. I'll look some more, but if you have a pointer, that would be great. > I thought we wanted to keep the Windows project separate from the > Linux project and not share this short of thing though... The source repositories have to stay separate due to licensing and hosting issues. I don't see why we have to keep the web content separate. In fact, since the Wiki is at the top level OpenIB domain, I think it's appropriate to have Windows related information there. It seems silly to duplicate the effort you put in to create the Wiki - we surely all have better things to do. If we don't want to share the Wiki, then the existing Wiki (and all Linux related content) needs to move to a new subdomain like linux.openib.org, with a top level page that directs people to the appropriate operating system. As it stands, having everything Linux related on the openib home page while limiting Windows related content to a single link to a different server is not a proper representation of what the OpenIB Alliance is doing. - Fab From roland at topspin.com Thu Jun 16 15:21:33 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 15:21:33 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <000501c572c0$d372a380$9c5aa8c0@infiniconsys.com> (Fab Tillier's message of "Thu, 16 Jun 2005 15:14:54 -0700") References: <000501c572c0$d372a380$9c5aa8c0@infiniconsys.com> Message-ID: <52mzpq7xjm.fsf@topspin.com> Fab> I can see how to edit pages, but don't see any links or Fab> buttons to create new pages... I'm likely just missing it. Fab> I'll look some more, but if you have a pointer, that would be Fab> great. Actually I guess you have to create a link on an existing page first, either implicitly by adding something in CamelCase, or explicitly by tagging it as a link. Then you can click on the link and start editing your new page. Roland> I thought we wanted to keep the Windows project separate Roland> from the Linux project and not share this short of thing Roland> though... Fab> The source repositories have to stay separate due to Fab> licensing and hosting issues. I don't see why we have to Fab> keep the web content separate. In fact, since the Wiki is at Fab> the top level OpenIB domain, I think it's appropriate to have Fab> Windows related information there. It seems silly to Fab> duplicate the effort you put in to create the Wiki - we Fab> surely all have better things to do. Well, the supposed "licensing" issues never made any sense to me -- what is the the legal difference between separate directories on one server and separate servers in the same domain? But if it's OK to share one wiki according to whatever the rules are, then I don't mind. Fab> If we don't want to share the Wiki, then the existing Wiki Fab> (and all Linux related content) needs to move to a new Fab> subdomain like linux.openib.org, with a top level page that Fab> directs people to the appropriate operating system. That actually sounds like a good idea to me. - R. From iod00d at hp.com Thu Jun 16 15:27:12 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 16 Jun 2005 15:27:12 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <000501c572c0$d372a380$9c5aa8c0@infiniconsys.com> References: <524qby9dco.fsf@topspin.com> <000501c572c0$d372a380$9c5aa8c0@infiniconsys.com> Message-ID: <20050616222712.GM9121@esmail.cup.hp.com> On Thu, Jun 16, 2005 at 03:14:54PM -0700, Fab Tillier wrote: > > I thought we wanted to keep the Windows project separate from the > > Linux project and not share this short of thing though... ... > I don't see why we have to keep the web content separate. In fact, > since the Wiki is at the top level OpenIB domain, I think it's appropriate to > have Windows related information there. I agree. The top level page should host both. I don't agree that all the existing linux content should be mangled to include windows content on the same pages. Windows folks should clone any pages needing more than a few lines changed to accommodate them. grant From hozer at hozed.org Thu Jun 16 15:26:57 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Thu, 16 Jun 2005 17:26:57 -0500 Subject: [openib-general] Re: SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050616190355.GB32088@mellanox.co.il> References: <20050615213047.GM577@kalmia.hozed.org> <20050616111255.A8268@topspin.com> <20050616190355.GB32088@mellanox.co.il> Message-ID: <20050616222657.GN577@kalmia.hozed.org> On Thu, Jun 16, 2005 at 10:03:55PM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > Subject: Re: SDP: device mthca0 does not support fast memory regions > > > > On Wed, Jun 15, 2005 at 04:30:47PM -0500, Troy Benjegerdes wrote: > > > I'm getting a 'SDP: device mthca0 does nto support fast memory regions' > > > error on PPC64 systems.. Is there something that needs to be done for > > > PPC, or could I have an older version of the mthca module hanging > > > around? > > > > I'm not sure about the state or plans for FMR support on PPC64, Roland > > will have a better idea. > > Should just work AFAIK. > But maybe your card hides the onboard memory? > Try lspci -vv to see if the card has > the huge bar (100Mbyte or more) corresponding to that memory. Yes, the DDR is hidden on a couple of the machines, since they are IBM Power4 systems, which have some stupid issue with the firmware. However, I'm getting the following problem on an Xserve G5: xl0:~# cat /proc/version Linux version 2.6.11.11-G5 (root at xl0.fast) (gcc version 3.4.2) #1 SMP Wed Jun 812:11:29 CDT 2005 ib_mthca: Mellanox InfiniBand HCA driver v0.06-pre (November 8, 2004) ib_mthca: Initializing Mellanox Technology MT23108 InfiniHost (0001:07:00.0) WARN: : Failed to find lock pointer. WARN: : Error <-22> initializing memory locking. WARN: : Error <-22> initializing SDP IOCB table. WARN: : INIT: Error <-22> initializing connection table. From mst at mellanox.co.il Thu Jun 16 15:34:51 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 17 Jun 2005 01:34:51 +0300 Subject: [openib-general] Re: SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050616222657.GN577@kalmia.hozed.org> References: <20050615213047.GM577@kalmia.hozed.org> <20050616111255.A8268@topspin.com> <20050616190355.GB32088@mellanox.co.il> <20050616222657.GN577@kalmia.hozed.org> Message-ID: <20050616223451.GE32394@mellanox.co.il> Quoting r. Troy Benjegerdes : > Subject: Re: SDP: device mthca0 does not support fast memory regions > > On Thu, Jun 16, 2005 at 10:03:55PM +0300, Michael S. Tsirkin wrote: > > Quoting r. Libor Michalek : > > > Subject: Re: SDP: device mthca0 does not support fast memory regions > > > > > > On Wed, Jun 15, 2005 at 04:30:47PM -0500, Troy Benjegerdes wrote: > > > > I'm getting a 'SDP: device mthca0 does nto support fast memory regions' > > > > error on PPC64 systems.. Is there something that needs to be done for > > > > PPC, or could I have an older version of the mthca module hanging > > > > around? > > > > > > I'm not sure about the state or plans for FMR support on PPC64, Roland > > > will have a better idea. > > > > Should just work AFAIK. > > But maybe your card hides the onboard memory? > > Try lspci -vv to see if the card has > > the huge bar (100Mbyte or more) corresponding to that memory. > > Yes, the DDR is hidden on a couple of the machines, since they are IBM > Power4 systems, which have some stupid issue with the firmware. > > However, I'm getting the following problem on an Xserve G5: > > xl0:~# cat /proc/version > Linux version 2.6.11.11-G5 (root at xl0.fast) (gcc version 3.4.2) #1 SMP > Wed Jun 812:11:29 CDT 2005 > > ib_mthca: Mellanox InfiniBand HCA driver v0.06-pre (November 8, 2004) > ib_mthca: Initializing Mellanox Technology MT23108 InfiniHost > (0001:07:00.0) > WARN: : Failed to find lock pointer. > WARN: : Error <-22> initializing memory locking. > WARN: : Error <-22> initializing SDP IOCB table. > WARN: : INIT: Error <-22> initializing connection table. > Some issue with the hack used currently to mlock pages. You can try my patch http://openib.org/pipermail/openib-general/2005-May/006322.html that replaces that with get_user_pages (Libor is waiting on this patch till we solve the aio/synch completion race). -- MST From ftillier at silverstorm.com Thu Jun 16 15:35:52 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 16 Jun 2005 15:35:52 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <52mzpq7xjm.fsf@topspin.com> Message-ID: <000601c572c3$c19a20e0$9c5aa8c0@infiniconsys.com> > From: Roland Dreier [mailto:roland at topspin.com] > Sent: Thursday, June 16, 2005 3:22 PM > > Fab> I can see how to edit pages, but don't see any links or > Fab> buttons to create new pages... I'm likely just missing it. > Fab> I'll look some more, but if you have a pointer, that would be > Fab> great. > > Actually I guess you have to create a link on an existing page first, > either implicitly by adding something in CamelCase, or explicitly by > tagging it as a link. Then you can click on the link and start > editing your new page. Thanks. I've been trying to RTFM, and I think I'm now making forward progress. > Fab> The source repositories have to stay separate due to > Fab> licensing and hosting issues. > > Well, the supposed "licensing" issues never made any sense to me -- > what is the the legal difference between separate directories on one > server and separate servers in the same domain? But if it's OK to > share one wiki according to whatever the rules are, then I don't mind. There were also hosting issues that prevented the Windows code from sharing the same server as the Linux code. I didn't create either of the licensing or hosting issues - I just went along with them independent of my personal opinion. > Fab> If we don't want to share the Wiki, then the existing Wiki > Fab> (and all Linux related content) needs to move to a new > Fab> subdomain like linux.openib.org, with a top level page that > Fab> directs people to the appropriate operating system. > > That actually sounds like a good idea to me. I think having this would probably make sense too. The simplest way to do this would be to have the DNS entries updated and just point linux.openib.org to the same IP as openib.org. However, I would like to keep a common home page if possible. In fact, having a common Wiki at the home page gives the OpenIB web content as much of a consistent look and feel as possible, independent of the related projects. - Fab From ftillier at silverstorm.com Thu Jun 16 15:38:34 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 16 Jun 2005 15:38:34 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <20050616222712.GM9121@esmail.cup.hp.com> Message-ID: <000701c572c4$22317430$9c5aa8c0@infiniconsys.com> > From: Grant Grundler [mailto:iod00d at hp.com] > Sent: Thursday, June 16, 2005 3:27 PM > > I don't agree that all the existing linux content should be > mangled to include windows content on the same pages. > Windows folks should clone any pages needing more than a few > lines changed to accommodate them. I don't plan on mangling any Linux pages. Putting information for both projects on the same page is only going to confuse people. It's like peppering code with #ifdef - it gets ugly really fast. For the main Wiki page, I plan on having a link towards the top to the Windows page. I also plan on having a separate Windows FAQ so we don't confuse things. - Fab From roland at topspin.com Thu Jun 16 15:44:11 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 15:44:11 -0700 Subject: [openib-general] Re: SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050616222657.GN577@kalmia.hozed.org> (Troy Benjegerdes's message of "Thu, 16 Jun 2005 17:26:57 -0500") References: <20050615213047.GM577@kalmia.hozed.org> <20050616111255.A8268@topspin.com> <20050616190355.GB32088@mellanox.co.il> <20050616222657.GN577@kalmia.hozed.org> Message-ID: <52ekb19b2c.fsf@topspin.com> Troy> WARN: : Failed to find lock pointer. WARN: : Error <-22> Troy> initializing memory locking. WARN: : Error <-22> Troy> initializing SDP IOCB table. WARN: : INIT: Error <-22> Troy> initializing connection table. Oh yeah, the current SDP code has no chance of working on ppc64 -- it munges around looking for a function pointer, which can't work because the ppc64 ABI involves function descriptors. - R. From roland at topspin.com Thu Jun 16 15:52:57 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 15:52:57 -0700 Subject: [openib-general] [PATCH] Check copy_from_user in user_mad.c Message-ID: <527jgt9anq.fsf@topspin.com> We should be checking the return value of copy_from_user(), right? --- infiniband/core/user_mad.c (revision 2641) +++ infiniband/core/user_mad.c (working copy) @@ -355,18 +355,26 @@ static ssize_t ib_umad_write(struct file if (!rmpp_active) { /* Copy message from user into send buffer */ - copy_from_user(packet->msg->mad, - buf + sizeof(struct ib_user_mad), length); + if (copy_from_user(packet->msg->mad, + buf + sizeof(struct ib_user_mad), length)) { + ret = -EFAULT; + goto err_msg; + } } else { rmpp_hdr_size = sizeof(struct ib_mad_hdr) + sizeof(struct ib_rmpp_hdr); + /* Only copy MAD headers (RMPP header in place) */ memcpy(packet->msg->mad, packet->mad.data, sizeof(struct ib_mad_hdr)); + /* Now, copy rest of message from user into send buffer */ - copy_from_user(((struct ib_rmpp_mad *)packet->msg->mad)->data, - buf + sizeof(struct ib_user_mad) + rmpp_hdr_size, - length - rmpp_hdr_size); + if (copy_from_user(((struct ib_rmpp_mad *) packet->msg->mad)->data, + buf + sizeof (struct ib_user_mad) + rmpp_hdr_size, + length - rmpp_hdr_size)) { + ret = -EFAULT; + goto err_msg; + } } /* From libor at topspin.com Thu Jun 16 15:56:58 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 16 Jun 2005 15:56:58 -0700 Subject: [openib-general] Re: SDP: device mthca0 does not support fast memory regions In-Reply-To: <52ekb19b2c.fsf@topspin.com>; from roland@topspin.com on Thu, Jun 16, 2005 at 03:44:11PM -0700 References: <20050615213047.GM577@kalmia.hozed.org> <20050616111255.A8268@topspin.com> <20050616190355.GB32088@mellanox.co.il> <20050616222657.GN577@kalmia.hozed.org> <52ekb19b2c.fsf@topspin.com> Message-ID: <20050616155658.B8268@topspin.com> On Thu, Jun 16, 2005 at 03:44:11PM -0700, Roland Dreier wrote: > Troy> WARN: : Failed to find lock pointer. WARN: : Error <-22> > Troy> initializing memory locking. WARN: : Error <-22> > Troy> initializing SDP IOCB table. WARN: : INIT: Error <-22> > Troy> initializing connection table. > > Oh yeah, the current SDP code has no chance of working on ppc64 -- it > munges around looking for a function pointer, which can't work because > the ppc64 ABI involves function descriptors. Yeah, I had the same memory come back when I saw the error message. I'll go back and look to apply the patch Michael submitted. Also does FMR registration currently perform the dma mapping, or does the consumer need to do it? -Libor From roland at topspin.com Thu Jun 16 16:06:33 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 16 Jun 2005 16:06:33 -0700 Subject: [openib-general] Re: SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050616155658.B8268@topspin.com> (Libor Michalek's message of "Thu, 16 Jun 2005 15:56:58 -0700") References: <20050615213047.GM577@kalmia.hozed.org> <20050616111255.A8268@topspin.com> <20050616190355.GB32088@mellanox.co.il> <20050616222657.GN577@kalmia.hozed.org> <52ekb19b2c.fsf@topspin.com> <20050616155658.B8268@topspin.com> Message-ID: <521x719a12.fsf@topspin.com> Libor> Yeah, I had the same memory come back when I saw the error Libor> message. I'll go back and look to apply the patch Michael Libor> submitted. Also does FMR registration currently perform the Libor> dma mapping, or does the consumer need to do it? The consumer needs to do it mapping -- the FMR verbs are expecting DMA addresses, the same as ib_reg_phys_mr(). All the FMR code does is take your addresses and stick them in the HCA's translation tables, so the addresses you pass need to be bus addresses. - R. From asafs at voltaire.com Fri Jun 17 01:43:04 2005 From: asafs at voltaire.com (Asaf Somekh) Date: Fri, 17 Jun 2005 11:43:04 +0300 Subject: [openib-general] OpenIB Reception at ISC2005 in Heidelberg - Invitation Message-ID: <5CE025EE7D88BA4599A2C8FEFCF226F546ECAD@taurus.voltaire.com> The OpenIB Alliance will be holding a reception for the OpenIB community at ISC2005 International Supercomputing Conference in Heidelberg Germany YOU ARE INVITED TO ATTEND THE RECEPTION When: Tue June 21st 2005, 6:00-8:00PM Where: Weisser Bock Hotel Restaurant & Bar Grosse Mautelgasse 24, Heidelberg Only blocks away from the convention center - Attached is a map with dirctions RSVP: Please reply to this email or contact me at asafs at voltaire.com +972 54 480 5283 Asaf Somekh OpenIB Alliance -------------- next part -------------- A non-text attachment was scrubbed... Name: ISC2005 OpenIB Invitation.jpg Type: image/jpeg Size: 425497 bytes Desc: ISC2005 OpenIB Invitation.jpg URL: From halr at voltaire.com Fri Jun 17 04:45:30 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 17 Jun 2005 07:45:30 -0400 Subject: [openib-general] Re: [PATCH] Check copy_from_user in user_mad.c In-Reply-To: <527jgt9anq.fsf@topspin.com> References: <527jgt9anq.fsf@topspin.com> Message-ID: <1119008703.4506.788.camel@localhost.localdomain> On Thu, 2005-06-16 at 18:52, Roland Dreier wrote: > We should be checking the return value of copy_from_user(), right? Right. -- Hal From mst at mellanox.co.il Fri Jun 17 08:00:06 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 17 Jun 2005 18:00:06 +0300 Subject: [openib-general] Re: Re: SDP: device mthca0 does not support fast memory regions In-Reply-To: <20050616155658.B8268@topspin.com> References: <20050615213047.GM577@kalmia.hozed.org> <20050616111255.A8268@topspin.com> <20050616190355.GB32088@mellanox.co.il> <20050616222657.GN577@kalmia.hozed.org> <52ekb19b2c.fsf@topspin.com> <20050616155658.B8268@topspin.com> Message-ID: <20050617150005.GA8463@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: Re: SDP: device mthca0 does not support fast memory regions > I'll go back and look to apply the patch Michael submitted. One obvious improvement that is yet to be implemented is to handle send iocbs from the interrupt context directly. -- MST From hch at lst.de Fri Jun 17 09:32:11 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 17 Jun 2005 18:32:11 +0200 Subject: [openib-general] [PATCH] cleanup dat provider registration Message-ID: <20050617163211.GA19071@lst.de> This huge patch revamps almost everything under dat, and merges the three layers of external API, dynamic registry and dictinary into a single one, greatly streamlining all of the code. Note that I don't have any RDMA hardware to actually takes this, so it probably needs some testing first. Signed-off-by: Christoph Hellwig Index: linux-kernel/dat/dr.c =================================================================== --- linux-kernel/dat/dr.c (revision 2647) +++ linux-kernel/dat/dr.c (working copy) @@ -1,242 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Dynamic Registry (DR) functions and data structures - * - * $Id$ - */ - -#include "core.h" -#include "dictionary.h" -#include "dr.h" - -struct dat_dr_entry { - int ref_count; - struct dat_provider_info info; - DAT_IA_OPEN_FUNC ia_open_func; -}; - -static spinlock_t g_dr_lock = SPIN_LOCK_UNLOCKED; -static struct dat_dictionary g_dr_dictionary = - DAT_DICTIONARY_INIT(g_dr_dictionary); - -void dat_dr_fini(void) -{ - dat_dictionary_destroy(&g_dr_dictionary); -} - -u32 dat_dr_insert(const struct dat_provider_info *info, - const DAT_IA_OPEN_FUNC ia_open_func) -{ - struct dat_dr_entry *data; - struct dat_dictionary_entry *dict_entry; - unsigned long flags; - u32 status; - - data = kmalloc(sizeof *data, GFP_ATOMIC); - if (!data) { - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - - data->ref_count = 0; - data->info = *info; - data->ia_open_func = ia_open_func; - - dict_entry = NULL; - status = dat_dictionary_entry_create(&dict_entry); - if (DAT_SUCCESS != status) - goto bail; - - spin_lock_irqsave(&g_dr_lock, flags); - - status = dat_dictionary_insert(&g_dr_dictionary, info, dict_entry, - data); - - spin_unlock_irqrestore(&g_dr_lock, flags); - -bail: - if (DAT_SUCCESS != status) { - if (NULL != data) - kfree(data); - - if (NULL != dict_entry) - dat_dictionary_entry_destroy(dict_entry); - } - - return status; -} - -u32 dat_dr_remove(const struct dat_provider_info *info) -{ - struct dat_dr_entry *data; - struct dat_dictionary_entry *dict_entry; - unsigned long flags; - u32 status; - - dict_entry = NULL; - spin_lock_irqsave(&g_dr_lock, flags); - - status = dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); - - if (DAT_SUCCESS != status) - goto bail; /* return status from dat_dictionary_search() */ - - if (0 != data->ref_count) { - status = DAT_ERROR(DAT_PROVIDER_IN_USE, 0); - goto bail; - } - - status = dat_dictionary_remove(&g_dr_dictionary, info, &dict_entry, - (void **) &data); - if (DAT_SUCCESS != status) - goto bail; /* return status from dat_dictionary_remove() */ - - kfree(data); - -bail: - spin_unlock_irqrestore(&g_dr_lock, flags); - - if (NULL != dict_entry) - dat_dictionary_entry_destroy(dict_entry); - - return status; -} - -u32 dat_dr_provider_open(const struct dat_provider_info *info, - DAT_IA_OPEN_FUNC *p_ia_open_func) -{ - struct dat_dr_entry *data; - unsigned long flags; - u32 status; - - spin_lock_irqsave(&g_dr_lock, flags); - - status = dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); - - spin_unlock_irqrestore(&g_dr_lock, flags); - - if (DAT_SUCCESS == status) { - data->ref_count++; - *p_ia_open_func = data->ia_open_func; - } - - return status; -} - -u32 dat_dr_provider_close(const struct dat_provider_info *info) -{ - struct dat_dr_entry *data; - unsigned long flags; - u32 status; - - spin_lock_irqsave(&g_dr_lock, flags); - - status = dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); - - spin_unlock_irqrestore(&g_dr_lock, flags); - - if (DAT_SUCCESS == status) - data->ref_count--; - - return status; -} - -u32 dat_dr_size(int *size) -{ - return dat_dictionary_size(&g_dr_dictionary, size); -} - -u32 dat_dr_list(int max_to_return, int *entries_returned, - struct dat_provider_info *(dat_provider_list[])) -{ - struct dat_dr_entry **array = NULL; - int i, array_size; - unsigned long flags; - u32 status = DAT_SUCCESS; - - /* - * The dictionary size may increase between the call to - * dat_dictionary_size() and dat_dictionary_enumerate(). - * Therefore we loop until a successful enumeration is made. - */ - *entries_returned = 0; - for (;;) { - status = dat_dictionary_size(&g_dr_dictionary, &array_size); - if (DAT_SUCCESS != status) - goto bail; - - if (0 == array_size) { - status = DAT_SUCCESS; - goto bail; - } - - array = kmalloc(array_size * sizeof *array, GFP_ATOMIC); - if (!array) { - status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - - spin_lock_irqsave(&g_dr_lock, flags); - - status = dat_dictionary_enumerate(&g_dr_dictionary, - (void **) array, array_size); - - spin_unlock_irqrestore(&g_dr_lock, flags); - - if (DAT_SUCCESS == status) - break; - else { - kfree(array); - array = NULL; - continue; - } - } - - for (i = 0; (i < max_to_return) && (i < array_size); i++) { - if (NULL == dat_provider_list[i]) { - status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - *dat_provider_list[i] = array[i]->info; - } - - *entries_returned = i; - -bail: - if (NULL != array) - kfree(array); - - return status; -} Index: linux-kernel/dat/dictionary.c =================================================================== --- linux-kernel/dat/dictionary.c (revision 2647) +++ linux-kernel/dat/dictionary.c (working copy) @@ -1,229 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * A dictionary data structure implemented with a linked list - * - * $Id$ - */ - -#include "dictionary.h" - -static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, - struct dat_provider_info *new_key); - -static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *a, - const struct dat_provider_info *b); - -void dat_dictionary_destroy(struct dat_dictionary *dictionary) -{ - struct list_head *cur_list, *next_list; - struct dat_dictionary_entry *cur_entry; - - BUG_ON(NULL == dictionary); - - list_for_each_safe(cur_list, next_list, &dictionary->list) { - cur_entry = list_entry(cur_list, struct dat_dictionary_entry, - list); - list_del(&cur_entry->list); - kfree(cur_entry); - } -} - -u32 dat_dictionary_size(const struct dat_dictionary *dictionary, int *size) -{ - BUG_ON(NULL == dictionary); - BUG_ON(NULL == size); - - *size = dictionary->size; - - return DAT_SUCCESS; -} - -u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry) -{ - struct dat_dictionary_entry *entry; - u32 status = DAT_SUCCESS; - - BUG_ON(NULL == p_entry); - - entry = kmalloc(sizeof *entry, GFP_ATOMIC); - if (!entry) { - status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - *p_entry = entry; - -bail: - return status; -} - -void dat_dictionary_entry_destroy(const struct dat_dictionary_entry *entry) -{ - kfree(entry); -} - -u32 dat_dictionary_insert(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - struct dat_dictionary_entry *entry, void *data) -{ - u32 status; - - BUG_ON(NULL == dictionary); - BUG_ON(NULL == entry); - - if (DAT_SUCCESS == dat_dictionary_search(dictionary, key, NULL)) { - status = DAT_ERROR(DAT_PROVIDER_ALREADY_REGISTERED, 0); - goto bail; - } - - status = dat_dictionary_key_dup(key, &entry->key); - if (DAT_SUCCESS != status) - goto bail; - - entry->data = data; - - /* insert node at end of list to preserve registration order */ - list_add_tail(&entry->list, &dictionary->list); - - dictionary->size++; - -bail: - return status; -} - -u32 dat_dictionary_search(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, void **p_data) -{ - struct dat_dictionary_entry *cur_entry; - u32 status; - - BUG_ON(NULL == dictionary); - BUG_ON(NULL == key); - - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); - - list_for_each_entry(cur_entry, &dictionary->list, list) { - if (TRUE == dat_dictionary_key_is_equal(&cur_entry->key, key)) { - if (NULL != p_data) - *p_data = cur_entry->data; - - status = DAT_SUCCESS; - goto bail; - } - } - -bail: - return status; -} - -u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, void *array[], - int array_size) -{ - struct dat_dictionary_entry *cur_entry; - int i; - u32 status; - - BUG_ON(NULL == dictionary); - BUG_ON(NULL == array); - - status = DAT_SUCCESS; - - if (array_size < dictionary->size) { - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); - goto bail; - } - - i = 0; - list_for_each_entry(cur_entry, &dictionary->list, list) - array[i++] = cur_entry->data; - -bail: - return status; -} - -u32 dat_dictionary_remove(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - struct dat_dictionary_entry **p_entry, void **p_data) -{ - struct list_head *cur_list, *next_list; - struct dat_dictionary_entry *cur_entry; - u32 status; - - BUG_ON(NULL == dictionary); - BUG_ON(NULL == key); - BUG_ON(NULL == p_entry); - - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); - - list_for_each_safe(cur_list, next_list, &dictionary->list) { - cur_entry = list_entry(cur_list, struct dat_dictionary_entry, - list); - if (TRUE == dat_dictionary_key_is_equal(&cur_entry->key, key)) { - if (NULL != p_data) - *p_data = cur_entry->data; - - list_del(&cur_entry->list); - - *p_entry = cur_entry; - - dictionary->size--; - - status = DAT_SUCCESS; - goto bail; - } - } - -bail: - return status; -} - -static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, - struct dat_provider_info *new_key) -{ - BUG_ON(NULL == old_key); - BUG_ON(NULL == new_key); - - strncpy(new_key->ia_name, old_key->ia_name, DAT_NAME_MAX_LENGTH); - return DAT_SUCCESS; -} - -static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *a, - const struct dat_provider_info *b) -{ - BUG_ON(NULL == a); - BUG_ON(NULL == b); - - if ((strlen(a->ia_name) == strlen(b->ia_name)) && - (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name)))) - return TRUE; - else - return FALSE; -} Index: linux-kernel/dat/core.c =================================================================== --- linux-kernel/dat/core.c (revision 2647) +++ linux-kernel/dat/core.c (working copy) @@ -1,75 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Core DAT registry functions and data structures - * - * $Id$ - */ - -#include "core.h" -#include "dr.h" - -MODULE_LICENSE("Dual BSD/GPL"); -MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); -MODULE_AUTHOR("James Lentini"); - -static DAT_DBG_MASK g_dbg_mask = DAT_DBG_TYPE_ERROR; -module_param_named(dbg_mask, g_dbg_mask, int, 0644); -MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug message types."); - -void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...) -{ - static char buf[1024]; - - if (type & g_dbg_mask) { - va_list args; - - va_start(args, fmt); - vsnprintf(buf, sizeof buf, fmt, args); - printk(KERN_ALERT "DAT: %s", buf); - va_end(args); - } -} - -static int __init dat_init(void) -{ - dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry started\n"); - return 0; -} - -module_init(dat_init); - -static void __exit dat_fini(void) -{ - dat_dr_fini(); - - dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry stopped\n"); -} - -module_exit(dat_fini); Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2647) +++ linux-kernel/dat/api.c (working copy) @@ -26,15 +26,53 @@ * and/or other materials provided with the distribution. */ -/* - * DAT functions for consumers - * - * $Id: consumer.c 2251 2005-05-04 05:57:41Z jlentini $ - */ +#include +#include +#include +#include +#include -#include "core.h" -#include "dr.h" +#include + +enum dat_dbg_type { + DAT_DBG_TYPE_ERROR = (1 << 0), + DAT_DBG_TYPE_GENERIC = (1 << 1), + DAT_DBG_TYPE_DR = (1 << 2), + DAT_DBG_TYPE_PROVIDER_API = (1 << 3), + DAT_DBG_TYPE_CONSUMER_API = (1 << 4), + DAT_DBG_TYPE_ALL = 0x1f +}; + +static int dat_dbg_mask = DAT_DBG_TYPE_ERROR; +module_param_named(dbg_mask, dat_dbg_mask, int, 0644); +MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug message types."); + +struct dat_dictionary_entry { + struct list_head list; + struct dat_provider_info info; + DAT_IA_OPEN_FUNC ia_open_func; + int ref_count; +}; + +static LIST_HEAD(dat_provider_list); +static int dat_provider_list_size; +static DEFINE_SPINLOCK(dat_provider_list_lock); + +static void dat_dbg_print(enum dat_dbg_type type, const char *fmt, ...) +{ + static char buf[1024]; + + if (type & dat_dbg_mask) { + va_list args; + + va_start(args, fmt); + vsnprintf(buf, sizeof buf, fmt, args); + printk(KERN_ALERT "DAT: %s", buf); + va_end(args); + } +} + static u32 dat_strerror_major(u32 value, const char **message) { switch (DAT_GET_TYPE(value)) { @@ -350,21 +388,39 @@ } } +static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *a, + const struct dat_provider_info *b) +{ + if (strlen(a->ia_name) == strlen(b->ia_name) && + !strncmp(a->ia_name, b->ia_name, strlen(a->ia_name))) + return 1; + return 0; +} + +static struct dat_dictionary_entry *dat_dictionary_search( + struct list_head *list, const struct dat_provider_info *key) +{ + struct dat_dictionary_entry *entry; + + list_for_each_entry(entry, list, list) + if (dat_dictionary_key_is_equal(&entry->info, key)) + return entry; + + return NULL; +} + u32 dat_ia_open(const char *name, int async_event_qlen, struct dat_evd **async_event_handle, struct dat_ia **ia) { - DAT_IA_OPEN_FUNC ia_open_func; struct dat_provider_info info; - u32 status; + struct dat_dictionary_entry *data; + unsigned long flags; size_t len; dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "%s(name:%s, async_qlen:%i, ver:%x.%x, thrd_safe:%x)\n", __func__, name, async_event_qlen); - if (DAT_IS_BAD_POINTER(name)) - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); - len = strlen(name); if (DAT_NAME_MAX_LENGTH < len) @@ -373,62 +429,67 @@ strncpy(info.ia_name, name, len); info.ia_name[len] = '\0'; - status = dat_dr_provider_open(&info, &ia_open_func); - if (status != DAT_SUCCESS) { + spin_lock_irqsave(&dat_provider_list_lock, flags); + data = dat_dictionary_search(&dat_provider_list, &info); + spin_unlock_irqrestore(&dat_provider_list_lock, flags); + + if (!data) { dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "%s: IA [%s] not found in dynamic registry\n", __func__, name); - return status; + return DAT_ERROR(DAT_PROVIDER_NOT_FOUND, + DAT_NAME_NOT_REGISTERED); } - return ia_open_func(name, async_event_qlen, async_event_handle, ia); + data->ref_count++; + return data->ia_open_func(name, async_event_qlen, async_event_handle, + ia); } - EXPORT_SYMBOL(dat_ia_open); -u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) +u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags close_flags) { struct dat_provider *provider; - struct dat_provider_attr provider_attr; - u32 status; + struct dat_provider_info info; + u32 status = DAT_SUCCESS; const char *ia_name; + struct dat_dictionary_entry *data; + unsigned long flags; + size_t len; dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "%s(ia:%x, flags:%x)\n", __func__, ia, flags); + "%s(ia:%x, flags:%x)\n", __func__, ia, close_flags); provider = ia->common.provider; ia_name = provider->device_name; - if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, - &provider_attr))) + status = provider->ia_close_func(ia, close_flags); + if (status != DAT_SUCCESS) { dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "dat_ia_query for IA %s failed\n", ia_name); - else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, "close function for IA %s failed\n", ia_name); - else { - struct dat_provider_info info; - size_t len; + return status; + } - len = strlen(ia_name); + len = strlen(ia_name); - BUG_ON(DAT_NAME_MAX_LENGTH < len); + BUG_ON(DAT_NAME_MAX_LENGTH < len); - strncpy(info.ia_name, ia_name, len); - info.ia_name[len] = '\0'; + strncpy(info.ia_name, ia_name, len); + info.ia_name[len] = '\0'; - status = dat_dr_provider_close(&info); - if (DAT_SUCCESS != status) - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "dynamic registry unable to close " - "provider for IA %s\n", - ia_name); + spin_lock_irqsave(&dat_provider_list_lock, flags); + data = dat_dictionary_search(&dat_provider_list, &info); + if (data) { + data->ref_count--; + } else { + status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, + DAT_NAME_NOT_REGISTERED); } + spin_unlock_irqrestore(&dat_provider_list_lock, flags); return status; } - EXPORT_SYMBOL(dat_ia_close); u32 dat_strerror(u32 value, const char **major_message, @@ -454,69 +515,138 @@ return DAT_SUCCESS; } - EXPORT_SYMBOL(dat_strerror); u32 dat_registry_add_provider(const struct dat_provider *provider, - const struct dat_provider_info *provider_info) + const struct dat_provider_info *info) { - dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, - "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", - __func__, provider_info->ia_name); + struct dat_dictionary_entry *entry; + unsigned long flags; + u32 status = DAT_SUCCESS; - if (DAT_IS_BAD_POINTER(provider)) - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); + entry = kmalloc(sizeof(*entry), GFP_ATOMIC); + if (!entry) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto out; + } - if (DAT_IS_BAD_POINTER(provider_info)) - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + entry->ref_count = 0; + entry->info = *info; + entry->ia_open_func = provider->ia_open_func; - return dat_dr_insert(provider_info, provider->ia_open_func); + spin_lock_irqsave(&dat_provider_list_lock, flags); + if (dat_dictionary_search(&dat_provider_list, info)) { + status = DAT_ERROR(DAT_PROVIDER_ALREADY_REGISTERED, 0); + goto out_unlock; + } + + strncpy(entry->info.ia_name, info->ia_name, DAT_NAME_MAX_LENGTH); + + /* insert node at end of list to preserve registration order */ + list_add_tail(&entry->list, &dat_provider_list); + dat_provider_list_size++; + + out_unlock: + spin_unlock_irqrestore(&dat_provider_list_lock, flags); + out: + if (status != DAT_SUCCESS) + kfree(entry); + return status; } +EXPORT_SYMBOL(dat_registry_add_provider); -EXPORT_SYMBOL(dat_registry_remove_provider); - u32 dat_registry_remove_provider(const struct dat_provider *provider, - const struct dat_provider_info *provider_info) + const struct dat_provider_info *info) { - dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, "%s(ia:%s)\n", __func__, - provider_info->ia_name); + struct dat_dictionary_entry *cur, *next; + unsigned long flags; + u32 status = DAT_SUCCESS; - if (DAT_IS_BAD_POINTER(provider)) - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); + spin_lock_irqsave(&dat_provider_list_lock, flags); + list_for_each_entry_safe(cur, next, &dat_provider_list, list) { + if (dat_dictionary_key_is_equal(&cur->info, info)) { + if (cur->ref_count) { + status = DAT_ERROR(DAT_PROVIDER_IN_USE, 0); + goto out; + } - return dat_dr_remove(provider_info); + list_del(&cur->list); + dat_provider_list_size--; + + kfree(cur); + goto out; + } + } + + status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); + out: + spin_unlock_irqrestore(&dat_provider_list_lock, flags); + return status; } +EXPORT_SYMBOL(dat_registry_remove_provider); -EXPORT_SYMBOL(dat_registry_add_provider); - u32 dat_registry_list_providers(int max_to_return, int *entries_returned, - struct dat_provider_info *(dat_provider_list[])) + struct dat_provider_info *provider_list[]) { + struct dat_dictionary_entry *cur; + unsigned long flags; + int i = 0; u32 status = DAT_SUCCESS; - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, - "%s(max_to_return:%i, entries_rtrn:%p, prvdr_list:%p)\n", - __func__, max_to_return, entries_returned, - dat_provider_list); - - if ((DAT_IS_BAD_POINTER(entries_returned))) - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - - if (0 != max_to_return && (DAT_IS_BAD_POINTER(dat_provider_list))) - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - - if (0 == max_to_return) { + if (!max_to_return) { /* the user is allowed to call with max_to_return set to zero. * in which case we simply return (in *entries_returned) the * number of providers currently installed. We must also * (per spec) return an error */ - (void)dat_dr_size(entries_returned); + *entries_returned = dat_provider_list_size; return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); - } else - status = dat_dr_list(max_to_return, entries_returned, - dat_provider_list); + } + + *entries_returned = 0; + + spin_lock_irqsave(&dat_provider_list_lock, flags); + if (!dat_provider_list_size) + goto out_unlock; + + list_for_each_entry(cur, &dat_provider_list, list) { + if (i >= max_to_return) + break; + if (!provider_list[i]) { + status = + DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto out_unlock; + } + provider_list[i++] = &cur->info; + } + *entries_returned = i; + + out_unlock: + spin_unlock_irqrestore(&dat_provider_list_lock, flags); return status; } +EXPORT_SYMBOL(dat_registry_list_providers); -EXPORT_SYMBOL(dat_registry_list_providers); +static int __init dat_init(void) +{ + dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry started\n"); + return 0; +} + +static void __exit dat_fini(void) +{ + struct dat_dictionary_entry *cur, *next; + + list_for_each_entry_safe(cur, next, &dat_provider_list, list) { + list_del(&cur->list); + kfree(cur); + } +} + +MODULE_LICENSE("Dual BSD/GPL"); +MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); +MODULE_AUTHOR("James Lentini"); + +module_init(dat_init); +module_exit(dat_fini); Index: linux-kernel/dat/dictionary.h =================================================================== --- linux-kernel/dat/dictionary.h (revision 2647) +++ linux-kernel/dat/dictionary.h (working copy) @@ -1,98 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * dictionary data structure - * - * $Id$ - */ - -#ifndef DICTIONARY_H -#define DICTIONARY_H - -#include "core.h" - -/********************************************************************* - * * - * Structures * - * * - *********************************************************************/ - -struct dat_dictionary_entry { - struct list_head list; - struct dat_provider_info key; - void *data; -}; - -struct dat_dictionary { - struct list_head list; - int size; -}; - -/********************************************************************* - * * - * Macros * - * * - *********************************************************************/ - -#define DAT_DICTIONARY_INIT(name) { LIST_HEAD_INIT((name).list), 0 } - -/********************************************************************* - * * - * Function Prototypes * - * * - *********************************************************************/ - -extern void dat_dictionary_destroy(struct dat_dictionary *dictionary); - -extern u32 dat_dictionary_size(const struct dat_dictionary *dictionary, - int *size); - -extern u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry); - -extern void dat_dictionary_entry_destroy( - const struct dat_dictionary_entry *entry); - -extern u32 dat_dictionary_insert(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - struct dat_dictionary_entry *entry, - void *data); - -extern u32 dat_dictionary_search(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - void **p_data); - -extern u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, - void *array[], const int array_size); - -extern u32 dat_dictionary_remove(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - struct dat_dictionary_entry **p_entry, - void **p_data); - -#endif /* DICTIONARY_H */ Index: linux-kernel/dat/dr.h =================================================================== --- linux-kernel/dat/dr.h (revision 2647) +++ linux-kernel/dat/dr.h (working copy) @@ -1,57 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Dynamic Registry (DR) functions and data stucture - * - * $Id$ - */ - -#ifndef DR_H -#define DR_H - -#include - -extern void dat_dr_fini(void); - -extern u32 dat_dr_insert(const struct dat_provider_info *info, - const DAT_IA_OPEN_FUNC ia_open_func); - -extern u32 dat_dr_remove(const struct dat_provider_info *info); - -extern u32 dat_dr_provider_open(const struct dat_provider_info *info, - DAT_IA_OPEN_FUNC *p_ia_open_func); - -extern u32 dat_dr_provider_close(const struct dat_provider_info *info); - -extern u32 dat_dr_size(int *size); - -extern u32 dat_dr_list(int max_to_return, int *entries_returned, - struct dat_provider_info *(dat_provider_list[])); - -#endif /* DR_H */ Index: linux-kernel/dat/Makefile =================================================================== --- linux-kernel/dat/Makefile (revision 2647) +++ linux-kernel/dat/Makefile (working copy) @@ -5,8 +5,4 @@ obj-$(CONFIG_DAT) += dat.o -dat-y := \ - api.o \ - core.o \ - dictionary.o \ - dr.o +dat-y += api.o Index: linux-kernel/dat/core.h =================================================================== --- linux-kernel/dat/core.h (revision 2647) +++ linux-kernel/dat/core.h (working copy) @@ -1,61 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * core DAT registry functions and data structures - * - * $Id$ - */ - -#ifndef CORE_H -#define CORE_H - -#include -#include -#include -#include -#include - -#include - -#define DAT_IS_BAD_POINTER(p) ( NULL == (p) ) - -typedef enum { - DAT_DBG_TYPE_ERROR = (1 << 0), - DAT_DBG_TYPE_GENERIC = (1 << 1), - DAT_DBG_TYPE_DR = (1 << 2), - DAT_DBG_TYPE_PROVIDER_API = (1 << 3), - DAT_DBG_TYPE_CONSUMER_API = (1 << 4), - DAT_DBG_TYPE_ALL = 0x1f -} DAT_DBG_TYPE; - -typedef int DAT_DBG_MASK; - -extern void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...); - -#endif /* CORE_H */ From hch at lst.de Fri Jun 17 09:40:55 2005 From: hch at lst.de (Christoph Hellwig) Date: Fri, 17 Jun 2005 18:40:55 +0200 Subject: [openib-general] Re: [PATCH][kdapl]update dat_rmr_bind API & delete dapl_hash In-Reply-To: <7414e54805061612203fc942cb@mail.gmail.com> References: <7414e54805061612203fc942cb@mail.gmail.com> Message-ID: <20050617164055.GA19221@lst.de> On Thu, Jun 16, 2005 at 09:20:27PM +0200, Itamar & Shira Rabenstein wrote: > >Itamar, > > > >Can you seperate the RMR api change and hash table removal into a > >seperate patch? > > I dont see why, the new input param to dat_rmr_bind (lmr_handle) come > in order that we will be able to remove the hash table so this is one > patch. In the Linux world we generally prefer faine-grained patches. While the RMR api change is a prerequsite to remove the hash code it's not directly related. > >Merging the HCA and IA code into the dapl_ia.[ch] files doesn't feel > >right to me. I think of the HCA as a DAT object just like IAs, EPs, > >LMRs, etc, and hence it should have its own file. > > > >james > > > There are 4 function in this files: > > 2 were merged to provider code: > dapl_hca_alloc(13 lines) :which i think is part of the provider code . > dapl_hca_free (1 line) == kfree , no need a function for this. > 2 were merged to ia code: > dapl_hca_link_ia (3 lines ) : no need a function to call list add > dapl_hca_unlink_ia(3 lines) : no need a function to call list_del > > I belive that we need to make the code more readable and to have this functions > is not a good idea and to add 2 files for this is for sure a bad idea ... I agree your reasoning for the second set of functions at least. But again this is not related to the API change at all, please separate it out to a patch of it's own, applying ontop of the API change and hash removal patches. From libor at topspin.com Fri Jun 17 10:29:32 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 17 Jun 2005 10:29:32 -0700 Subject: [openib-general] [PATCH][SDP] Further simplifications. Message-ID: <20050617102932.A9204@topspin.com> Following patch simplifies some code paths, including some error handling in the data paths. 3 files changed, 143 insertions(+), 377 deletions(-) -Libor Index: sdp_send.c =================================================================== --- sdp_send.c (revision 2628) +++ sdp_send.c (working copy) @@ -122,15 +122,6 @@ conn->send_cons = 0; } /* - * check queue membership. (first send attempt vs. flush) - */ - if (sdp_desc_q_member((struct sdpc_desc *) buff)) - sdp_desc_q_remove((struct sdpc_desc *) buff); - /* - * save the buffer for the event handler. - */ - sdp_buff_q_put_tail(&conn->send_post, buff); - /* * post send */ buff->sge.length = buff->tail - buff->data; @@ -152,11 +143,18 @@ sdp_buff_q_size(&conn->send_post), sdp_desc_q_size(&conn->r_src), sdp_desc_q_size(&conn->w_snk)); - - (void)sdp_buff_q_get_tail(&conn->send_post); goto done; } /* + * check queue membership. (first send attempt vs. flush) + */ + if (sdp_desc_q_member((struct sdpc_desc *) buff)) + sdp_desc_q_remove((struct sdpc_desc *) buff); + /* + * save the buffer for the event handler. + */ + sdp_buff_q_put_tail(&conn->send_post, buff); + /* * source cancels require us to save the sequence number * for validation of the cancel's completion. */ @@ -352,19 +350,6 @@ (unsigned)(buff->tail - buff->data), advt->size); /* - * dequeue if needed and the queue buffer - */ - if (sdp_desc_q_member((struct sdpc_desc *) buff) > 0) - sdp_desc_q_remove((struct sdpc_desc *) buff); - - sdp_desc_q_put_tail(&conn->w_snk, (struct sdpc_desc *)buff); - /* - * update send queue depth - */ - conn->s_wq_size++; - conn->send_pipe -= buff->data_size; - conn->oob_offset -= (conn->oob_offset > 0) ? buff->data_size : 0; - /* * post RDMA */ buff->sge.addr = virt_to_phys(buff->data); @@ -378,11 +363,15 @@ result = ib_post_send(conn->qp, &send_param, &bad_wr); if (result) { sdp_dbg_warn(conn, "Error <%d> posting rdma write", result); - - conn->s_wq_size--; goto error; } /* + * update send queue depth + */ + conn->s_wq_size++; + conn->send_pipe -= buff->data_size; + conn->oob_offset -= (conn->oob_offset > 0) ? buff->data_size : 0; + /* * If the available space is smaller then send size, complete the * advertisment. */ @@ -401,7 +390,14 @@ sdp_advt_destroy(sdp_advt_q_get(&conn->snk_pend)); conn->snk_recv--; } + /* + * dequeue buffer if needed and move to active queue + */ + if (sdp_desc_q_member((struct sdpc_desc *) buff) > 0) + sdp_desc_q_remove((struct sdpc_desc *) buff); + sdp_desc_q_put_tail(&conn->w_snk, (struct sdpc_desc *)buff); + return 0; error: return result; @@ -906,10 +902,10 @@ if (!sdp_advt_q_look(&conn->snk_pend) || (((struct sdpc_buff *)element)->flags & SDP_BUFF_F_OOB_PRES)) result = sdp_send_data_buff_post(conn, - (struct sdpc_buff *)element); + (struct sdpc_buff *)element); else result = sdp_send_data_buff_snk(conn, - (struct sdpc_buff *)element); + (struct sdpc_buff *)element); return result; } @@ -927,30 +923,17 @@ * non-zero result is generated. * (positive: no space; negative: error) */ - while (!result && - (element = sdp_desc_q_look_head(&conn->send_queue))) { + while ((element = sdp_desc_q_look_head(&conn->send_queue))) { result = sdp_send_data_queue_test(conn, element); - if (!result) - continue; - /* - * error - */ - if (result < 0) { - sdp_dbg_warn(conn, - "Error <%d> post data <%d> during flush", - result, element->type); - /* - * check for dangling element reference, - * since called functions can dequeue the - * element, and not know how to requeue it. - */ - if (!sdp_desc_q_member(element)) - sdp_desc_q_put_head(&conn->send_queue, - element); - } + if (result) + break; } + if (result < 0) + sdp_dbg_warn(conn, "Error <%d> post data <%d> during flush", + result, element->type); + return result; } @@ -1117,24 +1100,18 @@ * As long as there are buffers, try to post until a non-zero * result is generated. (positive: no space; negative: error) */ - while (!result && - (element = sdp_desc_q_look_head(&conn->send_ctrl))) { + while ((element = sdp_desc_q_look_head(&conn->send_ctrl))) { result = sdp_send_ctrl_buff_test(conn, (struct sdpc_buff *)element); - if (!result) - continue; - - if (result < 0) { - sdp_dbg_warn(conn, - "Error <%d> failed to flush control msg", - result); - - if (!sdp_desc_q_member(element)) - sdp_desc_q_put_head(&conn->send_ctrl, element); - } + if (result) + break; } + if (result < 0) + sdp_dbg_warn(conn, "Error <%d> failed to flush control msg", + result); + return result; } @@ -1543,10 +1520,8 @@ /* * check mode */ - if (conn->recv_mode != SDP_MODE_PIPE) { - result = -EPROTO; - goto error; - } + if (conn->recv_mode != SDP_MODE_PIPE) + return -EPROTO; /* * create the message, which contains just the bsdh header. * (don't need to worry about header space reservation) @@ -1847,25 +1822,27 @@ * for transmission, and the remote host needs to be notified of * present data. (rdma ping-pong letency test...) */ - if (!sdp_desc_q_size(&conn->send_queue)) { - /* - * might be more aggressive then we want it to be. maybe - * check if the active sink queue is empty as well? - */ - advt = sdp_advt_q_look(&conn->snk_pend); - if (advt && advt->post > 0) { + if (sdp_desc_q_size(&conn->send_queue)) + return 0; - result = sdp_send_ctrl_rdma_wr(conn, advt->post); - SDP_EXPECT(result >= 0); + /* + * might be more aggressive then we want it to be. maybe + * check if the active sink queue is empty as well? + */ + advt = sdp_advt_q_look(&conn->snk_pend); + if (!advt || !advt->post) + return 0; - sdp_advt_destroy(sdp_advt_q_get(&conn->snk_pend)); - /* - * update sink advertisments. - */ - conn->snk_recv--; - } - } + result = sdp_send_ctrl_rdma_wr(conn, advt->post); + if (result < 0) + return result; + sdp_advt_destroy(sdp_advt_q_get(&conn->snk_pend)); + /* + * update sink advertisments. + */ + conn->snk_recv--; + return 0; } @@ -1920,7 +1897,7 @@ /* * see if there is enough buffer to wake/notify writers */ - sdp_inet_wake_send(conn->sk); /* conn->sk->write_space(conn->sk); */ + sdp_inet_wake_send(conn->sk); return 0; done: Index: sdp_sent.c =================================================================== --- sdp_sent.c (revision 2628) +++ sdp_sent.c (working copy) @@ -111,202 +111,16 @@ return result; } -static int sdp_sent_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - return 0; -} - -static int sdp_sent_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - struct msg_hdr_rwch *rwch; - - rwch = (struct msg_hdr_rwch *) buff->data; - buff->data = buff->data + sizeof(struct msg_hdr_rwch); - - sdp_msg_net_to_cpu_rwch(rwch); - - return 0; -} - -static int sdp_sent_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - struct msg_hdr_rrch *rrch; - - rrch = (struct msg_hdr_rrch *) buff->data; - buff->data = buff->data + sizeof(struct msg_hdr_rrch); - - sdp_msg_net_to_cpu_rrch(rrch); - - return 0; -} - -static int sdp_sent_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - struct msg_hdr_mch *mch; - - mch = (struct msg_hdr_mch *) buff->data; - buff->data = buff->data + sizeof(struct msg_hdr_mch); - - sdp_msg_net_to_cpu_mch(mch); - - return 0; -} - -static int sdp_sent_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - return 0; -} - -static int sdp_sent_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - return 0; -} - -static int sdp_sent_snk_cancel_ack(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - return 0; -} - -static int sdp_sent_resize_buff_ack(struct sdp_opt *conn, - struct sdpc_buff *buff) -{ - struct msg_hdr_crbah *crbah; - - crbah = (struct msg_hdr_crbah *) buff->data; - buff->data = buff->data + sizeof(struct msg_hdr_crbah); - - sdp_msg_net_to_cpu_crbah(crbah); - - return 0; -} - -static int sdp_sent_suspend(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - struct msg_hdr_sch *sch; - - sch = (struct msg_hdr_sch *) buff->data; - buff->data = buff->data + sizeof(struct msg_hdr_sch); - - sdp_msg_net_to_cpu_sch(sch); - - return 0; -} - -static int sdp_sent_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - return 0; -} - -static int sdp_sent_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - struct msg_hdr_snkah *snkah; - - snkah = (struct msg_hdr_snkah *) buff->data; - buff->data = buff->data + sizeof(struct msg_hdr_snkah); - - sdp_msg_net_to_cpu_snkah(snkah); - - return 0; -} - -static int sdp_sent_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - struct msg_hdr_srcah *srcah; - - srcah = (struct msg_hdr_srcah *) buff->data; - buff->data = buff->data + sizeof(struct msg_hdr_srcah); - - sdp_msg_net_to_cpu_srcah(srcah); - - return 0; -} - /* - * sdp_sent_data - SDP data message event received - */ -static int sdp_sent_data(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - int result = 0; - - conn->send_qud -= buff->data_size; - - return result; -} - -/* - * sdp_sent_unsupported - Valid messages we're not sending - */ -static int sdp_sent_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff) -{ - /* - * Since the gateway only initates RDMA's but is never a target, and - * for a few other reasons, there are certain valid SDP messages - * which we never send. - */ - sdp_dbg_warn(conn, "Unexpected SDP message <%02x> sent!", - buff->bsdh_hdr->mid); - - return 0; -} - -/* - * Event Dispatch table. For performance a dispatch table is used to avoid - * a giant case statment for every single SDP event. This is a bit more - * confusing, relies on the layout of the Message IDs, and is less - * flexable. However, it is faster. - * - * Sparse table, the full table would be 16x16, where the low 4 bits, of - * the MID byte, are one dimension, and the high 4 bits are the other - * dimension. Since all rows, except for the first and last, are empty, - * only those are represented in the table. - */ -static sdp_event_cb_func send_event_funcs[SDP_MSG_EVENT_TABLE_SIZE] = { - NULL, /* SDP_MID_HELLO 0x00 */ - NULL, /* SDP_MID_HELLO_ACK 0x01 */ - sdp_sent_disconnect, /* SDP_MID_DISCONNECT 0x02 */ - sdp_sent_abort, /* SDP_MID_ABORT_CONN 0x03 */ - sdp_sent_send_sm, /* SDP_MID_SEND_SM 0x04 */ - sdp_sent_rdma_wr, /* SDP_MID_RDMA_WR_COMP 0x05 */ - sdp_sent_rdma_rd, /* SDP_MID_RDMA_RD_COMP 0x06 */ - sdp_sent_mode_change, /* SDP_MID_MODE_CHANGE 0x07 */ - sdp_sent_src_cancel, /* SDP_MID_SRC_CANCEL 0x08 */ - sdp_sent_snk_cancel, /* SDP_MID_SNK_CANCEL 0x09 */ - sdp_sent_snk_cancel_ack, /* SDP_MID_SNK_CANCEL_ACK 0x0A */ - sdp_sent_unsupported, /* SDP_MID_CH_RECV_BUF 0x0B */ - sdp_sent_resize_buff_ack, /* SDP_MID_CH_RECV_BUF_ACK 0x0C */ - sdp_sent_suspend, /* SDP_MID_SUSPEND 0x0D */ - sdp_sent_suspend_ack, /* SDP_MID_SUSPEND_ACK 0x0E */ - NULL, /* reserved 0x0F */ - NULL, /* reserved 0xF0 */ - NULL, /* reserved 0xF1 */ - NULL, /* reserved 0xF2 */ - NULL, /* reserved 0xF3 */ - NULL, /* reserved 0xF4 */ - NULL, /* reserved 0xF5 */ - NULL, /* reserved 0xF6 */ - NULL, /* reserved 0xF7 */ - NULL, /* reserved 0xF8 */ - NULL, /* reserved 0xF9 */ - NULL, /* reserved 0xFA */ - NULL, /* reserved 0xFB */ - NULL, /* reserved 0xFC */ - sdp_sent_snk_avail, /* SDP_MID_SNK_AVAIL 0xFD */ - sdp_sent_src_avail, /* SDP_MID_SRC_AVAIL 0xFE */ - sdp_sent_data /* SDP_MID_DATA 0xFF */ -}; - -/* * sdp_event_send - send event handler */ int sdp_event_send(struct sdp_opt *conn, struct ib_wc *comp) { - sdp_event_cb_func dispatch_func; - u32 free_count = 0; + struct sdpc_buff *head = NULL; + struct sdpc_buff *buff; u64 current_wrid = 0; - u32 offset; + u32 free_count = 0; int result; - struct sdpc_buff *buff; - struct sdpc_buff *head = NULL; /* * error handling @@ -374,29 +188,47 @@ * data fast path we collapse the next level dispatch function. * For all other buffers we go the slow path. */ - if (SDP_MID_DATA == buff->bsdh_hdr->mid) + result = 0; + + switch (buff->bsdh_hdr->mid) { + case SDP_MID_DATA: conn->send_qud -= buff->data_size; - else { - offset = buff->bsdh_hdr->mid & 0x1F; + break; + case SDP_MID_DISCONNECT: + result = sdp_sent_disconnect(conn, buff); + break; + case SDP_MID_ABORT_CONN: + result = sdp_sent_abort(conn, buff); + break; + case SDP_MID_SEND_SM: + case SDP_MID_RDMA_WR_COMP: + case SDP_MID_RDMA_RD_COMP: + case SDP_MID_MODE_CHANGE: + case SDP_MID_SRC_CANCEL: + case SDP_MID_SNK_CANCEL: + case SDP_MID_SNK_CANCEL_ACK: + case SDP_MID_CH_RECV_BUF_ACK: + case SDP_MID_SNK_AVAIL: + case SDP_MID_SRC_AVAIL: + break; + case SDP_MID_CH_RECV_BUF: + case SDP_MID_SUSPEND: + case SDP_MID_SUSPEND_ACK: + sdp_dbg_warn(conn, + "Unexpected SDP message <%02x> sent!", + buff->bsdh_hdr->mid); + break; + default: + sdp_dbg_warn(conn, "Send complete unknown MID <%d>", + buff->bsdh_hdr->mid); + result = -EINVAL; + break; + } - if (!(offset < SDP_MSG_EVENT_TABLE_SIZE) || - !send_event_funcs[offset]) { - sdp_dbg_warn(conn, - "Send complete unknown MID <%d>", - buff->bsdh_hdr->mid); - result = -EINVAL; - goto drop; - } - - SDP_CONN_STAT_SEND_MID_INC(conn, offset); - - dispatch_func = send_event_funcs[offset]; - result = dispatch_func(conn, buff); - if (result < 0) { - sdp_dbg_warn(conn, "Sent dispatch error. <%d>", - result); - goto drop; - } + if (result) { + sdp_dbg_warn(conn, "Sent dispatch error. <%d>", + result); + goto drop; } current_wrid = buff->wrid; Index: sdp_recv.c =================================================================== --- sdp_recv.c (revision 2628) +++ sdp_recv.c (working copy) @@ -74,8 +74,6 @@ * actually posting the thing. Completion event can happen before * post function returns. */ - sdp_buff_q_put_tail(&conn->recv_post, buff); - sdp_dbg_data(conn, "POST RECV BUFF wrid <%llu> of <%u> bytes.", (unsigned long long) buff->wrid, (unsigned)(buff->tail - buff->data)); @@ -96,10 +94,11 @@ if (result) { sdp_dbg_warn(conn, "Error <%d> posting receive buffer", result); - (void)sdp_buff_q_get_tail(&conn->recv_post); goto drop; } + sdp_buff_q_put_tail(&conn->recv_post, buff); + return 0; drop: sdp_buff_pool_put(buff); @@ -122,26 +121,21 @@ /* * check queue depth */ - if (!(conn->send_cq_size > conn->s_wq_size)) { - result = ENODEV; - goto done; - } + if (!(conn->send_cq_size > conn->s_wq_size)) + return ENODEV; /* * get a reference to the first SrcAvail advertisment. */ advt = sdp_advt_q_look(&conn->src_pend); - if (!advt) { - result = ENODEV; - goto done; - } + if (!advt) + return ENODEV; /* * get a buffer */ buff = sdp_buff_pool_get(); if (!buff) { sdp_dbg_warn(conn, "failed to allocate buff for rdma read."); - result = -ENOMEM; - goto error; + return -ENOMEM; } /* * The data pointer is backed up based on what the stream interface @@ -171,21 +165,11 @@ if (advt->size <= 0) sdp_advt_q_put(&conn->src_actv, sdp_advt_q_get(&conn->src_pend)); - /* - * save the buffer for the event handler. Make sure it's before - * actually posting the thing. Completion event can happen before - * post function returns. - */ - sdp_desc_q_put_tail(&conn->r_src, (struct sdpc_desc *) buff); sdp_dbg_data(conn, "POST READ BUFF wrid <%llu> of <%u> bytes.", (unsigned long long) buff->wrid, (unsigned)(buff->tail - buff->data)); /* - * update send queue depth - */ - conn->s_wq_size++; - /* * post rdma */ buff->sge.addr = virt_to_phys(buff->data); @@ -199,18 +183,18 @@ result = ib_post_send(conn->qp, &send_param, &bad_wr); if (result) { sdp_dbg_warn(conn, "Error <%d> posting rdma read", result); - - (void)sdp_desc_q_get_tail(&conn->r_src); - conn->s_wq_size--; - goto drop; } + /* + * Save buffer and update send queue depth + */ + sdp_desc_q_put_tail(&conn->r_src, (struct sdpc_desc *) buff); + conn->s_wq_size++; + return 0; drop: sdp_buff_pool_put(buff); -error: -done: return result; } @@ -230,18 +214,14 @@ /* * check queue depth */ - if (!(conn->send_cq_size > conn->s_wq_size)) { - result = ENODEV; - goto done; - } + if (!(conn->send_cq_size > conn->s_wq_size)) + return ENODEV; /* * get a reference to the first SrcAvail advertisment. */ advt = sdp_advt_q_look(&conn->src_pend); - if (!advt) { - result = ENODEV; - goto done; - } + if (!advt) + return ENODEV; /* * get a reference to the first IOCB pending. * @@ -249,10 +229,8 @@ * (final complete RDMA will clear it out.) */ iocb = sdp_iocb_q_look(&conn->r_pend); - if (!iocb) { - result = ENODEV; - goto done; - } + if (!iocb) + return ENODEV; /* * register IOCBs physical memory. */ @@ -307,10 +285,6 @@ (unsigned long long) iocb->wrid, zcopy, iocb->len, advt->size); /* - * update send queue depth - */ - conn->s_wq_size++; - /* * post RDMA */ send_param.next = NULL; @@ -321,13 +295,15 @@ result = ib_post_send(conn->qp, &send_param, &bad_wr); if (result) { sdp_dbg_warn(conn, "Error <%d> posting rdma read", result); - conn->s_wq_size--; goto error; } + /* + * update send queue depth + */ + conn->s_wq_size++; return 0; error: -done: return result; } @@ -342,32 +318,24 @@ /* * check if sink cancel is pending */ - if (conn->flags & SDP_CONN_F_SNK_CANCEL) { - result = ENODEV; - goto error; - } + if (conn->flags & SDP_CONN_F_SNK_CANCEL) + return ENODEV; /* * get the pending iocb */ iocb = sdp_iocb_q_look(&conn->r_pend); - if (!iocb) { - result = ENODEV; - goto error; - } + if (!iocb) + return ENODEV; /* * check zcopy threshold */ - if (conn->snk_zthresh > iocb->len) { - result = ENODEV; - goto error; - } + if (conn->snk_zthresh > iocb->len) + return ENODEV; /* * check number of outstanding sink advertisments */ - if (!(conn->r_max_adv > conn->snk_sent)) { - result = ENODEV; - goto error; - } + if (!(conn->r_max_adv > conn->snk_sent)) + return ENODEV; /* * registration */ @@ -381,15 +349,6 @@ goto error; } /* - * IOCB - */ - iocb->flags |= SDP_IOCB_F_ACTIVE; - iocb->flags |= SDP_IOCB_F_RDMA_W; - /* - * queue IOCB - */ - sdp_iocb_q_put_tail(&conn->r_snk, sdp_iocb_q_get_head(&conn->r_pend)); - /* * Either post a send, or buffer the packet in the tx queue */ result = sdp_send_ctrl_snk_avail(conn, @@ -399,16 +358,16 @@ if (result < 0) { sdp_dbg_warn(conn, "Error <%d> sending SnkAvail message", result); - - sdp_iocb_q_put_head(&conn->r_pend, - sdp_iocb_q_get_tail(&conn->r_snk)); - - iocb->flags &= ~SDP_IOCB_F_ACTIVE; - iocb->flags &= ~SDP_IOCB_F_RDMA_W; - goto error; } + /* + * Update and queue IOCB + */ + iocb->flags |= SDP_IOCB_F_ACTIVE; + iocb->flags |= SDP_IOCB_F_RDMA_W; + sdp_iocb_q_put_tail(&conn->r_snk, sdp_iocb_q_get_head(&conn->r_pend)); + conn->snk_sent++; return 0; @@ -1011,7 +970,6 @@ u8 update; if (buff->flags & SDP_BUFF_F_OOB_PRES) { - SDP_EXPECT((buff->tail > buff->data)); update = *value; *value = *(u8 *) (buff->tail - 1); @@ -1081,13 +1039,9 @@ buff = sdp_buff_q_fetch(&conn->recv_pool, sdp_inet_recv_urg_test, (void *)0); - if (buff) { + if (buff) sdp_buff_pool_put(buff); - result = sdp_recv_flush(conn); - SDP_EXPECT(result >= 0); - } - result = 1; } } else { @@ -1157,6 +1111,9 @@ result = sdp_inet_recv_urg(sk, msg, size, flags); copied = (result > 0) ? result : 0; result = (result > 0) ? 0 : result; + + if (copied) + ack = copied; goto done; } /* From rjwalsh at pathscale.com Fri Jun 17 14:21:19 2005 From: rjwalsh at pathscale.com (Robert Walsh) Date: Fri, 17 Jun 2005 14:21:19 -0700 Subject: [openib-general] OpenIB repository down? Message-ID: <1119043279.18567.5.camel@hematite.internal.keyresearch.com> Hi all, Is the OpenIB subversion repository dead at the moment? It's just sitting there for me when I try to do an update or a checkout. Regards, Robert. -- Robert Walsh Email: rjwalsh at pathscale.com PathScale, Inc. Phone: +1 650 934 8117 2071 Stierlin Court, Suite 200 Fax: +1 650 428 1969 Mountain View, CA 94043 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 481 bytes Desc: This is a digitally signed message part URL: From tomduffy at gmail.com Fri Jun 17 14:26:49 2005 From: tomduffy at gmail.com (Tom Duffy) Date: Fri, 17 Jun 2005 14:26:49 -0700 Subject: [openib-general] 2.6.12 is out, please apply my patch to SDP Message-ID: <9d3b7de70506171426396b4a1e@mail.gmail.com> Libor, The git change for 2.6.12 just went across the wire. Can you apply my SDP patch v6 to the tree? Thanks, -tduffy tree e4660ac807d16a7bd3af6db2dfce539acd94ba23 parent dfd11c2e61d3c0c417d341866ed53c44933b65b4 author Linus Torvalds Sat, 18 Jun 2005 02:48:29 -0700 committer Linus Torvalds Sat, 18 Jun 2005 02:48:29 -0700 Linux 2.6.12 Makefile | 2 +- 1 files changed, 1 insertion(+), 1 deletion(-) Index: Makefile =================================================================== --- 87fe96ee6729db58f10b024d14b16f60d46e6c9f/Makefile (mode:100644 sha1:9e005e18c71c4b7adc83a200e1ae10782c751920) +++ e4660ac807d16a7bd3af6db2dfce539acd94ba23/Makefile (mode:100644 sha1:0d1e74d50067dbce4bd3dc9bcbd7954460b5da61) @@ -1,7 +1,7 @@ VERSION = 2 PATCHLEVEL = 6 SUBLEVEL = 12 -EXTRAVERSION =-rc6 +EXTRAVERSION = NAME=Woozy Numbat # *DOCUMENTATION* - To unsubscribe from this list: send the line "unsubscribe bk-commits-head" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From roland at topspin.com Fri Jun 17 14:32:50 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 17 Jun 2005 14:32:50 -0700 Subject: [openib-general] OpenIB repository down? In-Reply-To: <1119043279.18567.5.camel@hematite.internal.keyresearch.com> (Robert Walsh's message of "Fri, 17 Jun 2005 14:21:19 -0700") References: <1119043279.18567.5.camel@hematite.internal.keyresearch.com> Message-ID: <52is0c654t.fsf@topspin.com> Robert> Hi all, Is the OpenIB subversion repository dead at the Robert> moment? It's just sitting there for me when I try to do Robert> an update or a checkout. Yup, it looks down from here as well. - R. From roland at topspin.com Fri Jun 17 16:14:00 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 17 Jun 2005 16:14:00 -0700 Subject: [openib-general] 2.6.12 is out, please apply my patch to SDP In-Reply-To: <9d3b7de70506171426396b4a1e@mail.gmail.com> (Tom Duffy's message of "Fri, 17 Jun 2005 14:26:49 -0700") References: <9d3b7de70506171426396b4a1e@mail.gmail.com> Message-ID: <52vf4c4lvr.fsf@topspin.com> Tom> The git change for 2.6.12 just went across the wire. Where did you see this change? www.kernel.org is still showing 2.6.11.12 as the latest, and the git tree on www.kernel.org doesn't have a 2.6.12 change yet. For example http://www.kernel.org/git/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git is showing Ingo Molnar's '[PATCH] timer exit cleanup' as the last patch committed. - R. From tomduffy at gmail.com Fri Jun 17 16:20:21 2005 From: tomduffy at gmail.com (Tom Duffy) Date: Fri, 17 Jun 2005 16:20:21 -0700 Subject: [openib-general] 2.6.12 is out, please apply my patch to SDP In-Reply-To: <52vf4c4lvr.fsf@topspin.com> References: <9d3b7de70506171426396b4a1e@mail.gmail.com> <52vf4c4lvr.fsf@topspin.com> Message-ID: <9d3b7de70506171620627252af@mail.gmail.com> On 6/17/05, Roland Dreier wrote: > Tom> The git change for 2.6.12 just went across the wire. > > Where did you see this change? www.kernel.org is still showing > 2.6.11.12 as the latest, and the git tree on www.kernel.org doesn't > have a 2.6.12 change yet. For example > > http://www.kernel.org/git/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git > > is showing Ingo Molnar's '[PATCH] timer exit cleanup' as the last > patch committed. Oh, it came across the bk-commits-head at vger list which tracks the git commits now. Sorry if I jumped the gun. -tduffy From roland at topspin.com Fri Jun 17 16:25:19 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 17 Jun 2005 16:25:19 -0700 Subject: [openib-general] 2.6.12 is out, please apply my patch to SDP In-Reply-To: <9d3b7de70506171620627252af@mail.gmail.com> (Tom Duffy's message of "Fri, 17 Jun 2005 16:20:21 -0700") References: <9d3b7de70506171426396b4a1e@mail.gmail.com> <52vf4c4lvr.fsf@topspin.com> <9d3b7de70506171620627252af@mail.gmail.com> Message-ID: <52psuk4lcw.fsf@topspin.com> Tom> Oh, it came across the bk-commits-head at vger list which tracks Tom> the git commits now. Got it... I guess it takes a while for everything to percolate through all the mirror scripts. The svn repo is down right now, and Libor is probably six sheets to the wind by 4pm on a Friday anyway so it's kind of a moot point... - R. From libor at topspin.com Fri Jun 17 17:03:07 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 17 Jun 2005 17:03:07 -0700 Subject: [openib-general] 2.6.12 is out, please apply my patch to SDP In-Reply-To: <52psuk4lcw.fsf@topspin.com>; from roland@topspin.com on Fri, Jun 17, 2005 at 04:25:19PM -0700 References: <9d3b7de70506171426396b4a1e@mail.gmail.com> <52vf4c4lvr.fsf@topspin.com> <9d3b7de70506171620627252af@mail.gmail.com> <52psuk4lcw.fsf@topspin.com> Message-ID: <20050617170307.B9204@topspin.com> On Fri, Jun 17, 2005 at 04:25:19PM -0700, Roland Dreier wrote: > Tom> Oh, it came across the bk-commits-head at vger list which tracks > Tom> the git commits now. > > Got it... I guess it takes a while for everything to percolate through > all the mirror scripts. > > The svn repo is down right now, and Libor is probably six sheets to > the wind by 4pm on a Friday anyway so it's kind of a moot point... I wish. :) However, I haven't been able to get to kernel.org yet, so it will have to wait, unless you've already downloaded the kernel and I can grap it from your home directory. -Libor From roland at topspin.com Fri Jun 17 17:09:34 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 17 Jun 2005 17:09:34 -0700 Subject: [openib-general] 2.6.12 is out, please apply my patch to SDP In-Reply-To: <20050617170307.B9204@topspin.com> (Libor Michalek's message of "Fri, 17 Jun 2005 17:03:07 -0700") References: <9d3b7de70506171426396b4a1e@mail.gmail.com> <52vf4c4lvr.fsf@topspin.com> <9d3b7de70506171620627252af@mail.gmail.com> <52psuk4lcw.fsf@topspin.com> <20050617170307.B9204@topspin.com> Message-ID: <52is0c4jb5.fsf@topspin.com> Libor> However, I haven't been able to get to kernel.org yet, so Libor> it will have to wait, unless you've already downloaded the Libor> kernel and I can grap it from your home directory. Neither the 2.6.12 kernel tarball nor the latest git tree seem to have been mirrored to the public kernel.org yet, so I don't have 2.6.12 yet. - R. From tomduffy at gmail.com Fri Jun 17 19:23:28 2005 From: tomduffy at gmail.com (Tom Duffy) Date: Fri, 17 Jun 2005 19:23:28 -0700 Subject: [openib-general] 2.6.12 is out, please apply my patch to SDP In-Reply-To: <52is0c4jb5.fsf@topspin.com> References: <9d3b7de70506171426396b4a1e@mail.gmail.com> <52vf4c4lvr.fsf@topspin.com> <9d3b7de70506171620627252af@mail.gmail.com> <52psuk4lcw.fsf@topspin.com> <20050617170307.B9204@topspin.com> <52is0c4jb5.fsf@topspin.com> Message-ID: <9d3b7de7050617192331d246ab@mail.gmail.com> On 6/17/05, Roland Dreier wrote: > Libor> However, I haven't been able to get to kernel.org yet, so > Libor> it will have to wait, unless you've already downloaded the > Libor> kernel and I can grap it from your home directory. > > Neither the 2.6.12 kernel tarball nor the latest git tree seem to have > been mirrored to the public kernel.org yet, so I don't have 2.6.12 yet. BTW, it seems to be up on kernel.org now. -tduffy From kahori-simeimail at fj9.so-net.ne.jp Fri Jun 17 12:25:25 2005 From: kahori-simeimail at fj9.so-net.ne.jp (ayumi) Date: Sat, 18 Jun 2005 04:25:25 +0900 Subject: [openib-general] =?iso-2022-jp?b?UmU6GyRCJE8kOCRhJEYhKRsoQg==?= Message-ID: <20050617.1925250187@kahori-simeimail-fj9.so-net.ne.jp> マドンナさんより貴方様を指名するメールが来ましたのでご連絡致しました。 『私立高校の教師しています。昔は良く生徒に告白されたけど、 今ではおばさん扱い。結婚もしていない私を解放して欲しいの。 物凄いストレス溜まってます。』 簡単なプロフィールを載せます。 39歳、年収480万円以上。T162・B87、DカップW61、H89 貴方に紹介する事をマドンナさんに通知してあります。 http://chao.hosting-geomax.jp/index.php/tak006/ 貴方の登録(無料)確認後はマドンナさんに 貴方が登録をした事を通知するので、登録(無料)後は マドンナさんから直接返事が来ます。 その為サイトを通す必要はありません。 無料登録をしてお待ちください。 http://chao.hosting-geomax.jp/index.php/tak006/ ※yahooなどの一般アドレスからでも登録可能です。 From mst at mellanox.co.il Sun Jun 19 00:33:36 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 19 Jun 2005 10:33:36 +0300 Subject: [openib-general] [PATCH (repost)] error values cleanup In-Reply-To: <20050615094747.GX21081@mellanox.co.il> References: <20050615094747.GX21081@mellanox.co.il> Message-ID: <20050619073336.GN20041@mellanox.co.il> Here's an update to the latest bits: Libor, does this make sense? Make sdp_cm_actv_error and sdp_conn_inet_error accept standard, negative error values (esp. for sdp_cm_actv_error it was sufficiently confusing to be passing a positive value to warrant a comment). There are still a couple of functions which return positive error values but these are static and these positive errors never propagate outside a single .c file. As a side effect, "(0 - result)" in a couple of places gets replaced with "result" which is also good. The patch does a couple of cosmetic cleanups as well. Signed-off-by: Michael S. Tsirkin Index: ulp/sdp/sdp_inet.c =================================================================== --- ulp/sdp/sdp_inet.c (revision 2656) +++ ulp/sdp/sdp_inet.c (working copy) @@ -278,7 +278,7 @@ error: /* * abortive close. */ - sdp_conn_inet_error(conn, ECONNRESET); + sdp_conn_inet_error(conn, -ECONNRESET); (void)ib_send_cm_dreq(conn->cm_id, NULL, 0); return result; Index: ulp/sdp/sdp_conn.c =================================================================== --- ulp/sdp/sdp_conn.c (revision 2656) +++ ulp/sdp/sdp_conn.c (working copy) @@ -84,21 +84,21 @@ void sdp_conn_inet_error(struct sdp_opt (void)sdp_inet_accept_q_remove(conn); SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); - SDP_CONN_SET_ERR(conn, error); + SDP_CONN_SET_ERR(conn, -error); conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; if (conn->sk->sk_socket) conn->sk->sk_socket->state = SS_UNCONNECTED; - sdp_iocb_q_cancel_all(conn, -error); + sdp_iocb_q_cancel_all(conn, error); sdp_inet_wake_error(conn->sk); } void sdp_conn_abort(struct sdp_opt *conn) { int result; - int error = ECONNRESET; + int error = -ECONNRESET; sdp_dbg_ctrl(conn, "Abort send. src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -133,7 +133,7 @@ void sdp_conn_abort(struct sdp_opt *conn conn->flags &= ~SDP_CONN_F_DIS_PEND; case SDP_CONN_ST_DIS_RECV_1: - error = EPIPE; + error = -EPIPE; case SDP_CONN_ST_ESTABLISHED: /* * abortive close. @@ -150,7 +150,7 @@ void sdp_conn_abort(struct sdp_opt *conn * outstanding CM request. Mark it in error, and CM * completion needs to complete the closing. */ - error = ECONNREFUSED; + error = -ECONNREFUSED; break; case SDP_CONN_ST_ERROR: case SDP_CONN_ST_CLOSED: Index: ulp/sdp/sdp_actv.c =================================================================== --- ulp/sdp/sdp_actv.c (revision 2656) +++ ulp/sdp/sdp_actv.c (working copy) @@ -41,8 +41,6 @@ void sdp_cm_actv_error(struct sdp_opt *c { int result; /* - * error value is positive error. - * * Handle errors within active connections stream. * First generate appropriate response, REJ, DREQ or nothing. * Second the socket must be notified of the error. @@ -88,14 +86,14 @@ void sdp_cm_actv_error(struct sdp_opt *c break; } - SDP_CONN_SET_ERR(conn, error); + SDP_CONN_SET_ERR(conn, -error); conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; if (conn->sk->sk_socket) conn->sk->sk_socket->state = SS_UNCONNECTED; - sdp_iocb_q_cancel_all(conn, (0 - error)); + sdp_iocb_q_cancel_all(conn, error); sdp_inet_wake_error(conn->sk); } @@ -319,7 +317,7 @@ int sdp_cm_rep_handler(struct ib_cm_id * return 0; error: - sdp_cm_actv_error(conn, (0 - result)); + sdp_cm_actv_error(conn, result); if (conn->state == SDP_CONN_ST_CLOSED) { conn->cm_id = NULL; @@ -501,7 +499,7 @@ static void sdp_cm_path_complete(u64 id, goto done; failed: - sdp_cm_actv_error(conn, (0 - status)); + sdp_cm_actv_error(conn, status); done: sdp_conn_unlock(conn); sdp_conn_put(conn); /* address resolution reference */ Index: ulp/sdp/sdp_pass.c =================================================================== --- ulp/sdp/sdp_pass.c (revision 2656) +++ ulp/sdp/sdp_pass.c (working copy) @@ -41,7 +41,7 @@ int sdp_cm_pass_establish(struct sdp_opt { struct ib_qp_attr *qp_attr; int attr_mask = 0; - int result = 0; + int result; sdp_dbg_ctrl(conn, "Passive Establish src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -62,27 +62,26 @@ int sdp_cm_pass_establish(struct sdp_opt result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask); if (result) { - sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS", - result); + sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS", result); goto error; } result = ib_modify_qp(conn->qp, qp_attr, attr_mask); if (result) { - sdp_dbg_warn(conn, "Error <%d> modifiying QP to RTS", result); + sdp_dbg_warn(conn, "Error <%d> modifying QP to RTS", result); goto error; } conn->send_buf = SDP_INET_SEND_SIZE; result = sdp_send_flush(conn); - if (0 > result) { + if (result < 0) { sdp_dbg_warn(conn, "Error <%d> flushing sends.", result); goto error; } result = sdp_recv_flush(conn); - if (0 > result) { + if (result < 0) { sdp_dbg_warn(conn, "Error <%d> flushing receives.", result); goto error; } @@ -94,7 +93,7 @@ int sdp_cm_pass_establish(struct sdp_opt error: kfree(qp_attr); done: - sdp_conn_inet_error(conn, -result); + sdp_conn_inet_error(conn, result); return result; } /* Index: ulp/sdp/sdp_event.c =================================================================== --- ulp/sdp/sdp_event.c (revision 2656) +++ ulp/sdp/sdp_event.c (working copy) @@ -184,7 +184,7 @@ void sdp_cq_event_handler(struct ib_cq * static void sdp_cm_to_error(struct sdp_opt *conn) { - sdp_conn_inet_error(conn, ECONNRESET); + sdp_conn_inet_error(conn, -ECONNRESET); conn->cm_id = NULL; sdp_conn_put(conn); /* CM reference */ } @@ -202,11 +202,11 @@ static int sdp_cm_idle(struct ib_cm_id * */ switch (conn->state) { case SDP_CONN_ST_REQ_SENT: - sdp_cm_actv_error(conn, ECONNREFUSED); + sdp_cm_actv_error(conn, -ECONNREFUSED); break; case SDP_CONN_ST_REQ_RECV: case SDP_CONN_ST_ESTABLISHED: - sdp_conn_inet_error(conn, ECONNREFUSED); + sdp_conn_inet_error(conn, -ECONNREFUSED); break; case SDP_CONN_ST_TIME_WAIT_1: sdp_dbg_warn(conn, "Unexpected connection state"); -- MST From glebn at voltaire.com Sun Jun 19 03:16:50 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Sun, 19 Jun 2005 13:16:50 +0300 Subject: [openib-general] cq_fd array in ibv_context Message-ID: <20050619101650.GD8601@minantech.com> Hello, Can somebody enlighten me what is the idea behind cq_fd in ibv_context been array. It seams that the size of this array is hardcoded to be 1 in ib_uverbs_add_one() (uverbs_dev->num_comp = 1). Am I missing something? Thanks, -- Gleb. From mst at mellanox.co.il Sun Jun 19 07:37:53 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 19 Jun 2005 17:37:53 +0300 Subject: [openib-general] [PATCH] mthca: add copyright statements Message-ID: <20050619143753.GU20041@mellanox.co.il> Roland, I went over the mthca logs and added a copyright statements to files I have changed. Please apply. Thanks, MST Signed-off-by: Michael S. Tsirkin Index: mthca_dev.h =================================================================== --- mthca_dev.h (revision 2656) +++ mthca_dev.h (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_main.c =================================================================== --- mthca_main.c (revision 2656) +++ mthca_main.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_memfree.h =================================================================== --- mthca_memfree.h (revision 2656) +++ mthca_memfree.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_provider.c =================================================================== --- mthca_provider.c (revision 2656) +++ mthca_provider.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_provider.h =================================================================== --- mthca_provider.h (revision 2656) +++ mthca_provider.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_profile.c =================================================================== --- mthca_profile.c (revision 2656) +++ mthca_profile.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_mad.c =================================================================== --- mthca_mad.c (revision 2656) +++ mthca_mad.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_cmd.c =================================================================== --- mthca_cmd.c (revision 2656) +++ mthca_cmd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_pd.c =================================================================== --- mthca_pd.c (revision 2656) +++ mthca_pd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_cq.c =================================================================== --- mthca_cq.c (revision 2656) +++ mthca_cq.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_profile.h =================================================================== --- mthca_profile.h (revision 2656) +++ mthca_profile.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_eq.c =================================================================== --- mthca_eq.c (revision 2656) +++ mthca_eq.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_config_reg.h =================================================================== --- mthca_config_reg.h (revision 2656) +++ mthca_config_reg.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_doorbell.h =================================================================== --- mthca_doorbell.h (revision 2656) +++ mthca_doorbell.h (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_cmd.h =================================================================== --- mthca_cmd.h (revision 2656) +++ mthca_cmd.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_mr.c =================================================================== --- mthca_mr.c (revision 2656) +++ mthca_mr.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_qp.c =================================================================== --- mthca_qp.c (revision 2656) +++ mthca_qp.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: mthca_memfree.c =================================================================== --- mthca_memfree.c (revision 2656) +++ mthca_memfree.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU -- MST From mst at mellanox.co.il Sun Jun 19 07:41:08 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 19 Jun 2005 17:41:08 +0300 Subject: [openib-general] [PATCH] ipoib: add copyright statements Message-ID: <20050619144108.GV20041@mellanox.co.il> Roland, I went over the ipoib logs and added copyright statements to files I have changed. Please apply. Thanks, MST Add copyright statements to files modified by Michael S. Tsirkin. Signed-off-by: Michael S. Tsirkin Index: ipoib_verbs.c =================================================================== --- ipoib_verbs.c (revision 2656) +++ ipoib_verbs.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: ipoib_ib.c =================================================================== --- ipoib_ib.c (revision 2656) +++ ipoib_ib.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU -- MST From mst at mellanox.co.il Sun Jun 19 07:43:29 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 19 Jun 2005 17:43:29 +0300 Subject: [openib-general] [PATCH] sdp: add copyright statements Message-ID: <20050619144328.GW20041@mellanox.co.il> Libor, I went over the sdp logs and added copyright statements to files I have changed. Please apply. Thanks, MST Add copyright statements to files modified by Michael S. Tsirkin. Signed-off-by: Michael S. Tsirkin Index: sdp_queue.h =================================================================== --- sdp_queue.h (revision 2656) +++ sdp_queue.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_write.c =================================================================== --- sdp_write.c (revision 2656) +++ sdp_write.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_link.c =================================================================== --- sdp_link.c (revision 2656) +++ sdp_link.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_rcvd.c =================================================================== --- sdp_rcvd.c (revision 2656) +++ sdp_rcvd.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_inet.c =================================================================== --- sdp_inet.c (revision 2656) +++ sdp_inet.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_link.h =================================================================== --- sdp_link.h (revision 2656) +++ sdp_link.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_proto.h =================================================================== --- sdp_proto.h (revision 2656) +++ sdp_proto.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_read.c =================================================================== --- sdp_read.c (revision 2656) +++ sdp_read.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_msgs.h =================================================================== --- sdp_msgs.h (revision 2656) +++ sdp_msgs.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_send.c =================================================================== --- sdp_send.c (revision 2656) +++ sdp_send.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_conn.c =================================================================== --- sdp_conn.c (revision 2656) +++ sdp_conn.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_actv.c =================================================================== --- sdp_actv.c (revision 2656) +++ sdp_actv.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_advt.c =================================================================== --- sdp_advt.c (revision 2656) +++ sdp_advt.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_recv.c =================================================================== --- sdp_recv.c (revision 2656) +++ sdp_recv.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_conn.h =================================================================== --- sdp_conn.h (revision 2656) +++ sdp_conn.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_proc.c =================================================================== --- sdp_proc.c (revision 2656) +++ sdp_proc.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_advt.h =================================================================== --- sdp_advt.h (revision 2656) +++ sdp_advt.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_pass.c =================================================================== --- sdp_pass.c (revision 2656) +++ sdp_pass.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_proc.h =================================================================== --- sdp_proc.h (revision 2656) +++ sdp_proc.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_sent.c =================================================================== --- sdp_sent.c (revision 2656) +++ sdp_sent.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_iocb.c =================================================================== --- sdp_iocb.c (revision 2656) +++ sdp_iocb.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_event.c =================================================================== --- sdp_event.c (revision 2656) +++ sdp_event.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_iocb.h =================================================================== --- sdp_iocb.h (revision 2656) +++ sdp_iocb.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_buff.c =================================================================== --- sdp_buff.c (revision 2656) +++ sdp_buff.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_dev.h =================================================================== --- sdp_dev.h (revision 2656) +++ sdp_dev.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_queue.c =================================================================== --- sdp_queue.c (revision 2656) +++ sdp_queue.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: sdp_buff.h =================================================================== --- sdp_buff.h (revision 2656) +++ sdp_buff.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU -- MST From mst at mellanox.co.il Sun Jun 19 07:46:20 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 19 Jun 2005 17:46:20 +0300 Subject: [openib-general] [PATCH] core: add copyright statements Message-ID: <20050619144619.GX20041@mellanox.co.il> Sean, I went over the core logs and added copyright statements to files I have changed. Please apply. Thanks, MST Add copyright statements to files modified by Michael S. Tsirkin. Signed-off-by: Michael S. Tsirkin Index: core/mad.c =================================================================== --- core/mad.c (revision 2656) +++ core/mad.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Voltaire, Inc. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/sysfs.c =================================================================== --- core/sysfs.c (revision 2656) +++ core/sysfs.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU -- MST From roland at topspin.com Sun Jun 19 12:13:41 2005 From: roland at topspin.com (Roland Dreier) Date: Sun, 19 Jun 2005 12:13:41 -0700 Subject: [openib-general] Re: [PATCH] mthca: add copyright statements In-Reply-To: <20050619143753.GU20041@mellanox.co.il> (Michael S. Tsirkin's message of "Sun, 19 Jun 2005 17:37:53 +0300") References: <20050619143753.GU20041@mellanox.co.il> Message-ID: <523bre2m8q.fsf@topspin.com> > + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). I believe you want to include the statement "All rights reserved." Also, who is the copyright holder -- MST as an individual or Mellanox Technologies? As it stands I'm not sure what this line is actually saying -- what does it mean to have your name in parentheses? Thanks, Roland From itamar at mellanox.co.il Mon Jun 20 02:01:32 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Mon, 20 Jun 2005 12:01:32 +0300 Subject: [openib-general] Re: [PATCH][kdapl]update dat_rmr_bind API & delete dapl_hash Message-ID: <91DB792C7985D411BEC300B40080D29CC35CCC@mtvex01.mtv.mtl.com> O.K i am resending this patch and spliting it to 5 small patches 1. Update dat_rmr_bind API (added lmr_handle input param) 2. Remove dapl_hash (there is no need for the hash) 3. Small changes in dapl_hca_alloc/dapl_hca_free functions 4. Integrate dapl_hca_link_ia/dapl_hca_unlink_ia into dapl_ia.c (no need for functios that just call LIST_ADD and LIST_DEL) 5. Integrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c (no need for 2 files just for 2 simple function that kmalloc and kfree. There is not any special logic in this functions that need to separate them into different files) Itamar > -----Original Message----- > From: Christoph Hellwig [mailto:hch at lst.de] > Sent: Friday, June 17, 2005 7:41 PM > To: Itamar & Shira Rabenstein > Cc: openib-general at openib.org > Subject: Re: [openib-general] Re: [PATCH][kdapl]update > dat_rmr_bind API > & delete dapl_hash > > > On Thu, Jun 16, 2005 at 09:20:27PM +0200, Itamar & Shira > Rabenstein wrote: > > >Itamar, > > > > > >Can you seperate the RMR api change and hash table removal into a > > >seperate patch? > > > > I dont see why, the new input param to dat_rmr_bind > (lmr_handle) come > > in order that we will be able to remove the hash table so > this is one > > patch. > > In the Linux world we generally prefer faine-grained patches. While > the RMR api change is a prerequsite to remove the hash code it's not > directly related. > > > >Merging the HCA and IA code into the dapl_ia.[ch] files > doesn't feel > > >right to me. I think of the HCA as a DAT object just like > IAs, EPs, > > >LMRs, etc, and hence it should have its own file. > > > > > >james > > > > > > There are 4 function in this files: > > > > 2 were merged to provider code: > > dapl_hca_alloc(13 lines) :which i think is part of > the provider code . > > dapl_hca_free (1 line) == kfree , no need a function for this. > > 2 were merged to ia code: > > dapl_hca_link_ia (3 lines ) : no need a function > to call list add > > dapl_hca_unlink_ia(3 lines) : no need a function > to call list_del > > > > I belive that we need to make the code more readable and to > have this functions > > is not a good idea and to add 2 files for this is for sure > a bad idea ... > > I agree your reasoning for the second set of functions at least. But > again this is not related to the API change at all, please separate it > out to a patch of it's own, applying ontop of the API change and hash > removal patches. > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From itamar at mellanox.co.il Mon Jun 20 01:42:55 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Mon, 20 Jun 2005 11:42:55 +0300 Subject: [openib-general] [PATCH][kdapl] Update dat_rmr_bind API Message-ID: <20050620084255.GA7060@mellanox.co.il> Update dat_rmr_bind function API (added lmr_handle as input param) Signed-off-by: Itamar Rabenstein Index: test/dapltest/test/dapl_bpool.c =================================================================== --- test/dapltest/test/dapl_bpool.c (revision 2656) +++ test/dapltest/test/dapl_bpool.c (working copy) @@ -237,6 +237,7 @@ bpool_ptr->reg_addr, bpool_ptr->reg_size)); ret = dat_rmr_bind ( bpool_ptr->rmr_handle, + bpool_ptr->lmr, &iov, mflags, bpool_ptr->ep, Index: test/dapltest/kdapl/kdapl_tdep_user.c =================================================================== --- test/dapltest/kdapl/kdapl_tdep_user.c (revision 2656) +++ test/dapltest/kdapl/kdapl_tdep_user.c (working copy) @@ -76,7 +76,7 @@ } if (params_ptr->test_type == TRANSACTION_TEST) { print_ioctl.cookie = ioctl (fd, KDAPL_IOCTL_GET_STAT_T, &Client_Stats_T); - DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_iterations, + DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_threads, params_ptr->u.Transaction_Cmd.eps_per_thread); } Index: dat-provider/dapl_rmr.c =================================================================== --- dat-provider/dapl_rmr.c (revision 2656) +++ dat-provider/dapl_rmr.c (working copy) @@ -149,26 +149,18 @@ } static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, - const struct dat_lmr_triplet *lmr_triplet, + struct dapl_lmr *lmr, + const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, struct dapl_ep *ep_ptr, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) { - struct dapl_lmr *lmr; struct dapl_cookie *cookie; u32 status; boolean_t is_signaled; - status = dapl_hash_search(rmr->common.owner_ia->hca->lmr_hash_table, - lmr_triplet->lmr_context, - (DAPL_HASH_DATA *) &lmr); - if (DAT_SUCCESS != status) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - /* * if the ep in unconnected return an error. IB requires that the * QP be connected to change a memory window binding since: @@ -353,27 +345,32 @@ * Output: */ u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, - const struct dat_lmr_triplet *lmr_triplet, + struct dat_lmr *lmr_handle, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) { struct dapl_rmr *rmr; + struct dapl_lmr *lmr; struct dapl_ep *ep_ptr; if (!rmr_handle) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); + if (!lmr_handle) + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); + if (!ep) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); rmr = (struct dapl_rmr *)rmr_handle; + lmr = (struct dapl_lmr *)lmr_handle; ep_ptr = (struct dapl_ep *)ep; /* if the rmr should be bound */ if (0 != lmr_triplet->segment_length) - return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, + return dapl_rmr_bind_fuse(rmr, lmr, lmr_triplet, mem_priv, ep_ptr, user_cookie, completion_flags, rmr_context); else /* the rmr should be unbound */ Index: dat-provider/dapl.h =================================================================== --- dat-provider/dapl.h (revision 2656) +++ dat-provider/dapl.h (working copy) @@ -543,6 +543,7 @@ struct dat_rmr_param *rmr_args); extern u32 dapl_rmr_bind(struct dat_rmr *rmr, + struct dat_lmr *lmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, Index: dat/dat.h =================================================================== --- dat/dat.h (revision 2656) +++ dat/dat.h (working copy) @@ -1144,7 +1144,8 @@ typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); -typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, struct dat_lmr *, + const struct dat_lmr_triplet *, enum dat_mem_priv_flags, struct dat_ep *, DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); @@ -1617,23 +1618,24 @@ return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline u32 dat_rmr_query(struct dat_rmr * rmr, struct dat_rmr_param *param) +static inline u32 dat_rmr_query(struct dat_rmr *rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline u32 dat_rmr_bind(struct dat_rmr * rmr, - const struct dat_lmr_triplet *iov, +static inline u32 dat_rmr_bind(struct dat_rmr *rmr, + struct dat_lmr *lmr, + const struct dat_lmr_triplet *iov, enum dat_mem_priv_flags mem_flags, struct dat_ep *ep, DAT_RMR_COOKIE cookie, enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) { - return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, iov, mem_flags, ep, - cookie, comp_flags, context); + return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, lmr, iov, mem_flags, + ep, cookie, comp_flags, context); } -static inline u32 dat_rmr_free(struct dat_rmr * rmr) +static inline u32 dat_rmr_free(struct dat_rmr *rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -- Itamar From itamar at mellanox.co.il Mon Jun 20 01:44:50 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Mon, 20 Jun 2005 11:44:50 +0300 Subject: [openib-general] [PATCH][kdapl] Remove dapl_hash (there is no need for the hash) Message-ID: <20050620084450.GA7070@mellanox.co.il> Remove dapl_hash (there is no need for the hash) Signed-off-by: Itamar Rabenstein diff -Nurp -X dontdiff dat-provider_bind/Makefile dat-provider/Makefile --- dat-provider_bind/Makefile Sun Jun 19 15:17:54 2005 +++ dat-provider/Makefile Sun Jun 19 15:24:03 2005 @@ -20,7 +20,6 @@ PROVIDER_MODULES := \ dapl_cr \ dapl_ep \ dapl_evd \ - dapl_hash \ dapl_hca_util \ dapl_ia \ dapl_lmr \ diff -Nurp -X dontdiff dat-provider_bind/dapl.h dat-provider/dapl.h --- dat-provider_bind/dapl.h Sun Jun 19 15:17:54 2005 +++ dat-provider/dapl.h Sun Jun 19 15:24:06 2005 @@ -84,9 +84,6 @@ typedef enum dapl_evd_completion { typedef void (*ib_async_handler_t) (struct ib_event *, void *); -typedef u64 DAPL_HASH_KEY; -typedef void *DAPL_HASH_DATA; - /********************************************************************* * * * Structures * @@ -125,8 +122,6 @@ struct dapl_hca { u8 port_num; struct ib_device *ib_hca_handle; struct ib_cq *null_cq; /* CQ with 0 entries */ - /* Memory Subsystem Support */ - struct dapl_hash_table *lmr_hash_table; /* Limits & useful HCA attributes */ struct dat_ia_attr ia_attr; struct dat_ep_attr ep_attr; diff -Nurp -X dontdiff dat-provider_bind/dapl_hash.c dat-provider/dapl_hash.c --- dat-provider_bind/dapl_hash.c Sun Jun 19 15:17:54 2005 +++ dat-provider/dapl_hash.c Thu Jan 1 02:00:00 1970 @@ -1,462 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Provides a generic hash table with chaining. - * - * $Id: dapl_hash.c 2640 2005-06-16 16:22:46Z jlentini $ - */ - -#include "dapl_hash.h" - -/********************************************************************* - * * - * Structures * - * * - *********************************************************************/ - -/* - * A hash table element - */ -typedef struct DAPL_HASH_ELEM { - struct DAPL_HASH_ELEM *next_element; - DAPL_HASH_KEY key; - void *datum; -} DAPL_HASH_ELEM; - -/* - * The hash table - */ -struct dapl_hash_table { - unsigned long num_entries; - unsigned long tbl_size; - DAPL_HASH_ELEM *table; - spinlock_t lock; - unsigned long flags; - /* - * statistics - we tally on insert operations, counting - * the number of entries in the whole hash table, as - * well as the length of chains we walk to insert. This - * ignores empty buckets, giving us data on overall table - * occupancy, as well as max/average chain length for - * the buckets used. If our hash function results in - * hot buckets, this will show it. - */ - uint64_t hash_tbl_inserts; /* total inserts ops */ - uint64_t hash_tbl_max; /* max in entire table */ - uint64_t hash_tbl_total; /* total in table */ - uint64_t hash_chn_max; /* longest chain */ - uint64_t hash_chn_total; /* total non-0 lenghts */ -}; - -/********************************************************************* - * * - * Defines * - * * - *********************************************************************/ - -/* datum value in empty table slots (use 0UL or ~0UL as appropriate) */ -#define NO_DATUM_VALUE ((void *) 0UL) -#define NO_DATUM(value) ((value) == NO_DATUM_VALUE) - -/* Lookup macro (which falls back to function to rehash) */ -#define DAPL_HASHLOOKUP( p_table, in_key, out_datum, bucket_head) \ - do { \ - DAPL_HASH_KEY save_key = in_key; \ - DAPL_HASH_ELEM *element = \ - &((p_table)->table)[DAPL_DOHASH(in_key,(p_table)->tbl_size)]; \ - in_key = save_key; \ - if (NO_DATUM(element->datum)) { \ - (bucket_head) = (void *)0; \ - } else if (element->key == (DAPL_HASH_KEY) (in_key)) { \ - (out_datum) = element->datum; \ - (bucket_head) = (void *)element; \ - } else if (element->next_element) { \ - dapl_hash_rehash(element, \ - (in_key), \ - (void **)&(out_datum), \ - (DAPL_HASH_ELEM **)&(bucket_head)); \ - } else { \ - (bucket_head) = (void *)0; \ - }\ - } while (0) - -/********************************************************************* - * * - * Internal Functions * - * * - *********************************************************************/ - -/* - * Rehash the key (used by add and lookup functions) - * - * Inputs: element element to rehash key - * key, datum datum for key head - * head head for list - */ -static void -dapl_hash_rehash(DAPL_HASH_ELEM * element, - DAPL_HASH_KEY key, void **datum, DAPL_HASH_ELEM ** head) -{ - /* - * assume we looked at the contents of element already, - * and start with the next element. - */ - dapl_os_assert(element->next_element); - dapl_os_assert(!NO_DATUM(element->datum)); - - *head = element; - while (1) { - element = element->next_element; - if (!element) { - break; - } - if (element->key == key) { - *datum = element->datum; - return; - } - } - *head = NULL; -} - -/* - * Add a new key to the hash table - * - * Inputs: - * table, key and datum to be added - * allow_dup - TRUE if dups are allowed - * Outputs: - * report_dup - should you care to know - * Returns: - * TRUE on success - */ -static boolean_t -dapl_hash_add(struct dapl_hash_table *table, DAPL_HASH_KEY key, void *datum, - boolean_t allow_dup, boolean_t *report_dup) -{ - void *olddatum; - DAPL_HASH_KEY hashValue, save_key = key; - DAPL_HASH_ELEM *found; - boolean_t status = FALSE; - unsigned int chain_len = 0; - - if (report_dup) { - (*report_dup) = FALSE; - } - - if (NO_DATUM(datum)) { - /* - * Reserved value used for datum - */ - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_hash_add() called with magic NO_DATA " - "value (%p) used as datum!\n", datum); - return FALSE; - } - - DAPL_HASHLOOKUP(table, key, olddatum, found); - if (found) { - /* - * key exists already - */ - if (report_dup) { - *report_dup = TRUE; - } - - if (!allow_dup) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_hash_add() called with duplicate " - "key (" F64x ")\n", key); - return FALSE; - } - } - - hashValue = DAPL_DOHASH(key, table->tbl_size); - key = save_key; - if (NO_DATUM(table->table[hashValue].datum)) { - /* - * Empty head - just fill it in - */ - table->table[hashValue].key = key; - table->table[hashValue].datum = datum; - table->table[hashValue].next_element = NULL; - table->num_entries++; - status = TRUE; - } else { - DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, - GFP_ATOMIC); - /* - * Add an element to the end of the chain - */ - if (newelement) { - DAPL_HASH_ELEM *lastelement; - newelement->key = key; - newelement->datum = datum; - newelement->next_element = NULL; - for (lastelement = &table->table[hashValue]; - lastelement->next_element; - lastelement = lastelement->next_element) { - /* Walk to the end of the chain */ - chain_len++; - } - lastelement->next_element = newelement; - table->num_entries++; - status = TRUE; - } else - status = FALSE; - } - - /* - * Tally up our counters. chain_len is one less than current chain - * length. - */ - chain_len++; - table->hash_tbl_inserts++; - table->hash_tbl_total += table->num_entries; - table->hash_chn_total += chain_len; - if (table->num_entries > table->hash_tbl_max) { - table->hash_tbl_max = table->num_entries; - } - if (chain_len > table->hash_chn_max) { - table->hash_chn_max = chain_len; - } - - return status; -} - -/* - * Remove element from hash bucket - * - * Inputs: - * element, key to be deleted - * Returns: - * TRUE on success - */ -static boolean_t -dapl_hash_delete_element(DAPL_HASH_ELEM * element, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_datum) -{ - DAPL_HASH_ELEM *curelement; - DAPL_HASH_ELEM *lastelement; - - lastelement = NULL; - for (curelement = element; - curelement; - lastelement = curelement, curelement = curelement->next_element) { - if (curelement->key == key) { - if (p_datum) { - *p_datum = curelement->datum; - } - if (lastelement) { - /* - * curelement was malloc'd; free it - */ - lastelement->next_element = - curelement->next_element; - kfree(curelement); - } else { - /* - * curelement is static list head - */ - DAPL_HASH_ELEM *n = curelement->next_element; - if (n) { - /* - * If there is a next element, copy its contents into the - * head and free the original next element. - */ - curelement->key = n->key; - curelement->datum = n->datum; - curelement->next_element = - n->next_element; - kfree(n); - } else { - curelement->datum = NO_DATUM_VALUE; - } - } - break; - } - } - - return (curelement != NULL ? TRUE : FALSE); -} - -/********************************************************************* - * * - * External Functions * - * * - *********************************************************************/ - -/* - * Create a new hash table with at least 'table_size' hash buckets. - */ -u32 dapl_hash_create(int table_size, struct dapl_hash_table **pp_table) -{ - struct dapl_hash_table *p_table; - int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); - u32 dat_status = DAT_SUCCESS; - int i; - - dapl_os_assert(pp_table); - - /* Allocate hash table */ - p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); - if (!p_table) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - /* Init hash table, allocate and init and buckets */ - memset(p_table, 0, sizeof *p_table); - p_table->tbl_size = table_size; - p_table->table = kmalloc(table_length, GFP_ATOMIC); - if (!p_table->table) { - kfree(p_table); - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - spin_lock_init(&p_table->lock); - for (i = 0; i < table_size; i++) { - p_table->table[i].datum = NO_DATUM_VALUE; - p_table->table[i].key = 0; - p_table->table[i].next_element = NULL; - } - - *pp_table = p_table; - -bail: - return DAT_SUCCESS; -} - -/* - * Destroy a hash table - */ -u32 dapl_hash_free(struct dapl_hash_table *p_table) -{ - dapl_os_assert(p_table && p_table->table); - - /* no need to destroy p_table->lock */ - kfree(p_table->table); - kfree(p_table); - - return DAT_SUCCESS; -} - -/* - * Returns the number of elements stored in the table - */ - -u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size) -{ - dapl_os_assert(p_table && p_size); - - *p_size = p_table->num_entries; - - return DAT_SUCCESS; -} - -/* - * Inserts the specified data into the table with the given key. - * Duplicates are not expected, and return in error, having done nothing. - */ - -u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA data) -{ - u32 dat_status = DAT_SUCCESS; - - dapl_os_assert(p_table); - - spin_lock_irqsave(&p_table->lock, p_table->flags); - if (!dapl_hash_add(p_table, key, data, FALSE, NULL)) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - } - spin_unlock_irqrestore(&p_table->lock, p_table->flags); - - return dat_status; -} - -/* - * Searches for the given key. If found, - * DAT_SUCCESS is returned and the associated - * data is returned in the DAPL_HASH_DATA - * pointer if that pointer is not NULL. - */ -u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA *p_data) -{ - u32 dat_status; - void *olddatum; - DAPL_HASH_ELEM *found; - - dapl_os_assert(p_table); - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, 0); - - spin_lock_irqsave(&p_table->lock, p_table->flags); - DAPL_HASHLOOKUP(p_table, key, olddatum, found); - spin_unlock_irqrestore(&p_table->lock, p_table->flags); - - if (found) { - if (p_data) { - *p_data = olddatum; - } - dat_status = DAT_SUCCESS; - } - - return dat_status; -} - -u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA *p_data) -{ - u32 dat_status; - DAPL_HASH_KEY hashValue, save_key = key; - - dapl_os_assert(p_table); - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, 0); - - if (p_table->num_entries == 0) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_hash_remove () called on empty hash table!\n"); - return dat_status; - } - - hashValue = DAPL_DOHASH(key, p_table->tbl_size); - key = save_key; - spin_lock_irqsave(&p_table->lock, p_table->flags); - if (dapl_hash_delete_element(&p_table->table[hashValue], key, p_data)) { - p_table->num_entries--; - dat_status = DAT_SUCCESS; - } - spin_unlock_irqrestore(&p_table->lock, p_table->flags); - - return dat_status; -} diff -Nurp -X dontdiff dat-provider_bind/dapl_hash.h dat-provider/dapl_hash.h --- dat-provider_bind/dapl_hash.h Sun Jun 19 15:17:54 2005 +++ dat-provider/dapl_hash.h Thu Jan 1 02:00:00 1970 @@ -1,81 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_hash.h 2640 2005-06-16 16:22:46Z jlentini $ - */ - -#ifndef DAPL_HASH_H -#define DAPL_HASH_H - -#include "dapl.h" - -/********************************************************************* - * * - * Defines * - * * - *********************************************************************/ - -/* - * Hash table size. - * - * Default is small; use the larger sample values for hash tables - * known to be heavily used. The sample values chosen are the - * largest primes below 2^8, 2^9, and 2^10. - */ -#define DAPL_DEF_HASHSIZE 251 -#define DAPL_MED_HASHSIZE 509 -#define DAPL_LRG_HASHSIZE 1021 - -#define DAPL_HASH_TABLE_DEFAULT_CAPACITY DAPL_DEF_HASHSIZE - -/* The hash function */ -#define DAPL_DOHASH(key,hashsize) dapl_os_mod64(key,hashsize) - -/********************************************************************* - * * - * Function Prototypes * - * * - *********************************************************************/ - -extern u32 dapl_hash_create(int capacity, struct dapl_hash_table **pp_table); - -extern u32 dapl_hash_free(struct dapl_hash_table *p_table); - -extern u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size); - -extern u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA data); - -extern u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA *p_data); - -extern u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, - DAPL_HASH_DATA *p_data); - -#endif /* DAPL_HASH_H */ diff -Nurp -X dontdiff dat-provider_bind/dapl_hca_util.c dat-provider/dapl_hca_util.c --- dat-provider_bind/dapl_hca_util.c Sun Jun 19 15:17:54 2005 +++ dat-provider/dapl_hca_util.c Sun Jun 19 15:24:06 2005 @@ -34,7 +34,6 @@ #include "dapl_openib_util.h" #include "dapl_provider.h" #include "dapl_hca_util.h" -#include "dapl_hash.h" /* * dapl_hca_alloc @@ -60,20 +59,13 @@ struct dapl_hca *dapl_hca_alloc(char *na if (hca) { memset(hca, 0, sizeof *hca); - if (DAT_SUCCESS == - dapl_hash_create(DAPL_HASH_TABLE_DEFAULT_CAPACITY, - &hca->lmr_hash_table)) { - spin_lock_init(&hca->lock); - INIT_LIST_HEAD(&hca->ia_list); - - hca->name = dapl_os_strdup(name); - hca->ib_hca_handle = device; - hca->port_num = port; - if (hca->name == NULL) { - kfree(hca); - hca = NULL; - } - } else { + spin_lock_init(&hca->lock); + INIT_LIST_HEAD(&hca->ia_list); + + hca->name = dapl_os_strdup(name); + hca->ib_hca_handle = device; + hca->port_num = port; + if (hca->name == NULL) { kfree(hca); hca = NULL; } @@ -99,7 +91,6 @@ struct dapl_hca *dapl_hca_alloc(char *na */ void dapl_hca_free(struct dapl_hca *hca) { - (void)dapl_hash_free(hca->lmr_hash_table); kfree(hca->name); kfree(hca); } diff -Nurp -X dontdiff dat-provider_bind/dapl_lmr.c dat-provider/dapl_lmr.c --- dat-provider_bind/dapl_lmr.c Sun Jun 19 15:17:54 2005 +++ dat-provider/dapl_lmr.c Sun Jun 19 15:24:06 2005 @@ -32,7 +32,6 @@ #include "dapl_openib_util.h" #include "dapl_ia.h" -#include "dapl_hash.h" static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, @@ -104,17 +103,6 @@ static inline u32 dapl_lmr_create_virtua if (DAT_SUCCESS != status) goto error2; - /* if the LMR context is already in the hash table */ - status = dapl_hash_search(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, NULL); - if (status == DAT_SUCCESS) - goto error3; - - status = dapl_hash_insert(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, lmr); - if (status != DAT_SUCCESS) - goto error3; - atomic_inc(&pz->pz_ref_count); if (lmr) @@ -130,8 +118,6 @@ static inline u32 dapl_lmr_create_virtua return DAT_SUCCESS; -error3: - (void)dapl_ib_mr_deregister(new_lmr); error2: dapl_lmr_dealloc(new_lmr); error1: @@ -168,17 +154,6 @@ static inline u32 dapl_lmr_create_physic if (DAT_SUCCESS != status) goto error2; - /* if the LMR context is already in the hash table */ - status = dapl_hash_search(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, NULL); - if (status == DAT_SUCCESS) - goto error3; - - status = dapl_hash_insert(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, lmr); - if (status != DAT_SUCCESS) - goto error3; - atomic_inc(&pz->pz_ref_count); if (lmr) @@ -194,8 +169,6 @@ static inline u32 dapl_lmr_create_physic return DAT_SUCCESS; -error3: - (void)dapl_ib_mr_deregister(new_lmr); error2: dapl_lmr_dealloc(new_lmr); error1: @@ -216,12 +189,6 @@ static inline u32 dapl_lmr_create_lmr(st DAT_REGION_DESCRIPTION reg_desc; u32 status; - status = dapl_hash_search(ia->hca->lmr_hash_table, - original_lmr->param.lmr_context, - (DAPL_HASH_DATA *) &lmr); - if (status != DAT_SUCCESS) - goto error1; - reg_desc.for_lmr = (struct dat_lmr *) original_lmr; new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, @@ -236,17 +203,6 @@ static inline u32 dapl_lmr_create_lmr(st if (DAT_SUCCESS != status) goto error2; - /* if the LMR context is already in the hash table */ - status = dapl_hash_search(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, NULL); - if (status == DAT_SUCCESS) - goto error3; - - status = dapl_hash_insert(ia->hca->lmr_hash_table, - new_lmr->param.lmr_context, lmr); - if (status != DAT_SUCCESS) - goto error3; - atomic_inc(&pz->pz_ref_count); if (lmr) @@ -263,8 +219,6 @@ static inline u32 dapl_lmr_create_lmr(st return DAT_SUCCESS; -error3: - dapl_ib_mr_deregister(new_lmr); error2: dapl_lmr_dealloc(new_lmr); error1: @@ -372,22 +326,12 @@ u32 dapl_lmr_free(struct dat_lmr *lmr) if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) return DAT_INVALID_STATE; - status = dapl_hash_remove( - dapl_lmr->common.owner_ia->hca->lmr_hash_table, - dapl_lmr->param.lmr_context, NULL); - if (status != DAT_SUCCESS) - goto error; - status = dapl_ib_mr_deregister(dapl_lmr); if (status == DAT_SUCCESS) { pz = (struct dapl_pz *)dapl_lmr->param.pz; atomic_dec(&pz->pz_ref_count); dapl_lmr_dealloc(dapl_lmr); - } else /* failure; put dapl_lmr back in hash table */ - dapl_hash_insert(dapl_lmr->common.owner_ia-> - hca->lmr_hash_table, - dapl_lmr->param.lmr_context, dapl_lmr); - + } break; } case DAT_MEM_TYPE_PLATFORM: diff -Nurp -X dontdiff dat-provider_bind/dapl_rmr.c dat-provider/dapl_rmr.c --- dat-provider_bind/dapl_rmr.c Sun Jun 19 15:17:54 2005 +++ dat-provider/dapl_rmr.c Sun Jun 19 15:24:06 2005 @@ -33,7 +33,6 @@ #include "dapl.h" #include "dapl_ep.h" #include "dapl_ia.h" -#include "dapl_hash.h" #include "dapl_cookie.h" #include "dapl_openib_util.h" diff -Nurp -X dontdiff dat-provider_bind/dapl_util.h dat-provider/dapl_util.h --- dat-provider_bind/dapl_util.h Sun Jun 19 15:17:54 2005 +++ dat-provider/dapl_util.h Sun Jun 19 15:24:30 2005 @@ -144,22 +144,6 @@ static inline char *dapl_os_strdup(const #endif /* - * dapl_os_mod64 - * - * Returne the modulo of a 64 bit number. Given that this is running - * on a 32 bit platform, we need to use the kernel macro to prevent - * exceptions from killing the machine. - */ -static inline long dapl_os_mod64(uint64_t key, unsigned long hashsize) -{ -#ifdef __ia64__ - return ((uint64_t) ((key) % (hashsize))) -#else - return do_div(key, hashsize); -#endif -} - -/* * Debug Functions */ -- Itamar From itamar at mellanox.co.il Mon Jun 20 01:46:16 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Mon, 20 Jun 2005 11:46:16 +0300 Subject: [openib-general] [PATCH][kdapl] Small changes in dapl_hca_alloc/dapl_hca_free functions Message-ID: <20050620084616.GA7076@mellanox.co.il> Small changes in dapl_hca_alloc/dapl_hca_free function Signed-off-by: Itamar Rabenstein diff -Nurp -X dontdiff dat-provider_hash/dapl_hca_util.c dat-provider/dapl_hca_util.c --- dat-provider_hash/dapl_hca_util.c Sun Jun 19 16:12:55 2005 +++ dat-provider/dapl_hca_util.c Sun Jun 19 16:30:07 2005 @@ -54,23 +54,18 @@ struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) { struct dapl_hca *hca; - - hca = kmalloc(sizeof *hca, GFP_ATOMIC); + int malloc_size = sizeof *hca + strlen(name) + 1; + + hca = kmalloc(malloc_size, GFP_ATOMIC); if (hca) { - memset(hca, 0, sizeof *hca); - + memset(hca, 0, malloc_size); spin_lock_init(&hca->lock); INIT_LIST_HEAD(&hca->ia_list); - - hca->name = dapl_os_strdup(name); + hca->name = (char *)hca + sizeof *hca; + strcpy(hca->name, name); hca->ib_hca_handle = device; hca->port_num = port; - if (hca->name == NULL) { - kfree(hca); - hca = NULL; - } } - return hca; } @@ -91,7 +86,6 @@ struct dapl_hca *dapl_hca_alloc(char *na */ void dapl_hca_free(struct dapl_hca *hca) { - kfree(hca->name); kfree(hca); } diff -Nurp -X dontdiff dat-provider_hash/dapl_util.h dat-provider/dapl_util.h --- dat-provider_hash/dapl_util.h Sun Jun 19 16:12:56 2005 +++ dat-provider/dapl_util.h Sun Jun 19 16:30:52 2005 @@ -121,18 +121,6 @@ static inline void *dapl_os_realloc(void } /* - * String Functions - */ - -static inline char *dapl_os_strdup(const char *str) -{ - char *ns = kmalloc(strlen(str) + 1, GFP_KERNEL); - if (ns) - strcpy(ns, str); - return ns; -} - -/* * *printf format helper. We use the C string constant concatenation * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. -- Itamar From itamar at mellanox.co.il Mon Jun 20 01:47:56 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Mon, 20 Jun 2005 11:47:56 +0300 Subject: [openib-general] [PATCH][kdapl] Integrate dapl_hca_link_ia/dapl_hca_unlink_ia into dapl_ia.c Message-ID: <20050620084756.GA7088@mellanox.co.il> Integrate dapl_hca_link_ia/dapl_hca_unlink_ia into dapl_ia.c (no need for functions that just call LIST_ADD and LIST_DEL) Signed-off-by: Itamar Rabenstein diff -Nurp -X dontdiff dat-provider_simp/dapl_hca_util.c dat-provider/dapl_hca_util.c --- dat-provider_simp/dapl_hca_util.c Sun Jun 19 18:36:31 2005 +++ dat-provider/dapl_hca_util.c Sun Jun 19 18:38:33 2005 @@ -88,49 +88,3 @@ void dapl_hca_free(struct dapl_hca *hca) { kfree(hca); } - -/* - * dapl_hca_link_ia - * - * Add an ia to the HCA structure - * - * Input: - * hca - * ia_ptr - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia_ptr) -{ - spin_lock_irqsave(&hca->lock, hca->flags); - list_add(&ia_ptr->list, &hca->ia_list); - spin_unlock_irqrestore(&hca->lock, hca->flags); -} - -/* - * dapl_hca_unlink_ia - * - * Remove an ia from the hca info structure - * - * Input: - * hca - * ia_ptr - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia) -{ - spin_lock_irqsave(&hca->lock, hca->flags); - list_del(&ia->list); - spin_unlock_irqrestore(&hca->lock, hca->flags); -} diff -Nurp -X dontdiff dat-provider_simp/dapl_hca_util.h dat-provider/dapl_hca_util.h --- dat-provider_simp/dapl_hca_util.h Sun Jun 19 18:36:31 2005 +++ dat-provider/dapl_hca_util.h Sun Jun 19 18:36:45 2005 @@ -39,8 +39,4 @@ struct dapl_hca *dapl_hca_alloc(char *na void dapl_hca_free(struct dapl_hca *hca); -void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia); - -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia); - #endif diff -Nurp -X dontdiff dat-provider_simp/dapl_ia.c dat-provider/dapl_ia.c --- dat-provider_simp/dapl_ia.c Sun Jun 19 18:36:31 2005 +++ dat-provider/dapl_ia.c Sun Jun 19 18:39:03 2005 @@ -34,7 +34,6 @@ #include "dapl_ia.h" #include "dapl_provider.h" #include "dapl_evd.h" -#include "dapl_hca_util.h" #include "dapl_openib_util.h" #include "dapl_sp.h" #include "dapl_cr.h" @@ -71,7 +70,9 @@ static struct dapl_ia *dapl_ia_alloc(str INIT_LIST_HEAD(&ia->psp_list); INIT_LIST_HEAD(&ia->srq_list); - dapl_hca_link_ia(hca, ia); + spin_lock_irqsave(&hca->lock, hca->flags); + list_add(&ia->list, &hca->ia_list); + spin_unlock_irqrestore(&hca->lock, hca->flags); return ia; } @@ -379,7 +380,9 @@ void dapl_ia_free(struct dapl_ia *ia) dapl_os_assert(list_empty(&ia->psp_list)); dapl_os_assert(list_empty(&ia->rsp_list)); - dapl_hca_unlink_ia(ia->hca, ia); + spin_lock_irqsave(&ia->hca->lock, ia->hca->flags); + list_del(&ia->list); + spin_unlock_irqrestore(&ia->hca->lock, ia->hca->flags); /* no need to destroy ia->common.lock */ kfree(ia); -- Itamar From itamar at mellanox.co.il Mon Jun 20 01:50:10 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Mon, 20 Jun 2005 11:50:10 +0300 Subject: [openib-general] [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c Message-ID: <20050620085010.GA7206@mellanox.co.il> ntegrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c (no need for 2 files just for 2 simple function that kmalloc and kfree. There is not any special logic in this functions that need to separate them into different files) Signed-off-by: Itamar Rabenstein diff -Nurp -X dontdiff dat-provider_link_ia/Makefile dat-provider/Makefile --- dat-provider_link_ia/Makefile Sun Jun 19 18:43:16 2005 +++ dat-provider/Makefile Sun Jun 19 19:45:15 2005 @@ -20,7 +20,6 @@ PROVIDER_MODULES := \ dapl_cr \ dapl_ep \ dapl_evd \ - dapl_hca_util \ dapl_ia \ dapl_lmr \ dapl_provider \ diff -Nurp -X dontdiff dat-provider_link_ia/dapl_hca_util.c dat-provider/dapl_hca_util.c --- dat-provider_link_ia/dapl_hca_util.c Sun Jun 19 18:43:16 2005 +++ dat-provider/dapl_hca_util.c Thu Jan 1 02:00:00 1970 @@ -1,90 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_hca_util.c 2640 2005-06-16 16:22:46Z jlentini $ - */ - -#include "dapl.h" -#include "dapl_openib_util.h" -#include "dapl_provider.h" -#include "dapl_hca_util.h" - -/* - * dapl_hca_alloc - * - * alloc and initialize an HCA struct - * - * Input: - * name - * port - * - * Output: - * hca - * - * Returns: - * none - * - */ -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) -{ - struct dapl_hca *hca; - int malloc_size = sizeof *hca + strlen(name) + 1; - - hca = kmalloc(malloc_size, GFP_ATOMIC); - if (hca) { - memset(hca, 0, malloc_size); - spin_lock_init(&hca->lock); - INIT_LIST_HEAD(&hca->ia_list); - hca->name = (char *)hca + sizeof *hca; - strcpy(hca->name, name); - hca->ib_hca_handle = device; - hca->port_num = port; - } - return hca; -} - -/* - * dapl_hca_free - * - * free an IA INFO struct - * - * Input: - * hca - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_hca_free(struct dapl_hca *hca) -{ - kfree(hca); -} diff -Nurp -X dontdiff dat-provider_link_ia/dapl_hca_util.h dat-provider/dapl_hca_util.h --- dat-provider_link_ia/dapl_hca_util.h Sun Jun 19 18:43:16 2005 +++ dat-provider/dapl_hca_util.h Thu Jan 1 02:00:00 1970 @@ -1,42 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_hca_util.h 2640 2005-06-16 16:22:46Z jlentini $ - */ - -#ifndef DAPL_HCA_UTIL_H -#define DAPL_HCA_UTIL_H - -#include "dapl.h" - -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port); - -void dapl_hca_free(struct dapl_hca *hca); - -#endif diff -Nurp -X dontdiff dat-provider_link_ia/dapl_provider.c dat-provider/dapl_provider.c --- dat-provider_link_ia/dapl_provider.c Sun Jun 19 18:43:16 2005 +++ dat-provider/dapl_provider.c Sun Jun 19 19:47:16 2005 @@ -35,7 +35,6 @@ #include #include "dapl.h" -#include "dapl_hca_util.h" #include "dapl_provider.h" #include "dapl_util.h" #include "dapl_openib_util.h" @@ -246,6 +245,24 @@ static void dapl_provider_info_init(stru provider_info->ia_name[i+1] = '\0'; } +static struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) +{ + struct dapl_hca *hca; + int malloc_size = sizeof *hca + strlen(name) + 1; + + hca = kmalloc(malloc_size, GFP_ATOMIC); + if (hca) { + memset(hca, 0, malloc_size); + spin_lock_init(&hca->lock); + INIT_LIST_HEAD(&hca->ia_list); + hca->name = (char *)hca + sizeof *hca; + strcpy(hca->name, name); + hca->ib_hca_handle = device; + hca->port_num = port; + } + return hca; +} + static void dapl_add_port(struct ib_device *device, u8 port) { struct dat_provider_info provider_info; @@ -305,7 +322,7 @@ error: (void)dapl_provider_list_remove(provider_info.ia_name); if (NULL != hca) - dapl_hca_free(hca); + kfree(hca); } } @@ -338,7 +355,7 @@ static void dapl_remove_port(struct ib_d provider_info.ia_name); } - dapl_hca_free(provider->extension); + kfree(provider->extension); dapl_provider_list_remove(provider_info.ia_name); } -- Itamar From tomduffy at gmail.com Mon Jun 20 08:00:13 2005 From: tomduffy at gmail.com (Tom Duffy) Date: Mon, 20 Jun 2005 08:00:13 -0700 Subject: [openib-general] [PATCH][kdapl] Small changes in dapl_hca_alloc/dapl_hca_free functions In-Reply-To: <20050620084616.GA7076@mellanox.co.il> References: <20050620084616.GA7076@mellanox.co.il> Message-ID: <9d3b7de705062008003b2a1219@mail.gmail.com> On 6/20/05, Itamar Rabenstein wrote: > Small changes in dapl_hca_alloc/dapl_hca_free function > > Signed-off-by: Itamar Rabenstein > > diff -Nurp -X dontdiff dat-provider_hash/dapl_hca_util.c dat-provider/dapl_hca_util.c > --- dat-provider_hash/dapl_hca_util.c Sun Jun 19 16:12:55 2005 > +++ dat-provider/dapl_hca_util.c Sun Jun 19 16:30:07 2005 > @@ -54,23 +54,18 @@ > struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) > { > struct dapl_hca *hca; > - > - hca = kmalloc(sizeof *hca, GFP_ATOMIC); > + int malloc_size = sizeof *hca + strlen(name) + 1; > + > + hca = kmalloc(malloc_size, GFP_ATOMIC); > if (hca) { > - memset(hca, 0, sizeof *hca); > - > + memset(hca, 0, malloc_size); > spin_lock_init(&hca->lock); > INIT_LIST_HEAD(&hca->ia_list); > - > - hca->name = dapl_os_strdup(name); > + hca->name = (char *)hca + sizeof *hca; > + strcpy(hca->name, name); Shouldn't you use strncpy() and bound it? -tduffy From mst at mellanox.co.il Mon Jun 20 08:14:04 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 20 Jun 2005 18:14:04 +0300 Subject: [openib-general] [PATCH] ipoib tune rx ring size Message-ID: <20050620151404.GW20041@mellanox.co.il> Reducing the rx ring size gives me bandwidth win of about 1%. Signed-off-by: Michael S. Tsirkin Index: ulp/ipoib/ipoib.h =================================================================== --- ulp/ipoib/ipoib.h (revision 2656) +++ ulp/ipoib/ipoib.h (working copy) @@ -62,7 +62,7 @@ enum { IPOIB_ENCAP_LEN = 4, - IPOIB_RX_RING_SIZE = 128, + IPOIB_RX_RING_SIZE = 64, IPOIB_TX_RING_SIZE = 64, IPOIB_NUM_WC = 4, -- MST From iod00d at hp.com Mon Jun 20 08:23:08 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 20 Jun 2005 08:23:08 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050615215334.GA22691@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050615150440.GB20041@mellanox.co.il> <20050615173058.GB4379@esmail.cup.hp.com> <20050615215334.GA22691@mellanox.co.il> Message-ID: <20050620152308.GA10674@esmail.cup.hp.com> On Thu, Jun 16, 2005 at 12:53:34AM +0300, Michael S. Tsirkin wrote: > Quoting r. Grant Grundler : > > 2) I working towards making the server side a daemon. > > I dont see this necessarily as an improvement. > I'm happy running server and client with ssh from a script > with multiple parameters, this works well for me - actually better > than netserver. Michael, You are right that I could run the test as a normal user. I don't recall why (if I ever had a reason) I was running the test as root. Could you commit your script along side the rdma_lat.c/rdma_bw.c sources? thanks, grant From iod00d at hp.com Mon Jun 20 09:13:49 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 20 Jun 2005 09:13:49 -0700 Subject: [openib-general] Re: [PATCH] mthca: add copyright statements In-Reply-To: <523bre2m8q.fsf@topspin.com> References: <20050619143753.GU20041@mellanox.co.il> <523bre2m8q.fsf@topspin.com> Message-ID: <20050620161349.GA10881@esmail.cup.hp.com> On Sun, Jun 19, 2005 at 12:13:41PM -0700, Roland Dreier wrote: > > + * Copyright (c) 2005 Mellanox Technologies (Michael S. Tsirkin). > > I believe you want to include the statement "All rights reserved." > Also, who is the copyright holder -- MST as an individual or Mellanox > Technologies? As it stands I'm not sure what this line is actually > saying -- what does it mean to have your name in parentheses? The name in parenthesis normally indicates a contact person (ie person adding the copyright). Copyright belongs to Mellanox. grant From iod00d at hp.com Mon Jun 20 09:30:44 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 20 Jun 2005 09:30:44 -0700 Subject: [openib-general] [PATCH] ipoib tune rx ring size In-Reply-To: <20050620151404.GW20041@mellanox.co.il> References: <20050620151404.GW20041@mellanox.co.il> Message-ID: <20050620163044.GC10881@esmail.cup.hp.com> On Mon, Jun 20, 2005 at 06:14:04PM +0300, Michael S. Tsirkin wrote: > Reducing the rx ring size gives me bandwidth win of about 1%. Michael, Doesn't this sound counter-intuitive to you? Can you share which configuration (chipset, CPU, IO bus, HCA) you measured this on? I've also seen this to be true with gige - but only in one odd configuration. thanks, grant From itamar at mellanox.co.il Mon Jun 20 10:00:14 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Mon, 20 Jun 2005 20:00:14 +0300 Subject: [openib-general] [PATCH][kdapl] Small changes in dapl_hca_all oc/dapl_hca_free functions Message-ID: <91DB792C7985D411BEC300B40080D29CC35CD6@mtvex01.mtv.mtl.com> Hi Tom, I was just copied to original code from dapl_os_strdup() to dapl_hca_alloc(). so if it is a problem is it was there in the code before my patch. But why do you think we need strncpy and not strcpy ? name param is null terminated string. Itamar > -----Original Message----- > From: Tom Duffy [mailto:tomduffy at gmail.com] > Sent: Monday, June 20, 2005 6:00 PM > To: Itamar Rabenstein > Cc: James.Lentini at netapp.com; openib-general at openib.org > Subject: Re: [openib-general] [PATCH][kdapl] Small changes in > dapl_hca_alloc/dapl_hca_free functions > > > On 6/20/05, Itamar Rabenstein wrote: > > Small changes in dapl_hca_alloc/dapl_hca_free function > > > > Signed-off-by: Itamar Rabenstein > > > > diff -Nurp -X dontdiff dat-provider_hash/dapl_hca_util.c > dat-provider/dapl_hca_util.c > > --- dat-provider_hash/dapl_hca_util.c Sun Jun 19 16:12:55 2005 > > +++ dat-provider/dapl_hca_util.c Sun Jun 19 16:30:07 2005 > > @@ -54,23 +54,18 @@ > > struct dapl_hca *dapl_hca_alloc(char *name, struct > ib_device *device, u8 port) > > { > > struct dapl_hca *hca; > > - > > - hca = kmalloc(sizeof *hca, GFP_ATOMIC); > > + int malloc_size = sizeof *hca + strlen(name) + 1; > > + > > + hca = kmalloc(malloc_size, GFP_ATOMIC); > > if (hca) { > > - memset(hca, 0, sizeof *hca); > > - > > + memset(hca, 0, malloc_size); > > spin_lock_init(&hca->lock); > > INIT_LIST_HEAD(&hca->ia_list); > > - > > - hca->name = dapl_os_strdup(name); > > + hca->name = (char *)hca + sizeof *hca; > > + strcpy(hca->name, name); > > Shouldn't you use strncpy() and bound it? > > -tduffy > From tduffy at sun.com Mon Jun 20 09:48:20 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 20 Jun 2005 09:48:20 -0700 Subject: [openib-general] [PATCH][kdapl] Small changes in dapl_hca_all oc/dapl_hca_free functions In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35CD6@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35CD6@mtvex01.mtv.mtl.com> Message-ID: <1119286100.25973.2.camel@duffman> On Mon, 2005-06-20 at 20:00 +0300, Itamar Rabenstein wrote: > Hi Tom, > I was just copied to original code from dapl_os_strdup() to > dapl_hca_alloc(). > so if it is a problem is it was there in the code before my patch. > But why do you think we need strncpy and not strcpy ? > name param is null terminated string. It is generally not a good idea to allow an arbitrary sized string in the kernel. Especially when you are kmalloc()ing the size. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Mon Jun 20 09:51:15 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 20 Jun 2005 12:51:15 -0400 Subject: [openib-general] Re: [PATCH] [RMPP] fix handling of duplicate RMPP ACK In-Reply-To: References: Message-ID: <1119286275.6884.5.camel@ggreen_laptop.us.voltaire.com> Hi Sean, On Tue, 2005-06-07 at 17:58, Sean Hefty wrote: > Hal, can you test this and see if it corrects the problems that you were > seeing on the SA side? Just tried this with Solaris 10 and it works fine either way. The problem I was having appeared related but was separate from this and has since been resolved. Thanks. -- Hal From halr at voltaire.com Mon Jun 20 10:11:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 20 Jun 2005 13:11:51 -0400 Subject: [openib-general] [PATCH] OpenSM: Set retries to 0 if RMPP In-Reply-To: <1118183846.22316.19.camel@duffman> References: <1118182327.4477.30.camel@localhost.localdomain> <1118182869.22316.8.camel@duffman> <1118183846.22316.19.camel@duffman> Message-ID: <1119287511.6884.15.camel@ggreen_laptop.us.voltaire.com> Hi Tom, On Tue, 2005-06-07 at 18:37, Tom Duffy wrote: > On Tue, 2005-06-07 at 15:21 -0700, Tom Duffy wrote: > > On Tue, 2005-06-07 at 18:12 -0400, Hal Rosenstock wrote: > > > With this change, able to interoperate with Solaris 10 SA client (which > > > uses RMPP) with OpenSM > > > > Cool. I will give it a try. What do you mean by interop? Were you > > able to bring up IPoIB and ping? > > Nevermind. Just tried it with Solaris Nevada build 16 and it works > too! What's Nevada ? > Albeit with some minor caveats. For instance, a broadcast ping from > Linux does not see the Solaris node: This worked for me with a Linux and Solaris 10 node. > This is a linux system: > > [root at sins-stinger-10 ~]# ping 192.168.0.0 -b > WARNING: pinging broadcast address > PING 192.168.0.0 (192.168.0.0) 56(84) bytes of data. > 64 bytes from 192.168.0.233: icmp_seq=0 ttl=64 time=0.054 ms > 64 bytes from 192.168.0.26: icmp_seq=0 ttl=64 time=0.159 ms (DUP!) > 64 bytes from 192.168.0.233: icmp_seq=1 ttl=64 time=0.016 ms > 64 bytes from 192.168.0.26: icmp_seq=1 ttl=64 time=0.114 ms (DUP!) > > From the Solaris system: > > [root at nisus ~]# ping -s 192.168.0.0 > PING 192.168.0.0: 56 data bytes > 64 bytes from 192.168.0.78: icmp_seq=0. time=0.501 ms > 64 bytes from 192.168.0.233: icmp_seq=0. time=5.95 ms > 64 bytes from 192.168.0.26: icmp_seq=0. time=10.4 ms > > Solaris 10 was unhappy: > > [root at blissom ~]# ifconfig ibd0 plumb > ifconfig: SIOCSLIFNAME for ip: ibd0: no such interface > Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_ATTACH_REQ failed: DL_BADPPA > Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_BIND_REQ failed: DL_OUTSTATE > Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_PHYS_ADDR_REQ failed: DL_OUTSTATE > Jun 7 15:29:02 blissom.SFBay.Sun.COM ip: ibd0: DL_UNBIND_REQ failed: DL_OUTSTATE > > Which made my Nevada box unhappy: > > < from dmesg > > interface ibd0 to 192.168.0.78 broken: in=3 ierr=0 out=5 oerr=2 Can you retry with the latest and greatest (both OpenSM as well as the latest mad_rmpp code in the kernel) ? Thanks. -- Hal From tduffy at sun.com Mon Jun 20 11:06:12 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 20 Jun 2005 11:06:12 -0700 Subject: [openib-general] [PATCHv7] SDP: make SDP buildable on 2.6.12-final Message-ID: <1119290772.25973.9.camel@duffman> Here is the updated patch based off r2663 and building off of 2.6.12. Signed-off-by: Tom Duffy Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -91,8 +91,8 @@ static int sdp_rcvd_disconnect(struct sd /* * async notification. POLL_HUP on full duplex close only. */ - sdp_inet_wake_generic(conn->sk); - sk_wake_async(conn->sk, 1, band); + sdp_inet_wake_generic(sk_sdp(conn)); + sk_wake_async(sk_sdp(conn), 1, band); return 0; error: @@ -1169,7 +1169,7 @@ int sdp_event_recv(struct sdp_opt *conn, * If data was consumed by the protocol, signal * the user. */ - sdp_inet_wake_recv(conn->sk, conn->byte_strm); + sdp_inet_wake_recv(sk_sdp(conn), conn->byte_strm); /* * It's possible that a new recv buffer advertisment opened up the * recv window and we can flush buffered send data Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -130,9 +130,9 @@ MODULE_PARM_DESC(sdp_debug_level, */ void sdp_inet_wake_send(struct sock *sk) { - struct sdp_opt *conn; + struct sdp_opt *conn = sdp_sk(sk); - if (!sk || !(conn = SDP_GET_CONN(sk))) + if (!sk) return; if (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) && @@ -295,27 +295,22 @@ error: static int sdp_inet_release(struct socket *sock) { struct sdp_opt *conn; - struct sock *sk; + struct sock *sk = sock->sk; int result; long timeout; u32 flags; - if (!sock->sk) { + if (!sk) { sdp_dbg_warn(NULL, "release empty <%d:%d> flags <%08lx>", sock->type, sock->state, sock->flags); return 0; } - sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "RELEASE: linger <%d:%lu> data <%d:%d>", sock_flag(sk, SOCK_LINGER), sk->sk_lingertime, conn->byte_strm, conn->src_recv); - /* - * clear out sock, so we only do this once. - */ - sock->sk = NULL; sdp_conn_lock(conn); conn->shutdown = SHUTDOWN_MASK; @@ -412,6 +407,7 @@ done: * finally drop socket reference. (socket API reference) */ sock_orphan(sk); + sock->sk = NULL; sdp_conn_unlock(conn); sdp_conn_put(conn); @@ -431,7 +427,7 @@ static int sdp_inet_bind(struct socket * int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "BIND: family <%d> addr <%08x:%04x>", addr->sin_family, addr->sin_addr.s_addr, addr->sin_port); @@ -522,7 +518,7 @@ static int sdp_inet_connect(struct socke int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "CONNECT: family <%d> addr <%08x:%04x>", addr->sin_family, addr->sin_addr.s_addr, addr->sin_port); @@ -684,7 +680,7 @@ static int sdp_inet_listen(struct socket int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "LISTEN: addr <%08x:%04x> backlog <%04x>", conn->src_addr, conn->src_port, backlog); @@ -745,7 +741,7 @@ static int sdp_inet_accept(struct socket long timeout; listen_sk = listen_sock->sk; - listen_conn = SDP_GET_CONN(listen_sk); + listen_conn = sdp_sk(listen_sk); sdp_dbg_ctrl(listen_conn, "ACCEPT: addr <%08x:%04x>", listen_conn->src_addr, listen_conn->src_port); @@ -801,7 +797,7 @@ static int sdp_inet_accept(struct socket goto listen_done; } } else { - accept_sk = accept_conn->sk; + accept_sk = sk_sdp(accept_conn); switch (accept_conn->state) { case SDP_CONN_ST_REQ_RECV: @@ -870,7 +866,7 @@ static int sdp_inet_getname(struct socke struct sdp_opt *conn; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "GETNAME: src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -910,7 +906,7 @@ static unsigned int sdp_inet_poll(struct * recheck the falgs on being woken. */ sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "POLL: socket flags <%08lx>", sock->flags); @@ -997,7 +993,7 @@ static int sdp_inet_ioctl(struct socket int value; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "IOCTL: command <%d> argument <%08lx>", cmd, arg); /* @@ -1119,7 +1115,7 @@ static int sdp_inet_setopt(struct socket int result = 0; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "SETSOCKOPT: level <%d> option <%d>", level, optname); @@ -1184,7 +1180,7 @@ static int sdp_inet_getopt(struct socket int len; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "GETSOCKOPT: level <%d> option <%d>", level, optname); @@ -1242,7 +1238,7 @@ static int sdp_inet_shutdown(struct sock int result = 0; struct sdp_opt *conn; - conn = SDP_GET_CONN(sock->sk); + conn = sdp_sk(sock->sk); sdp_dbg_ctrl(conn, "SHUTDOWN: flag <%d>", flag); /* @@ -1350,7 +1346,7 @@ static int sdp_inet_create(struct socket sock->ops = &lnx_stream_ops; sock->state = SS_UNCONNECTED; - sock_graft(conn->sk, sock); + sock_graft(sk_sdp(conn), sock); conn->pid = current->pid; @@ -1377,6 +1373,11 @@ static struct net_proto_family sdp_proto /* * SDP host module load/unload functions */ +struct proto sdp_sk_proto = { + .name = "SDP", + .owner = THIS_MODULE, + .obj_size = sizeof(struct sdp_opt), +}; /* * sdp_init - initialize the sdp host module @@ -1387,6 +1388,12 @@ static int __init sdp_init(void) sdp_dbg_init("SDP module load."); + result = proto_register(&sdp_sk_proto, 1); + if (result < 0) { + sdp_warn("INIT: Error <%d> registering sk proto,", result); + goto error_proto; + } + /* * proc entries */ @@ -1460,6 +1467,8 @@ error_link: error_advt: sdp_main_proc_cleanup(); error_proc: + proto_unregister(&sdp_sk_proto); +error_proto: return result; /* success */ } @@ -1493,6 +1502,8 @@ static void __exit sdp_exit(void) * proc tables */ sdp_main_proc_cleanup(); + + proto_unregister(&sdp_sk_proto); } module_init(sdp_init); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c (working copy) @@ -183,7 +183,7 @@ int sdp_event_read(struct sdp_opt *conn, */ conn->byte_strm += result; - sdp_inet_wake_recv(conn->sk, conn->byte_strm); + sdp_inet_wake_recv(sk_sdp(conn), conn->byte_strm); } else { if (result < 0) sdp_dbg_warn(conn, "Error <%d> receiving buff", @@ -220,7 +220,7 @@ int sdp_event_read(struct sdp_opt *conn, iocb->flags &= ~(SDP_IOCB_F_ACTIVE | SDP_IOCB_F_RDMA_R); - if (conn->sk->sk_rcvlowat > iocb->post) + if (sk_sdp(conn)->sk_rcvlowat > iocb->post) break; SDP_CONN_STAT_READ_INC(conn, iocb->post); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c (working copy) @@ -1695,7 +1695,7 @@ static int sdp_inet_write_cancel(struct /* * lock the socket while we operate. */ - conn = SDP_GET_CONN(si->sock->sk); + conn = sdp_sk(si->sock->sk); sdp_conn_lock(conn); sdp_dbg_ctrl(conn, "Cancel Write IOCB. <%08x:%04x> <%08x:%04x>", @@ -1898,7 +1898,7 @@ int sdp_send_flush(struct sdp_opt *conn) /* * see if there is enough buffer to wake/notify writers */ - sdp_inet_wake_send(conn->sk); + sdp_inet_wake_send(sk_sdp(conn)); return 0; done: @@ -1927,7 +1927,7 @@ int sdp_inet_send(struct kiocb *req, str oob = (msg->msg_flags & MSG_OOB); sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "send state <%04x> size <%Zu> flags <%08x>", conn->state, size, msg->msg_flags); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (working copy) @@ -41,6 +41,7 @@ void sdp_cm_actv_error(struct sdp_opt *conn, int error) { int result; + struct sock *sk; /* * error value is positive error. * @@ -93,11 +94,12 @@ void sdp_cm_actv_error(struct sdp_opt *c conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; - if (conn->sk->sk_socket) - conn->sk->sk_socket->state = SS_UNCONNECTED; + sk = sk_sdp(conn); + if (sk->sk_socket) + sk->sk_socket->state = SS_UNCONNECTED; sdp_iocb_q_cancel_all(conn, (0 - error)); - sdp_inet_wake_error(conn->sk); + sdp_inet_wake_error(sk); } /* @@ -114,7 +116,7 @@ static int sdp_cm_actv_establish(struct conn->src_addr, conn->src_port, conn->dst_addr, conn->dst_port); - sk = conn->sk; + sk = sk_sdp(conn); qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL); if (!qp_attr) @@ -532,7 +534,7 @@ int sdp_cm_connect(struct sdp_opt *conn) result = sdp_link_path_lookup(htonl(conn->dst_addr), htonl(conn->src_addr), - conn->sk->sk_bound_dev_if, + sk_sdp(conn)->sk_bound_dev_if, sdp_cm_path_complete, conn, &conn->plid); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -76,6 +76,8 @@ static u32 sdp_psn_generate(void) void sdp_conn_inet_error(struct sdp_opt *conn, int error) { + struct sock *sk; + sdp_dbg_ctrl(conn, "report connection error <%d>", error); /* * the connection has failed, move to error, and notify anyone @@ -89,11 +91,12 @@ void sdp_conn_inet_error(struct sdp_opt conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; - if (conn->sk->sk_socket) - conn->sk->sk_socket->state = SS_UNCONNECTED; + sk = sk_sdp(conn); + if (sk->sk_socket) + sk->sk_socket->state = SS_UNCONNECTED; sdp_iocb_q_cancel_all(conn, -error); - sdp_inet_wake_error(conn->sk); + sdp_inet_wake_error(sk); } void sdp_conn_abort(struct sdp_opt *conn) @@ -403,7 +406,7 @@ int sdp_inet_port_get(struct sdp_opt *co static s32 rover = -1; unsigned long flags; - sk = conn->sk; + sk = sk_sdp(conn); /* * lock table */ @@ -414,7 +417,7 @@ int sdp_inet_port_get(struct sdp_opt *co if (port > 0) { for (look = dev_root_s.bind_list, port_ok = 1; look; look = look->bind_next) { - srch = look->sk; + srch = sk_sdp(look); /* * 1) same port * 2) linux force reuse is off. @@ -842,17 +845,8 @@ void sdp_conn_destruct(struct sdp_opt *c if (dump) sdp_conn_state_dump(conn); - /* - * free the OS socket structure - */ - if (!conn->sk) - sdp_dbg_warn(conn, "destruct, no socket! continuing."); - else { - sk_free(conn->sk); - conn->sk = NULL; - } - kmem_cache_free(dev_root_s.conn_cache, conn); + sk_free(sk_sdp(conn)); } /* @@ -1197,6 +1191,8 @@ error_attr: return result; } +extern struct proto sdp_sk_proto; + /* * sdp_conn_alloc - allocate a new socket, and init. */ @@ -1206,8 +1202,7 @@ struct sdp_opt *sdp_conn_alloc(int prior struct sock *sk; int result; - sk = sk_alloc(dev_root_s.proto, priority, - sizeof(struct inet_sock), dev_root_s.sock_cache); + sk = sk_alloc(dev_root_s.proto, priority, &sdp_sk_proto, 1); if (!sk) { sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%d>", dev_root_s.proto, priority); @@ -1230,23 +1225,8 @@ struct sdp_opt *sdp_conn_alloc(int prior sk->sk_state_change = sdp_inet_wake_generic; sk->sk_data_ready = sdp_inet_wake_recv; sk->sk_error_report = sdp_inet_wake_error; - /* - * Allocate must be called from process context, since QP - * create/modifies must be in that context. - */ - conn = kmem_cache_alloc(dev_root_s.conn_cache, priority); - if (!conn) { - sdp_dbg_warn(conn, "connection alloc error. <%d>", priority); - result = -ENOMEM; - goto error; - } - memset(conn, 0, sizeof(struct sdp_opt)); - /* - * The STRM interface specific data is map/cast over the TCP specific - * area of the sock. - */ - SDP_SET_CONN(sk, conn); + conn = sdp_sk(sk); SDP_CONN_ST_INIT(conn); conn->cm_id = NULL; @@ -1263,7 +1243,6 @@ struct sdp_opt *sdp_conn_alloc(int prior conn->parent = NULL; conn->pid = 0; - conn->sk = sk; conn->hashent = SDP_DEV_SK_INVALID; conn->flags = 0; conn->shutdown = 0; @@ -1369,7 +1348,7 @@ struct sdp_opt *sdp_conn_alloc(int prior sdp_dbg_warn(conn, "Error <%d> conn table insert <%d:%d>", result, dev_root_s.sk_entry, dev_root_s.sk_size); - goto error_conn; + goto error; } /* * set reference @@ -1383,8 +1362,6 @@ struct sdp_opt *sdp_conn_alloc(int prior * done */ return conn; -error_conn: - kmem_cache_free(dev_root_s.conn_cache, conn); error: sk_free(sk); return NULL; @@ -1553,7 +1530,7 @@ int sdp_proc_dump_conn_data(char *buffer continue; conn = dev_root_s.sk_array[counter]; - sk = conn->sk; + sk = sk_sdp(conn); offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_FORM, conn->hashent, @@ -2037,26 +2014,6 @@ int sdp_conn_table_init(int proto_family goto error_iocb; } - dev_root_s.conn_cache = kmem_cache_create("sdp_conn", - sizeof(struct sdp_opt), - 0, SLAB_HWCACHE_ALIGN, - NULL, NULL); - if (!dev_root_s.conn_cache) { - sdp_warn("Failed to initialize connection cache."); - result = -ENOMEM; - goto error_conn; - } - - dev_root_s.sock_cache = kmem_cache_create("sdp_sock", - sizeof(struct inet_sock), - 0, SLAB_HWCACHE_ALIGN, - NULL, NULL); - if (!dev_root_s.sock_cache) { - sdp_warn("Failed to initialize sock cache."); - result = -ENOMEM; - goto error_sock; - } - /* * start listening */ @@ -2065,7 +2022,7 @@ int sdp_conn_table_init(int proto_family if (!dev_root_s.listen_id) { sdp_warn("Failed to create listen connection identifier."); result = -ENOMEM; - goto error_cm_id; + goto error_conn; } result = ib_cm_listen(dev_root_s.listen_id, @@ -2082,10 +2039,6 @@ int sdp_conn_table_init(int proto_family return 0; error_listen: ib_destroy_cm_id(dev_root_s.listen_id); -error_cm_id: - kmem_cache_destroy(dev_root_s.sock_cache); -error_sock: - kmem_cache_destroy(dev_root_s.conn_cache); error_conn: sdp_main_iocb_cleanup(); error_iocb: @@ -2127,14 +2080,6 @@ void sdp_conn_table_clear(void) */ sdp_main_iocb_cleanup(); /* - * delete conn cache - */ - kmem_cache_destroy(dev_root_s.conn_cache); - /* - * delete sock cache - */ - kmem_cache_destroy(dev_root_s.sock_cache); - /* * stop listening */ ib_destroy_cm_id(dev_root_s.listen_id); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -707,7 +707,7 @@ static int sdp_recv_buff_iocb_pending(st */ if (!iocb->len || (!conn->src_recv && - !(conn->sk->sk_rcvlowat > iocb->post))) { + !(sk_sdp(conn)->sk_rcvlowat > iocb->post))) { /* * complete IOCB */ @@ -755,7 +755,7 @@ int sdp_recv_buff(struct sdp_opt *conn, */ if (buff->flags & SDP_BUFF_F_OOB_PEND) { conn->rcv_urg_cnt++; - sdp_inet_wake_urg(conn->sk); + sdp_inet_wake_urg(sk_sdp(conn)); } /* * loop while there are available IOCB's, break if there is no @@ -851,7 +851,7 @@ static int sdp_inet_read_cancel(struct k /* * lock the socket while we operate. */ - conn = SDP_GET_CONN(si->sock->sk); + conn = sdp_sk(si->sock->sk); sdp_conn_lock(conn); sdp_dbg_ctrl(conn, "Cancel Read IOCB. <%08x:%04x> <%08x:%04x>", @@ -997,7 +997,7 @@ static int sdp_inet_recv_urg(struct sock int result = 0; u8 value; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); if (sock_flag(sk, SOCK_URGINLINE) || !conn->rcv_urg_cnt) return -EINVAL; @@ -1079,7 +1079,7 @@ int sdp_inet_recv(struct kiocb *req, st struct sdpc_buff_q peek_queue; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "state <%08x> size <%Zu> pending <%d> falgs <%08x>", conn->state, size, conn->byte_strm, flags); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -109,16 +109,18 @@ enum sdp_mode { */ #define SDP_MSG_EVENT_TABLE_SIZE 0x20 -/* - * connection handle within a socket. - */ -#define SDP_GET_CONN(sk) \ - (*((struct sdp_opt **)&(sk)->sk_protinfo)) -#define SDP_SET_CONN(sk, conn) \ - (*((struct sdp_opt **)&(sk)->sk_protinfo) = (conn)) +static inline struct sdp_opt *sdp_sk(struct sock *sk) +{ + return (struct sdp_opt *)sk; +} + +static inline struct sock *sk_sdp(struct sdp_opt *conn) +{ + return (struct sock *)conn; +} #define SDP_CONN_SET_ERR(conn, val) \ - ((conn)->error = (conn)->sk->sk_err = (val)) + ((conn)->error = sk_sdp(conn)->sk_err = (val)) #define SDP_CONN_GET_ERR(conn) \ ((conn)->error) @@ -180,10 +182,15 @@ struct sdp_conn_lock { * SDP Connection structure. */ struct sdp_opt { + /* + * inet_sock must be first member of sdp_opt + * NOTE: this depends on inet_sock having struct sock as its + * first member + */ + struct inet_sock in; __s32 hashent; /* connection ID/hash entry */ atomic_t refcnt; /* connection reference count. */ - struct sock *sk; /* * SDP specific data */ @@ -494,7 +501,7 @@ static inline int sdp_conn_error(struct * lock, however the linux socket error, needs to be xchg'd since the * SO_ERROR getsockopt happens outside of the connection lock. */ - int error = xchg(&conn->sk->sk_err, 0); + int error = xchg(&sk_sdp(conn)->sk_err, 0); SDP_CONN_SET_ERR(conn, 0); return -error; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (working copy) @@ -88,7 +88,7 @@ int sdp_cm_pass_establish(struct sdp_opt goto error; } - sdp_inet_wake_send(conn->sk); + sdp_inet_wake_send(sk_sdp(conn)); kfree(qp_attr); return 0; @@ -262,8 +262,8 @@ static int sdp_cm_listen_lookup(struct s /* * check backlog */ - listen_sk = listen_conn->sk; - sk = conn->sk; + listen_sk = sk_sdp(listen_conn); + sk = sk_sdp(conn); if (listen_conn->backlog_cnt > listen_conn->backlog_max) { sdp_dbg_warn(listen_conn, @@ -296,13 +296,16 @@ static int sdp_cm_listen_lookup(struct s */ sk->sk_lingertime = listen_sk->sk_lingertime; sk->sk_rcvlowat = listen_sk->sk_rcvlowat; - sk->sk_debug = listen_sk->sk_debug; - sk->sk_localroute = listen_sk->sk_localroute; + if (sock_flag(listen_sk, SOCK_DBG)) + sock_set_flag(sk, SOCK_DBG); + if (sock_flag(listen_sk, SOCK_LOCALROUTE)) + sock_set_flag(sk, SOCK_LOCALROUTE); sk->sk_sndbuf = listen_sk->sk_sndbuf; sk->sk_rcvbuf = listen_sk->sk_rcvbuf; sk->sk_no_check = listen_sk->sk_no_check; sk->sk_priority = listen_sk->sk_priority; - sk->sk_rcvtstamp = listen_sk->sk_rcvtstamp; + if (sock_flag(listen_sk, SOCK_RCVTSTAMP)) + sock_set_flag(sk, SOCK_RCVTSTAMP); sk->sk_rcvtimeo = listen_sk->sk_rcvtimeo; sk->sk_sndtimeo = listen_sk->sk_sndtimeo; sk->sk_reuse = listen_sk->sk_reuse; @@ -441,7 +444,7 @@ int sdp_cm_req_handler(struct ib_cm_id * goto done; } /* - * Lock the new connection before modifyingg it into any tables. + * Lock the new connection before modifying it into any tables. */ sdp_conn_lock(conn); /* Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (revision 2663) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (working copy) @@ -198,11 +198,6 @@ struct sdev_root { * SDP wide listen */ struct ib_cm_id *listen_id; /* listen handle */ - /* - * cache's - */ - kmem_cache_t *conn_cache; - kmem_cache_t *sock_cache; }; #endif /* _SDP_DEV_H */ From tduffy at sun.com Mon Jun 20 11:09:54 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 20 Jun 2005 11:09:54 -0700 Subject: [openib-general] [PATCH] OpenSM: Set retries to 0 if RMPP In-Reply-To: <1119287511.6884.15.camel@ggreen_laptop.us.voltaire.com> References: <1118182327.4477.30.camel@localhost.localdomain> <1118182869.22316.8.camel@duffman> <1118183846.22316.19.camel@duffman> <1119287511.6884.15.camel@ggreen_laptop.us.voltaire.com> Message-ID: <1119290994.25973.11.camel@duffman> On Mon, 2005-06-20 at 13:11 -0400, Hal Rosenstock wrote: > > Nevermind. Just tried it with Solaris Nevada build 16 and it works > > too! > > What's Nevada ? Sorry, that is the code name for Solaris 11, also what OpenSolaris is based off of. > Can you retry with the latest and greatest (both OpenSM as well as the > latest mad_rmpp code in the kernel) ? Thanks. Yes, I will give that a whirl sometime this week. -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From libor at topspin.com Mon Jun 20 11:28:29 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 20 Jun 2005 11:28:29 -0700 Subject: [openib-general] Re: [PATCHv7] SDP: make SDP buildable on 2.6.12-final In-Reply-To: <1119290772.25973.9.camel@duffman>; from tduffy@sun.com on Mon, Jun 20, 2005 at 11:06:12AM -0700 References: <1119290772.25973.9.camel@duffman> Message-ID: <20050620112829.A16823@topspin.com> On Mon, Jun 20, 2005 at 11:06:12AM -0700, Tom Duffy wrote: > Here is the updated patch based off r2663 and building off of 2.6.12. > > Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c > =================================================================== > --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2663) > +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) > @@ -295,27 +295,22 @@ error: > static int sdp_inet_release(struct socket *sock) > { > struct sdp_opt *conn; > - struct sock *sk; > + struct sock *sk = sock->sk; > int result; > long timeout; > u32 flags; > > - if (!sock->sk) { > + if (!sk) { > sdp_dbg_warn(NULL, "release empty <%d:%d> flags <%08lx>", > sock->type, sock->state, sock->flags); > return 0; > } > > - sk = sock->sk; > - conn = SDP_GET_CONN(sk); > + conn = sdp_sk(sk); > > sdp_dbg_ctrl(conn, "RELEASE: linger <%d:%lu> data <%d:%d>", > sock_flag(sk, SOCK_LINGER), sk->sk_lingertime, > conn->byte_strm, conn->src_recv); > - /* > - * clear out sock, so we only do this once. > - */ > - sock->sk = NULL; > > sdp_conn_lock(conn); > conn->shutdown = SHUTDOWN_MASK; > @@ -412,6 +407,7 @@ done: > * finally drop socket reference. (socket API reference) > */ > sock_orphan(sk); > + sock->sk = NULL; > sdp_conn_unlock(conn); > sdp_conn_put(conn); > Looks good, but I do have one question, why did you move the clearing of 'struct sock' from 'struct socket' to the end of the function? Specifically if we are waiting on linger during the release I don't think we want sock to have an sk, this avoids having any other user system calls on this socket. This is done the same way in af_inet.c, the clear comes before the close/linger... -Libor From tduffy at sun.com Mon Jun 20 11:30:12 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 20 Jun 2005 11:30:12 -0700 Subject: [openib-general] Re: [PATCHv7] SDP: make SDP buildable on 2.6.12-final In-Reply-To: <20050620112829.A16823@topspin.com> References: <1119290772.25973.9.camel@duffman> <20050620112829.A16823@topspin.com> Message-ID: <1119292212.25973.15.camel@duffman> On Mon, 2005-06-20 at 11:28 -0700, Libor Michalek wrote: > Looks good, but I do have one question, why did you move the clearing > of 'struct sock' from 'struct socket' to the end of the function? > Specifically if we are waiting on linger during the release I don't > think we want sock to have an sk, this avoids having any other user > system calls on this socket. This is done the same way in af_inet.c, > the clear comes before the close/linger... I moved it recently, earlier versions of the patch didn't do that. You can move it back if needed. I was trying to trace down a close bug, and this did not fix it. So, final answer is: keep it the way it was. -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From bjordan.ics at gmail.com Mon Jun 20 12:52:11 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Mon, 20 Jun 2005 15:52:11 -0400 Subject: [openib-general] [PATCH] ipoib tune rx ring size In-Reply-To: <20050620163044.GC10881@esmail.cup.hp.com> References: <20050620151404.GW20041@mellanox.co.il> <20050620163044.GC10881@esmail.cup.hp.com> Message-ID: <78d18e2050620125227aec02b@mail.gmail.com> On 6/20/05, Grant Grundler wrote: > On Mon, Jun 20, 2005 at 06:14:04PM +0300, Michael S. Tsirkin wrote: > > Reducing the rx ring size gives me bandwidth win of about 1%. > > Michael, > Doesn't this sound counter-intuitive to you? > > Can you share which configuration (chipset, CPU, IO bus, HCA) > you measured this on? > > I've also seen this to be true with gige - but only in one odd configuration. > > thanks, > grant I'd be skeptical also. I've found this to be true in only one very contrived instance. When receiving UDP traffic on a slow host (no user level end to end flow control), you can improve receive throughput by reducing the number of outstanding receives. The CPU can be saturated receiving packets which are overflowing the socket (being discarded), and the user can be CPU starved (unable to empty the socket). In this case, you are better off having the HCA miss the packets than you are having the driver process each packet, only to discard it in the upper layers. This is not a real world situation, though. -- Bill Jordan SilverStorm Technologies From andrei.petrov at veritas.com Mon Jun 20 13:06:24 2005 From: andrei.petrov at veritas.com (Andrei Petrov) Date: Mon, 20 Jun 2005 13:06:24 -0700 Subject: [openib-general] ib_post_recv buffer re-use Message-ID: <20050620200624.GB4502@liz.veritas.com> Application I'm working on needs to receive and process messages from RQ, so I dma_map and ib_post_recv pool of buffers. Then after I receive a message I plan to reuse already mapped buffer for consequent ib_post_recv. Is that safe? Should I worry about dma syncronisation? Thanks, From johann at pathscale.com Mon Jun 20 13:22:18 2005 From: johann at pathscale.com (Johann George) Date: Mon, 20 Jun 2005 13:22:18 -0700 Subject: [openib-general] SDP on 2.6.12 Message-ID: <20050620202218.GA28621@cuprite.internal.keyresearch.com> SDP does not compile on the 2.6.12 kernel. It references sk_debug, sk_localroute and sk_rcvtstamp which are no longer defined in sock.h. (Using SVN 2663). Johann From mst at mellanox.co.il Mon Jun 20 13:23:30 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 20 Jun 2005 23:23:30 +0300 Subject: [openib-general] Re: [PATCH] ipoib tune rx ring size In-Reply-To: <78d18e2050620125227aec02b@mail.gmail.com> References: <20050620151404.GW20041@mellanox.co.il> <20050620163044.GC10881@esmail.cup.hp.com> <78d18e2050620125227aec02b@mail.gmail.com> Message-ID: <20050620202330.GA10215@mellanox.co.il> Quoting r. William Jordan : > Subject: Re: [PATCH] ipoib tune rx ring size > > On 6/20/05, Grant Grundler wrote: > > On Mon, Jun 20, 2005 at 06:14:04PM +0300, Michael S. Tsirkin wrote: > > > Reducing the rx ring size gives me bandwidth win of about 1%. > > > > Michael, > > Doesn't this sound counter-intuitive to you? The explanation could be RX livelock 4x HCA is already fast enough to saturate the CPU. > > Can you share which configuration (chipset, CPU, IO bus, HCA) > > you measured this on? > > > > I've also seen this to be true with gige - but only in one odd configuration. > > I'd be skeptical also. I've found this to be true in only one very > contrived instance. > Well, I see it in practice, and its easy enough to reproduce even with simplest netperf benchmark. I have two back-to-back systems. Each has two xeons 3.4G , 1Mb cache in x86_64 mode. The systems are intel nocona, HT disabled, with arbel in tavor mode. Grant, are you willing to test the patch and post what effect if any this has on netperf? -- MST From tduffy at sun.com Mon Jun 20 13:53:43 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 20 Jun 2005 13:53:43 -0700 Subject: [openib-general] SDP on 2.6.12 In-Reply-To: <20050620202218.GA28621@cuprite.internal.keyresearch.com> References: <20050620202218.GA28621@cuprite.internal.keyresearch.com> Message-ID: <1119300823.25973.19.camel@duffman> On Mon, 2005-06-20 at 13:22 -0700, Johann George wrote: > SDP does not compile on the 2.6.12 kernel. It references sk_debug, > sk_localroute and sk_rcvtstamp which are no longer defined in sock.h. > (Using SVN 2663). Yup. Please see my patch. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From iod00d at hp.com Mon Jun 20 14:04:37 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 20 Jun 2005 14:04:37 -0700 Subject: [openib-general] [PATCH] ipoib tune rx ring size In-Reply-To: <78d18e2050620125227aec02b@mail.gmail.com> References: <20050620151404.GW20041@mellanox.co.il> <20050620163044.GC10881@esmail.cup.hp.com> <78d18e2050620125227aec02b@mail.gmail.com> Message-ID: <20050620210437.GH10881@esmail.cup.hp.com> On Mon, Jun 20, 2005 at 03:52:11PM -0400, William Jordan wrote: > When receiving UDP traffic on a slow host (no user level end to end > flow control), you can improve receive throughput by reducing the > number of outstanding receives. The CPU can be saturated receiving > packets which are overflowing the socket (being discarded), and the > user can be CPU starved (unable to empty the socket). Hrm, I thought Linux's NAPI avoids this problem. Jamal gave an excellent talk on this issue a few years ago: http://lwn.net/2000/0928/a/fast-forwarding.php3 I was thinking of a HW config issue. But I want to hear what Michael has to say before tainting the waters with other issues. > This is not a real world situation, though. It's very real. That's what DoS attacks are about. It just not expected on a private network with trusted hosts. grant From iod00d at hp.com Mon Jun 20 14:19:16 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 20 Jun 2005 14:19:16 -0700 Subject: [openib-general] ib_post_recv buffer re-use In-Reply-To: <20050620200624.GB4502@liz.veritas.com> References: <20050620200624.GB4502@liz.veritas.com> Message-ID: <20050620211916.GJ10881@esmail.cup.hp.com> On Mon, Jun 20, 2005 at 01:06:24PM -0700, Andrei Petrov wrote: > Application I'm working on needs to receive and process messages > from RQ, so I dma_map and ib_post_recv pool of buffers. > Then after I receive a message I plan to reuse already mapped > buffer for consequent ib_post_recv. Is that safe? Should I worry > about dma syncronisation? Can you define "application" more clearly? Is this an in-kernel "application" or user space application? Can you be more specific about the life cycles of the buffers? Which interfaces allocates them and which code "consumes" them? I suggest reading Documentation/DMA-API.txt in a 2.6.11 or 2.6.12 source tree to get an idea of how DMA is managed by linux kernel. grant From roland at topspin.com Mon Jun 20 14:19:34 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 20 Jun 2005 14:19:34 -0700 Subject: [openib-general] ib_post_recv buffer re-use In-Reply-To: <20050620200624.GB4502@liz.veritas.com> (Andrei Petrov's message of "Mon, 20 Jun 2005 13:06:24 -0700") References: <20050620200624.GB4502@liz.veritas.com> Message-ID: <52ekawybdl.fsf@topspin.com> Andrei> Application I'm working on needs to receive and process Andrei> messages from RQ, so I dma_map and ib_post_recv pool of Andrei> buffers. Then after I receive a message I plan to reuse Andrei> already mapped buffer for consequent ib_post_recv. Is that Andrei> safe? Should I worry about dma syncronisation? In general it's not safe for the CPU to touch DMA buffers without unmapping them first. However you could look at the dma_sync_xxx API to see if that fits what you're trying to do. - R. From mst at mellanox.co.il Mon Jun 20 14:29:00 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 21 Jun 2005 00:29:00 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050620152308.GA10674@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050615150440.GB20041@mellanox.co.il> <20050615173058.GB4379@esmail.cup.hp.com> <20050615215334.GA22691@mellanox.co.il> <20050620152308.GA10674@esmail.cup.hp.com> Message-ID: <20050620212900.GB5228@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [PATCH] rdma_lat-09 and results > > On Thu, Jun 16, 2005 at 12:53:34AM +0300, Michael S. Tsirkin wrote: > > Quoting r. Grant Grundler : > > > 2) I working towards making the server side a daemon. > > > > I dont see this necessarily as an improvement. > > I'm happy running server and client with ssh from a script > > with multiple parameters, this works well for me - actually better > > than netserver. > > Michael, > You are right that I could run the test as a normal user. > I don't recall why (if I ever had a reason) I was running > the test as root. > > Could you commit your script along side the rdma_lat.c/rdma_bw.c sources? > > thanks, > grant > OK I put it in the repository (it was a one liner, but its bigger now that I put in some comments). There's a bit of a hack in that I sleep 2 seconds after launching the server - to give it time to start listening on a port. This could be solved some way if its bothering someone. -- MST From tduffy at sun.com Mon Jun 20 14:45:48 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 20 Jun 2005 14:45:48 -0700 Subject: [openib-general] [PATCH] OpenSM: Set retries to 0 if RMPP In-Reply-To: <1118186678.4477.239.camel@localhost.localdomain> References: <1118182327.4477.30.camel@localhost.localdomain> <1118182869.22316.8.camel@duffman> <1118183846.22316.19.camel@duffman> <1118186678.4477.239.camel@localhost.localdomain> Message-ID: <1119303948.25973.29.camel@duffman> On Tue, 2005-06-07 at 19:27 -0400, Hal Rosenstock wrote: > I didn't try broadcast but the limited broadcast (192.168.0.255_ and it > worked. Does it for you ? I will try this next time. Not sure this is an > OpenSM issue. This wasn't an opensm issue. I have apparently gotten used to Linux where it will automagically set the netmask to 0xffffff00 if you are using a class c network. Solaris defaults to 0xff000000, so it was not responding to the broadcast pings. If I set the netmask properly, it works. Yeah! -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From andrei.petrov at veritas.com Mon Jun 20 14:50:05 2005 From: andrei.petrov at veritas.com (Andrei Petrov) Date: Mon, 20 Jun 2005 14:50:05 -0700 Subject: [openib-general] ib_post_recv buffer re-use In-Reply-To: <52ekawybdl.fsf@topspin.com> References: <20050620200624.GB4502@liz.veritas.com> <52ekawybdl.fsf@topspin.com> Message-ID: <20050620215005.GD4502@liz.veritas.com> On Mon, Jun 20, 2005 at 02:19:34PM -0700, Roland Dreier wrote: > Andrei> Application I'm working on needs to receive and process > Andrei> messages from RQ, so I dma_map and ib_post_recv pool of > Andrei> buffers. Then after I receive a message I plan to reuse > Andrei> already mapped buffer for consequent ib_post_recv. Is that > Andrei> safe? Should I worry about dma syncronisation? > > In general it's not safe for the CPU to touch DMA buffers without > unmapping them first. However you could look at the dma_sync_xxx API > to see if that fits what you're trying to do. > Oh, I see, as infiniband layer doesn't control how app sets up dma then it's up to app to handle it accordingly. Yes, dma_sync_xxx should do then. Thanks, From andrei.petrov at veritas.com Mon Jun 20 14:56:14 2005 From: andrei.petrov at veritas.com (Andrei Petrov) Date: Mon, 20 Jun 2005 14:56:14 -0700 Subject: [openib-general] ib_post_recv buffer re-use In-Reply-To: <20050620211916.GJ10881@esmail.cup.hp.com> References: <20050620200624.GB4502@liz.veritas.com> <20050620211916.GJ10881@esmail.cup.hp.com> Message-ID: <20050620215614.GE4502@liz.veritas.com> On Mon, Jun 20, 2005 at 02:19:16PM -0700, Grant Grundler wrote: > On Mon, Jun 20, 2005 at 01:06:24PM -0700, Andrei Petrov wrote: > > Application I'm working on needs to receive and process messages > > from RQ, so I dma_map and ib_post_recv pool of buffers. > > Then after I receive a message I plan to reuse already mapped > > buffer for consequent ib_post_recv. Is that safe? Should I worry > > about dma syncronisation? > > Can you define "application" more clearly? > Is this an in-kernel "application" or user space application? > It's kernel module which uses IB verbs API as in gen2. > Can you be more specific about the life cycles of the buffers? > Which interfaces allocates them and which code "consumes" them? > I'd say map it once, use (ib_post_recv) during entire application life, and application allocates and consumes them internally. > I suggest reading Documentation/DMA-API.txt in a 2.6.11 or 2.6.12 > source tree to get an idea of how DMA is managed by linux kernel. > Yeah, I got my answer there, thanks. > grant --Andrei From andrei.petrov at veritas.com Mon Jun 20 17:55:06 2005 From: andrei.petrov at veritas.com (Andrei Petrov) Date: Mon, 20 Jun 2005 17:55:06 -0700 Subject: [openib-general] CQ completion handler context Message-ID: <20050621005506.GF4502@liz.veritas.com> In what context CQ completion handler is called? I'm talking about kernel module using ib verbs. Thanks, From roland at topspin.com Mon Jun 20 19:22:40 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 20 Jun 2005 19:22:40 -0700 Subject: [openib-general] CQ completion handler context In-Reply-To: <20050621005506.GF4502@liz.veritas.com> (Andrei Petrov's message of "Mon, 20 Jun 2005 17:55:06 -0700") References: <20050621005506.GF4502@liz.veritas.com> Message-ID: <523brcxxcf.fsf@topspin.com> Andrei> In what context CQ completion handler is called? I'm Andrei> talking about kernel module using ib verbs. In general you should assume it is called from an unknown context. The current mthca driver calls it from interrupt context, but future drivers might use softirq or even process context. - R. From kjreilly at us.ibm.com Mon Jun 20 19:23:48 2005 From: kjreilly at us.ibm.com (Kevin Reilly) Date: Mon, 20 Jun 2005 22:23:48 -0400 Subject: [openib-general] mapping between IP address and device name Message-ID: Maybe somebody could help me understand the proper way to map between an IP address assigned to a port to the "device name" and "port number" in the gen2 architecture. If I have an IP address can I map it to a name that i get back from ibv_get_device_name() or pass to ibv_open_device(). Here is the problem. Let's say there is a subsystem that used heartbeating over IP interfaces to ascertain the adapters and ports state, up or down. The data actively maintained be this heartbeating subsystem is used by a job scheduler that desires to only schedule tasks on HCAs that are UP and have active ports. The job scheduler does not want to launch jobs on adapters that are broken, not connected to a switch or not configured. The job scheduler needs a way to map an IP address into a name and port number sitable to be passed to the job to use when ibv_open_device() is called or to match the output of ibv_get_device_name(). Kevin J. Reilly STSM, HPC Architecture -Federation/HPS Chief Engineer -HPC interconnect architect (office) 845-433-7976 (tieline) 8-293-7976 From roland at topspin.com Mon Jun 20 20:40:47 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 20 Jun 2005 20:40:47 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: (Kevin Reilly's message of "Mon, 20 Jun 2005 22:23:48 -0400") References: Message-ID: <52vf48wf5s.fsf@topspin.com> Kevin> Maybe somebody could help me understand the proper way to Kevin> map between an IP address assigned to a port to the "device Kevin> name" and "port number" in the gen2 architecture. If I Kevin> have an IP address can I map it to a name that i get back Kevin> from ibv_get_device_name() or pass to ibv_open_device(). I think the question you have posed is not really the one you want to answer. After all, even in a world without IB, there is no way to map a local IP address uniquely to a network interface or ethernet port -- it is perfectly valid to have a host with multiple ethernet interfaces, all of which have been configured with the same IP address. It _is_ valid to ask, for a given remote IP address, which network interface will be used to reach it based on the local host's routing configuration. In the IB world, we can hope that the network interface chosen by the route table is an IPoIB device. From the IPoIB device, we can work backwards to get a local IB port. For example the network device's hardware address contains the associated port GID, and by searching the GID tables of all the known IB devices, we can find which port has that GID. There was some discussion a while ago about an "IB address translation" service proposed by the Voltaire crew, which would encapsulate some of this. However, we didn't make much progress towards a good API design, and the discussion fizzled out. - R. From libor at topspin.com Mon Jun 20 22:07:27 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 20 Jun 2005 22:07:27 -0700 Subject: [openib-general] Re: [PATCHv7] SDP: make SDP buildable on 2.6.12-final In-Reply-To: <1119290772.25973.9.camel@duffman>; from tduffy@sun.com on Mon, Jun 20, 2005 at 11:06:12AM -0700 References: <1119290772.25973.9.camel@duffman> Message-ID: <20050620220727.A17704@topspin.com> On Mon, Jun 20, 2005 at 11:06:12AM -0700, Tom Duffy wrote: > Here is the updated patch based off r2663 and building off of 2.6.12. > > Signed-off-by: Tom Duffy Thanks a lot Tom for tracking the changes this whole time. Committed revision 2665. -Libor From libor at topspin.com Mon Jun 20 22:14:25 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 20 Jun 2005 22:14:25 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52vf48wf5s.fsf@topspin.com>; from roland@topspin.com on Mon, Jun 20, 2005 at 08:40:47PM -0700 References: <52vf48wf5s.fsf@topspin.com> Message-ID: <20050620221425.B17704@topspin.com> On Mon, Jun 20, 2005 at 08:40:47PM -0700, Roland Dreier wrote: > Kevin> Maybe somebody could help me understand the proper way to > Kevin> map between an IP address assigned to a port to the "device > Kevin> name" and "port number" in the gen2 architecture. If I > Kevin> have an IP address can I map it to a name that i get back > Kevin> from ibv_get_device_name() or pass to ibv_open_device(). > > It _is_ valid to ask, for a given remote IP address, which network > interface will be used to reach it based on the local host's routing > configuration. > > In the IB world, we can hope that the network interface chosen by the > route table is an IPoIB device. From the IPoIB device, we can work > backwards to get a local IB port. For example the network device's > hardware address contains the associated port GID, and by searching > the GID tables of all the known IB devices, we can find which port has > that GID. > > There was some discussion a while ago about an "IB address > translation" service proposed by the Voltaire crew, which would > encapsulate some of this. However, we didn't make much progress > towards a good API design, and the discussion fizzled out. Also, if you want to look at an example of how this is done using the IPoIB network device and Linux routing code you can look here: src/linux-kernel/infiniband/ulp/sdp/sdp_link.[ch] The SDP code needed this functionality, so I put together a fairly simple version that's private to SDP, as MST pointed out locking is still missing from the code. Eventually this type of code should become a central service, since there's a number of modules that would want this info. -Libor From mst at mellanox.co.il Mon Jun 20 23:21:32 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 21 Jun 2005 09:21:32 +0300 Subject: [openib-general] [PATCH] SDP: use container_of in place of cast to sdp_opt In-Reply-To: <20050620220727.A17704@topspin.com> References: <1119290772.25973.9.camel@duffman> <20050620220727.A17704@topspin.com> Message-ID: <20050621062132.GC18896@mellanox.co.il> Here's a patch to replace explicit cast to sdp_opt with container_of, removing assumptions on structure layout. As a side note, wouldnt sk_to_sdp and sdp_to_sk be better names for sdp_sk and sk_sdp? --- Use container_of in place of cast to sdp_opt. Signed-off-by: Michael S. Tsirkin Index: ulp/sdp/sdp_actv.c =================================================================== --- ulp/sdp/sdp_actv.c (revision 2665) +++ ulp/sdp/sdp_actv.c (working copy) @@ -343,7 +343,7 @@ { struct ib_cm_req_param param; struct sdp_msg_hello *hello_msg; - struct sdp_opt *conn = (struct sdp_opt *) arg; + struct sdp_opt *conn = arg; struct sdpc_buff *buff; int result = 0; /* Index: ulp/sdp/sdp_conn.h =================================================================== --- ulp/sdp/sdp_conn.h (revision 2665) +++ ulp/sdp/sdp_conn.h (working copy) @@ -109,16 +109,6 @@ */ #define SDP_MSG_EVENT_TABLE_SIZE 0x20 -static inline struct sdp_opt *sdp_sk(struct sock *sk) -{ - return (struct sdp_opt *)sk; -} - -static inline struct sock *sk_sdp(struct sdp_opt *conn) -{ - return (struct sock *)conn; -} - #define SDP_CONN_SET_ERR(conn, val) \ ((conn)->error = sk_sdp(conn)->sk_err = (val)) #define SDP_CONN_GET_ERR(conn) \ @@ -182,11 +172,6 @@ * SDP Connection structure. */ struct sdp_opt { - /* - * inet_sock must be first member of sdp_opt - * NOTE: this depends on inet_sock having struct sock as its - * first member - */ struct inet_sock in; __s32 hashent; /* connection ID/hash entry */ atomic_t refcnt; /* connection reference count. */ @@ -386,6 +371,16 @@ #endif }; +static inline struct sdp_opt *sdp_sk(struct sock *sk) +{ + return container_of(sk, struct sdp_opt, in.sk); +} + +static inline struct sock *sk_sdp(struct sdp_opt *conn) +{ + return &conn->in.sk; +} + #define SDP_WRAP_GT(x, y) ((signed int)((x) - (y)) > 0) #define SDP_WRAP_LT(x, y) ((signed int)((x) - (y)) < 0) #define SDP_WRAP_GTE(x, y) ((signed int)((x) - (y)) >= 0) -- MST From iod00d at hp.com Tue Jun 21 00:45:38 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 21 Jun 2005 00:45:38 -0700 Subject: [openib-general] Re: [PATCH] ipoib tune rx ring size In-Reply-To: <20050620202330.GA10215@mellanox.co.il> References: <20050620151404.GW20041@mellanox.co.il> <20050620163044.GC10881@esmail.cup.hp.com> <78d18e2050620125227aec02b@mail.gmail.com> <20050620202330.GA10215@mellanox.co.il> Message-ID: <20050621074538.GC13221@esmail.cup.hp.com> On Mon, Jun 20, 2005 at 11:23:30PM +0300, Michael S. Tsirkin wrote: > Grant, are you willing to test the patch and post what effect if any this has > on netperf? Conclusion: RX_RING @ 64 degrades netperf by about 1%. I'm certain my runs are CPU bound. I'm not certain that's true for your config. Can you post netperf output? Original SVN r2577 code running netperf 2.4.0: (Sorry stats are busted on this verion of netperf - I'm tracking down that bug) /usr/local/bin/netperf -l 60 -H 10.0.0.30 -t TCP_STREAM -T 1 -c 1024 -C 1024 -- -m 131072 -s 131072 -S 131072 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 249856 249856 131072 60.00 2728.02 34.45 42.64 2.069 2.561 With IPOIB_RX_RING_SIZE at 64: 249856 249856 131072 60.00 2702.37 34.39 42.46 2.085 2.574 249856 249856 131072 60.00 2700.92 34.35 42.36 2.084 2.569 249856 249856 131072 60.00 2701.02 34.61 42.36 2.100 2.569 I'm satisfied the runs are fairly consistent. More caveats: Original run had CPU pinned to same CPU that was taking interrupts for both ends of the connection. I just happened to get the same CPU on "netperf" client side and was pinning the netserver ("-T 1") side. (If I didn't, I'd see ~3300-3500 Mb/s.) However, binding the tasks to the non-interrupt CPU ("-T 0,0") results in essentially no change in perf. With RX_RING @ 128: 249856 249856 131072 60.00 3585.08 61.31 70.67 2.802 3.230 249856 249856 131072 60.00 3577.81 60.28 70.74 2.760 3.239 249856 249856 131072 60.00 3577.27 61.16 70.38 2.801 3.223 With RX_RING @ 64: 249856 249856 131072 60.00 3590.58 60.94 70.48 2.781 3.216 249856 249856 131072 60.00 3569.83 61.25 70.47 2.811 3.234 249856 249856 131072 60.00 3564.18 60.32 70.48 2.773 3.240 hth, grant From mst at mellanox.co.il Tue Jun 21 01:27:58 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 21 Jun 2005 11:27:58 +0300 Subject: [openib-general] Re: [PATCH] ipoib tune rx ring size In-Reply-To: <20050621074538.GC13221@esmail.cup.hp.com> References: <20050620151404.GW20041@mellanox.co.il> <20050620163044.GC10881@esmail.cup.hp.com> <78d18e2050620125227aec02b@mail.gmail.com> <20050620202330.GA10215@mellanox.co.il> <20050621074538.GC13221@esmail.cup.hp.com> Message-ID: <20050621082758.GY20041@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [PATCH] ipoib tune rx ring size > > However, binding the tasks to the non-interrupt CPU ("-T 0,0") results > in essentially no change in perf. > Interesting. In theory, this could be caused by tx_lock being taken on completion path, in effect serializing tx with interrupt handler. Grant, could you try the patch below, and see what happens? Its probably interesting to try this with both 128 and 64 rx ring size. Index: ulp/ipoib/ipoib_main.c =================================================================== --- ulp/ipoib/ipoib_main.c (revision 2656) +++ ulp/ipoib/ipoib_main.c (working copy) @@ -837,6 +837,7 @@ static void ipoib_setup(struct net_devic spin_lock_init(&priv->lock); spin_lock_init(&priv->tx_lock); + spin_lock_init(&priv->queue_lock); init_MUTEX(&priv->mcast_mutex); init_MUTEX(&priv->vlan_mutex); Index: ulp/ipoib/ipoib.h =================================================================== --- ulp/ipoib/ipoib.h (revision 2656) +++ ulp/ipoib/ipoib.h (working copy) @@ -107,8 +107,10 @@ struct ipoib_buf { /* * Device private locking: tx_lock protects members used in TX fast * path (and we use LLTX so upper layers don't do extra locking). - * lock protects everything else. lock nests inside of tx_lock (ie - * tx_lock must be acquired first if needed). + * queue_lock protects the tx_tail value and netif queue stopped status. + * lock protects everything else. + * queue_lock and lock nest inside of tx_lock + * (ie tx_lock must be acquired first if needed). */ struct ipoib_dev_priv { spinlock_t lock; @@ -158,6 +160,8 @@ struct ipoib_dev_priv { struct ib_sge tx_sge; struct ib_send_wr tx_wr; + spinlock_t queue_lock; + struct ib_wc ibwc[IPOIB_NUM_WC]; struct list_head dead_ahs; Index: ulp/ipoib/ipoib_ib.c =================================================================== --- ulp/ipoib/ipoib_ib.c (revision 2656) +++ ulp/ipoib/ipoib_ib.c (working copy) @@ -229,6 +229,7 @@ static void ipoib_ib_handle_wc(struct ne } else { struct ipoib_buf *tx_req; unsigned long flags; + int stopped; if (wr_id >= IPOIB_TX_RING_SIZE) { ipoib_warn(priv, "completion event with wrid %d (> %d)\n", @@ -250,12 +251,16 @@ static void ipoib_ib_handle_wc(struct ne dev_kfree_skb_any(tx_req->skb); - spin_lock_irqsave(&priv->tx_lock, flags); + spin_lock_irqsave(&priv->queue_lock, flags); ++priv->tx_tail; - if (netif_queue_stopped(dev) && - priv->tx_head - priv->tx_tail <= IPOIB_TX_RING_SIZE / 2) - netif_wake_queue(dev); - spin_unlock_irqrestore(&priv->tx_lock, flags); + stopped = netif_queue_stopped(dev); + spin_unlock_irqrestore(&priv->queue_lock, flags); + if (stopped) { + spin_lock_irqsave(&priv->tx_lock, flags); + if (priv->tx_head - priv->tx_tail <= IPOIB_TX_RING_SIZE / 2) + netif_wake_queue(dev); + spin_unlock_irqrestore(&priv->tx_lock, flags); + } if (wc->status != IB_WC_SUCCESS && wc->status != IB_WC_WR_FLUSH_ERR) @@ -336,14 +341,20 @@ void ipoib_send(struct net_device *dev, DMA_TO_DEVICE); dev_kfree_skb_any(skb); } else { + unsigned long flags; dev->trans_start = jiffies; address->last_send = priv->tx_head; ++priv->tx_head; if (priv->tx_head - priv->tx_tail == IPOIB_TX_RING_SIZE) { - ipoib_dbg(priv, "TX ring full, stopping kernel net queue\n"); - netif_stop_queue(dev); + spin_lock_irqsave(&priv->queue_lock, flags); + + if (priv->tx_head - priv->tx_tail == IPOIB_TX_RING_SIZE) { + ipoib_dbg(priv, "TX ring full, stopping kernel net queue\n"); + netif_stop_queue(dev); + } + spin_unlock_irqrestore(&priv->queue_lock, flags); } } } -- MST From itamar at mellanox.co.il Tue Jun 21 04:51:34 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 21 Jun 2005 14:51:34 +0300 Subject: [openib-general] kdapl locking problem Message-ID: <20050621115134.GA17046@mellanox.co.il> >Jun 20 09:13:50 localhost kernel: dapl_cm_active_cb_handler 138 event = 9 >Jun 20 09:13:50 localhost kernel: dapl_evd_connection_callback 760 event=16389 >Jun 20 09:13:50 localhost kernel: dapl_evd_connection_callback 798 ep = c0a17bf8 >Jun 20 09:13:50 localhost kernel: dapl_ib_disconnect_clean 579 ep=c0a17bf8 >Jun 20 09:13:50 localhost kernel: dapl_ib_disconnect 538 ep=c0a17bf8 flags=0 >Jun 20 09:13:50 localhost kernel: dapl_destroy_cm_id 63 >Jun 20 09:13:50 localhost kernel: dapl_modify_qp_state_to_error 300 >Jun 20 09:13:50 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_ep.c:1111: spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c0a17c08) already locked by drivers/infiniband/ulp/dat-provider/dapl_evd.c/759 >Jun 20 09:13:51 localhost kernel: dapl_modify_qp_state_to_error 305 >Jun 20 09:13:51 localhost kernel: dapl_evd_connection_callback 800 ep = c0a17bf8 >Jun 20 09:13:51 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_evd.c:802: spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c0a17c08) not locked >Jun 20 09:13:51 localhost kernel: dapl_cr_callback 434 event=16389 >Jun 20 09:13:51 localhost kernel: dapl_cr_callback 512 ep = c9821bf8 >Jun 20 09:13:51 localhost kernel: dapl_ib_disconnect_clean 579 ep=c982m: DREQ rcvd Hi Hal, I think i have found the problem. It is not legal to call ib_modify_qp() inside spin_lock. This function can sleep. Please try the patch below and if it works please tell james to ci it. fix locking problem in cm callback functions Signed-off-by: Itamar Rabenstein Index: dapl_openib_util.h =================================================================== --- dapl_openib_util.h (revision 2665) +++ dapl_openib_util.h (working copy) @@ -125,7 +125,7 @@ void dapl_ib_reinit_ep(struct dapl_ep *ep); -void dapl_ib_disconnect_clean(struct dapl_ep *ep, boolean_t passive); +void dapl_ib_disconnect_clean(struct dapl_ep *ep); u32 dapl_ib_get_async_event(struct ib_event *cause, enum dat_event_number *async_event); Index: dapl_cr.c =================================================================== --- dapl_cr.c (revision 2665) +++ dapl_cr.c (working copy) @@ -479,8 +479,7 @@ /* If someone pulled the plug on the EP or connection, * just exit */ - spin_unlock_irqrestore(&ep->common.lock, - ep->common.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); status = DAT_SUCCESS; /* Set evd = NULL so we don't generate an event below */ evd = NULL; @@ -504,36 +503,23 @@ /* The disconnect has already occurred, we are now * cleaned up and ready to exit */ - spin_unlock_irqrestore(&ep->common.lock, - ep->common.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); return; } ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, FALSE); spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); + dapl_ib_disconnect_clean(ep); break; case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: - /* - * After posting an accept the requesting node has - * stopped talking. - */ + case DAT_CONNECTION_EVENT_BROKEN: spin_lock_irqsave(&ep->common.lock, ep->common.flags); ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, FALSE); spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - + dapl_ib_disconnect_clean(ep); break; - case DAT_CONNECTION_EVENT_BROKEN: - spin_lock_irqsave(&ep->common.lock, ep->common.flags); - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, FALSE); - spin_unlock_irqrestore(&ep->common.lock, - ep->common.flags); - - break; default: evd = NULL; dapl_os_assert(0); /* shouldn't happen */ Index: dapl_evd.c =================================================================== --- dapl_evd.c (revision 2665) +++ dapl_evd.c (working copy) @@ -760,7 +760,6 @@ switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: - { /* * If we don't have an EP at this point we are very screwed up */ @@ -784,65 +783,28 @@ private_data_size); } spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - break; - } case DAT_CONNECTION_EVENT_DISCONNECTED: - { - /* - * EP is now fully disconnected; initiate any post processing - * to reset the underlying QP and get the EP ready for - * another connection - */ ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->common.lock, - ep->common.flags); - + spin_unlock_irqrestore(&ep->common.lock,ep->common.flags); + dapl_ib_disconnect_clean(ep); break; - } case DAT_CONNECTION_EVENT_PEER_REJECTED: - { - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - - break; - } case DAT_CONNECTION_EVENT_UNREACHABLE: - { - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - - break; - } case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: - { - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - - break; - } case DAT_CONNECTION_EVENT_BROKEN: - { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, FALSE); spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - + dapl_ib_disconnect_clean(ep); break; - } case DAT_CONNECTION_REQUEST_EVENT: default: - { spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); evd = NULL; dapl_os_assert(0); /* shouldn't happen */ break; } - } /* * Post the event Index: dapl_openib_cm.c =================================================================== --- dapl_openib_cm.c (revision 2665) +++ dapl_openib_cm.c (working copy) @@ -562,13 +562,12 @@ * void * */ -void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t active) +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr) { int status; dapl_dbg_log(DAPL_DBG_TYPE_CM, - " >>> dapl_ib_disconnect_clean: EP: %p active %d\n", - ep_ptr, active); + " >>> dapl_ib_disconnect_clean: EP: %p \n", ep_ptr); /* * Clean up outstanding connection state -- Itamar From halr at voltaire.com Tue Jun 21 06:05:29 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 21 Jun 2005 09:05:29 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: References: Message-ID: <1119358993.4476.2371.camel@localhost.localdomain> Hi Kevin, On Mon, 2005-06-20 at 22:23, Kevin Reilly wrote: > Maybe somebody could help me understand the proper way to map between an IP > address assigned to a port to the > "device name" and "port number" in the gen2 architecture. If I have an IP > address can I map it to a name that i get back > from ibv_get_device_name() or pass to ibv_open_device(). > Here is the problem. Let's say there is a subsystem that used > heartbeating over IP interfaces to ascertain the adapters > and ports state, up or down. The data actively maintained be this > heartbeating subsystem is used by a job scheduler that > desires to only schedule tasks on HCAs that are UP and have active ports. > The job scheduler does not want to launch jobs > on adapters that are broken, not connected to a switch or not configured. > The job scheduler needs a way to map an IP address into a name and > port number sitable to be passed to the job to > use when ibv_open_device() is called or to match the output of > ibv_get_device_name(). With kernel IB Address Translation (IBAT), you can resolve an IP address (remote or local) and obtain the device and port are obtained from the ib_route structure. ib_at_route_by_ip: * ib_at_route_by_ip - asynchronously resolve ip route to ib route * @dst_ip: destination ip * @src_ip: source ip - optional * @tos: ip type of service * @flags: ib_at_route_flags * @ib_route: out structure * @async_comp: asynchronous callback structure - optional * * Resolve the specified dst_ip to a &struct ib_route structure. * src_ip can be provided to force specific output interface. * flags can be used to select resolving method; currently IB-ARP or ATS. and then: * ib_at_paths_by_route - asynchronously resolve ib route to ib path records * @ib_route: ib route to resolve * @mpath_type: ib_at_multipathing_type * @path_arr: SA path record array - out * @npath: maximal number of paths to return * @async_comp: asynchronous callback structure - optional * * Resolve the specified ib_route to a SA path record array. * Number of returned paths will not exceed npath. * Multipathing type may be used to obtain redundant paths for APM, * other failover schemes, bandwidth aggregation or source based routing. * Note that multipathing request is meaningless unless npath is greater than 1. * * Returned ib_route structure includes the recommended pkey and qos_tag for * this route. IBAT is not currently in the OpenIB trunk. See users/jlentini/linux-kernel/patches/ib_at.h or branches/shaharf-ibat/src/linux-kernel/infiniband/include/ib_at.h for more details. -- Hal > Kevin J. Reilly > STSM, HPC Architecture > -Federation/HPS Chief Engineer > -HPC interconnect architect > (office) 845-433-7976 (tieline) 8-293-7976 > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From halr at voltaire.com Tue Jun 21 06:08:43 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 21 Jun 2005 09:08:43 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52vf48wf5s.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> Message-ID: <1119359005.4476.2373.camel@localhost.localdomain> On Mon, 2005-06-20 at 23:40, Roland Dreier wrote: > There was some discussion a while ago about an "IB address > translation" service proposed by the Voltaire crew, which would > encapsulate some of this. However, we didn't make much progress > towards a good API design, and the discussion fizzled out. IBAT is currently being used in conjunction with kdapl. There is further work to incorporate ATS (service records) going on currently as well. Perhaps it is time to restart the IBAT API discussion as to what needs to change to make it acceptable for the mainstream. -- Hal From halr at voltaire.com Tue Jun 21 06:11:56 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 21 Jun 2005 09:11:56 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <20050620221425.B17704@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <20050620221425.B17704@topspin.com> Message-ID: <1119359128.4476.2377.camel@localhost.localdomain> On Tue, 2005-06-21 at 01:14, Libor Michalek wrote: > On Mon, Jun 20, 2005 at 08:40:47PM -0700, Roland Dreier wrote: > > Kevin> Maybe somebody could help me understand the proper way to > > Kevin> map between an IP address assigned to a port to the "device > > Kevin> name" and "port number" in the gen2 architecture. If I > > Kevin> have an IP address can I map it to a name that i get back > > Kevin> from ibv_get_device_name() or pass to ibv_open_device(). > > > > It _is_ valid to ask, for a given remote IP address, which network > > interface will be used to reach it based on the local host's routing > > configuration. > > > > In the IB world, we can hope that the network interface chosen by the > > route table is an IPoIB device. From the IPoIB device, we can work > > backwards to get a local IB port. For example the network device's > > hardware address contains the associated port GID, and by searching > > the GID tables of all the known IB devices, we can find which port has > > that GID. > > > > There was some discussion a while ago about an "IB address > > translation" service proposed by the Voltaire crew, which would > > encapsulate some of this. However, we didn't make much progress > > towards a good API design, and the discussion fizzled out. > > Also, if you want to look at an example of how this is done using the > IPoIB network device and Linux routing code you can look here: > > src/linux-kernel/infiniband/ulp/sdp/sdp_link.[ch] > > The SDP code needed this functionality, so I put together a fairly simple > version that's private to SDP, as MST pointed out locking is still missing > from the code. There is similar code in gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/at.c > Eventually this type of code should become a central > service, since there's a number of modules that would want this info. This is exactly what IBAT is intended to be. -- Hal From halr at voltaire.com Tue Jun 21 08:13:07 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 21 Jun 2005 11:13:07 -0400 Subject: [openib-general] page allocation failure with kdapl/mthca Message-ID: <1119366787.4477.26.camel@localhost.localdomain> Hi Roland, With the latest mthca changes (this test worked prior to these changes), I get the following running kdapl in "loopback": Jun 21 08:56:09 localhost kernel: DT_Mdep_Thread_: page allocation failure. order:0, mode:0x20 Jun 21 08:56:09 localhost kernel: [] __alloc_pages+0x2b2/0x440 Jun 21 08:56:09 localhost kernel: [] cache_alloc_debugcheck_after+0x6e/0x1a0 Jun 21 08:56:09 localhost kernel: [] __get_free_pages+0x1f/0x40 Jun 21 08:56:09 localhost kernel: [] dma_alloc_coherent+0xce/0x100 Jun 21 08:56:09 localhost kernel: [] __kmalloc+0x9d/0xe0 Jun 21 08:56:10 localhost kernel: [] mthca_alloc_wqe_buf+0x44e/0x490 [ib_mthca] Jun 21 08:56:10 localhost kernel: [] buffered_rmqueue+0xd2/0x2b0 Jun 21 08:56:10 localhost kernel: [] mthca_alloc_qp_common+0x71/0x1b0 [ib_mthca] Jun 21 08:56:10 localhost kernel: [] mthca_alloc_qp+0xb4/0x270 [ib_mthca] Jun 21 08:56:10 localhost kernel: [] mthca_create_qp+0x186/0x2f0 [ib_mthca] Jun 21 08:56:10 localhost kernel: [] ib_create_qp+0x2c/0x90 [ib_core] Jun 21 08:56:10 localhost kernel: [] dapl_ib_qp_alloc+0x156/0x377 [ib_dat_provider] Jun 21 08:56:10 localhost kernel: [] __kmalloc+0x9d/0xe0 Jun 21 08:56:10 localhost kernel: [] dapl_evd_qp_async_error_callback+0x0/0xf5 [ib_dat_provider] Jun 21 08:56:10 localhost kernel: [] dapl_ep_create+0x37b/0x42a [ib_dat_provider] Jun 21 08:56:10 localhost kernel: [] DT_Transaction_Main+0x90c/0x21a0 [kdapltest] Jun 21 08:56:10 localhost kernel: [] __change_page_attr+0x2d/0x170 Jun 21 08:56:10 localhost kernel: [] kernel_map_pages+0x28/0x70 Jun 21 08:56:10 localhost kernel: [] cache_free_debugcheck+0x196/0x2d0 Jun 21 08:56:10 localhost kernel: [] DT_Mdep_Thread_Start_Routine+0x1f/0x30 [kdapltest] Jun 21 08:56:10 localhost kernel: [] DT_Mdep_Thread_Start_Routine+0x0/0x30 [kdapltest] Jun 21 08:56:10 localhost kernel: [] kernel_thread_helper+0x5/0x10 Jun 21 08:56:10 localhost kernel: ------------[ cut here ]------------ Jun 21 08:56:10 localhost kernel: kernel BUG at mm/page_alloc.c:915! Jun 21 08:56:10 localhost kernel: invalid operand: 0000 [#1] Jun 21 08:56:10 localhost kernel: DEBUG_PAGEALLOC Jun 21 08:56:10 localhost kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at dat ib_ipoib ib_sa ide_cd cdrom lp ipv6 autofs parport_pc parport uhci_hcd ehci_hcd ib_mthca ib_mad ib_core ohci_hcd eepro100 mii evdev usbcore Jun 21 08:56:10 localhost kernel: CPU: 0 Jun 21 08:56:10 localhost kernel: EIP: 0060:[] Not tainted VLI Jun 21 08:56:10 localhost kernel: EFLAGS: 00010206 (2.6.11.6) Jun 21 08:56:10 localhost kernel: EIP is at free_pages+0x24/0x40 Jun 21 08:56:10 localhost kernel: eax: 0009a5a5 ebx: 00000000 ecx: c0f4adf8 edx: 00000000 Jun 21 08:56:10 localhost kernel: esi: cdb1f7f8 edi: 00000000 ebp: 0000001d esp: c6f23c68 Jun 21 08:56:10 localhost kernel: ds: 007b es: 007b ss: 0068 Jun 21 08:56:10 localhost kernel: Process DT_Mdep_Thread_ (pid: 7983, threadinfo=c6f22000 task=c607da90) Jun 21 08:56:10 localhost kernel: Stack: d08dcd3d cf907c3c 00001000 5a5a5a5a 00000000 c0146bb2 00000000 c101e940 Jun 21 08:56:10 localhost kernel: 00000000 00000001 00000282 c101e940 c04bddf8 fffffff4 c04bddf8 0000000c Jun 21 08:56:10 localhost kernel: 00000040 00b01000 c6f23dd4 c0f4adf8 cdb1f7f8 cdca37f0 d08dd291 cdb1f7f8 Jun 21 08:56:10 localhost kernel: Call Trace: Jun 21 08:56:10 localhost kernel: [] mthca_alloc_wqe_buf+0x34d/0x490 [ib_mthca] Jun 21 08:56:10 localhost kernel: [] buffered_rmqueue+0xd2/0x2b0 Jun 21 08:56:10 localhost kernel: [] mthca_alloc_qp_common+0x71/0x1b0 [ib_mthca] Jun 21 08:56:10 localhost kernel: [] mthca_alloc_qp+0xb4/0x270 [ib_mthca] Jun 21 08:56:10 localhost kernel: [] mthca_create_qp+0x186/0x2f0 [ib_mthca] Jun 21 08:56:10 localhost kernel: [] ib_create_qp+0x2c/0x90 [ib_core] Jun 21 08:56:10 localhost kernel: [] dapl_ib_qp_alloc+0x156/0x377 [ib_dat_provider] Jun 21 08:56:10 localhost kernel: [] __kmalloc+0x9d/0xe0 Jun 21 08:56:10 localhost kernel: [] dapl_evd_qp_async_error_callback+0x0/0xf5 [ib_dat_provider] Jun 21 08:56:10 localhost kernel: [] dapl_ep_create+0x37b/0x42a [ib_dat_provider] Jun 21 08:56:10 localhost kernel: [] DT_Transaction_Main+0x90c/0x21a0 [kdapltest] Jun 21 08:56:10 localhost kernel: [] __change_page_attr+0x2d/0x170 Jun 21 08:56:10 localhost kernel: [] kernel_map_pages+0x28/0x70 Jun 21 08:56:10 localhost kernel: [] cache_free_debugcheck+0x196/0x2d0 Jun 21 08:56:10 localhost kernel: [] DT_Mdep_Thread_Start_Routine+0x1f/0x30 [kdapltest] Jun 21 08:56:10 localhost kernel: [] DT_Mdep_Thread_Start_Routine+0x0/0x30 [kdapltest] Jun 21 08:56:10 localhost kernel: [] kernel_thread_helper+0x5/0x10 Jun 21 08:56:10 localhost kernel: Code: 8d b4 26 00 00 00 00 85 c0 74 2c 05 00 00 00 40 c1 e8 0c 3b 05 40 aa 3f c0 73 10 8b 0d 50 aa 3f c0 c1 e0 05 01 c8 e9 7c ff ff ff <0f> 0b 93 03 67 6b 31 c0 eb e6 89 f6 c3 eb 0d 90 90 90 90 90 90 Is this related to the recent changes in mthca ? Does something in kdapl need to change ? Thanks. -- Hal From tduffy at sun.com Tue Jun 21 08:38:11 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 21 Jun 2005 08:38:11 -0700 Subject: [openib-general] Re: [PATCH] SDP: use container_of in place of cast to sdp_opt In-Reply-To: <20050621062132.GC18896@mellanox.co.il> References: <1119290772.25973.9.camel@duffman> <20050620220727.A17704@topspin.com> <20050621062132.GC18896@mellanox.co.il> Message-ID: <1119368291.29476.7.camel@duffman> On Tue, 2005-06-21 at 09:21 +0300, Michael S. Tsirkin wrote: > Here's a patch to replace explicit cast to sdp_opt with container_of, removing > assumptions on structure layout. > As a side note, wouldnt sk_to_sdp and sdp_to_sk be better names for > sdp_sk and sk_sdp? I modeled this code based off of every other socket class in Linux. They use casts and shortened names. -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From caitlin.bestler at gmail.com Tue Jun 21 08:44:42 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 21 Jun 2005 08:44:42 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: References: Message-ID: <469958e0050621084468d35106@mail.gmail.com> Expanding slightly on Hal's response, the correct method to correlate an *IP* address with a device is to consult the *IP* routing tables. While there are exceptions where you want to control the local address, it is far more common that you need to reach a remote address (or be reachable from a remote address), as pointed out in Roland's response. So the general algorithm is to check the local routing tables to see what device would be the first hop. >From there it is merely a matter of connecting the device declared for purposes of IP routing with the device that provides RDMA services. As for the differences between *IP* and *IB* addresses, that boils down to a determination of who is responsible for reconciling the difference: the network stack or the application. On 6/20/05, Kevin Reilly wrote: > > > > > Maybe somebody could help me understand the proper way to map between an IP > address assigned to a port to the > "device name" and "port number" in the gen2 architecture. If I have an IP > address can I map it to a name that i get back > from ibv_get_device_name() or pass to ibv_open_device(). > Here is the problem. Let's say there is a subsystem that used > heartbeating over IP interfaces to ascertain the adapters > and ports state, up or down. The data actively maintained be this > heartbeating subsystem is used by a job scheduler that > desires to only schedule tasks on HCAs that are UP and have active ports. > The job scheduler does not want to launch jobs > on adapters that are broken, not connected to a switch or not configured. > The job scheduler needs a way to map an IP address into a name and > port number sitable to be passed to the job to > use when ibv_open_device() is called or to match the output of > ibv_get_device_name(). > > Kevin J. Reilly > STSM, HPC Architecture > -Federation/HPS Chief Engineer > -HPC interconnect architect > (office) 845-433-7976 (tieline) 8-293-7976 > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From timur.tabi at ammasso.com Tue Jun 21 08:51:14 2005 From: timur.tabi at ammasso.com (Timur Tabi) Date: Tue, 21 Jun 2005 10:51:14 -0500 Subject: [openib-general] get_user_pages() and shared memory question Message-ID: <42B83772.90304@ammasso.com> Hi, I have a question about the OpenIB driver's use of get_user_pages() and mlock(). Last time I looked at the code, it was calling mlock() in user space and then calling get_user_pages() in the driver to pin the pages. I was hoping to get some feedback from other developers familiar with this code. I've come across an application that allocates shared memory and then registers it. No problem there. But upon exit, it appears the application is deallocating the shared memory region and then deregistering it. This causes munlock() to fail. This led me to another question: Is it possible for a page of memory that's been "grabbed" with get_user_pages() to ever be allocated to another process? I'm assuming the answer is no, but I have a specific case I want to ask about. Let's say an application allocates some shared memory, and then calls into a driver which calls get_user_pages(). The driver exits without releasing the pages, so they now have a reference count on them. Then the application deallocates the shared memory. At this point, the virtual addresses disappear, and no process owns them, but the pages still have a reference count. Another process now tries to allocate a shared memory buffer. Is there any way that this new buffer can contain those pages that were grabbed with get_user_pages() (i.e. that already have a reference count)? Until 2.6.7, there was a bug in the VM where a page that was grabbed with get_user_pages() could be swapped out. Would that bug affect anything I'm talking about? -- Timur Tabi Staff Software Engineer timur.tabi at ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 From halr at voltaire.com Tue Jun 21 08:54:57 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 21 Jun 2005 11:54:57 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <469958e0050621084468d35106@mail.gmail.com> References: <469958e0050621084468d35106@mail.gmail.com> Message-ID: <1119369297.4477.40.camel@localhost.localdomain> On Tue, 2005-06-21 at 11:44, Caitlin Bestler wrote: > Expanding slightly on Hal's response, the correct method to correlate > an *IP* address with a device is to consult the *IP* routing tables. > While there are exceptions where you want to control the local > address, With IBAT there is a way to force the local interface if that is what is desired. > it is far more common that you need to reach a remote > address (or be reachable from a remote address), as pointed out > in Roland's response. So the general algorithm is to check the > local routing tables to see what device would be the first hop. That's the basic algorithm which IBAT (and SDP) use. -- Hal > >From there it is merely a matter of connecting the device > declared for purposes of IP routing with the device that provides > RDMA services. > > As for the differences between *IP* and *IB* addresses, that > boils down to a determination of who is responsible for reconciling > the difference: the network stack or the application. > > > > On 6/20/05, Kevin Reilly wrote: > > > > > > > > > > Maybe somebody could help me understand the proper way to map between an IP > > address assigned to a port to the > > "device name" and "port number" in the gen2 architecture. If I have an IP > > address can I map it to a name that i get back > > from ibv_get_device_name() or pass to ibv_open_device(). > > Here is the problem. Let's say there is a subsystem that used > > heartbeating over IP interfaces to ascertain the adapters > > and ports state, up or down. The data actively maintained be this > > heartbeating subsystem is used by a job scheduler that > > desires to only schedule tasks on HCAs that are UP and have active ports. > > The job scheduler does not want to launch jobs > > on adapters that are broken, not connected to a switch or not configured. > > The job scheduler needs a way to map an IP address into a name and > > port number sitable to be passed to the job to > > use when ibv_open_device() is called or to match the output of > > ibv_get_device_name(). > > > > Kevin J. Reilly > > STSM, HPC Architecture > > -Federation/HPS Chief Engineer > > -HPC interconnect architect > > (office) 845-433-7976 (tieline) 8-293-7976 > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From roland at topspin.com Tue Jun 21 09:37:20 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 21 Jun 2005 09:37:20 -0700 Subject: [openib-general] Re: page allocation failure with kdapl/mthca In-Reply-To: <1119366787.4477.26.camel@localhost.localdomain> (Hal Rosenstock's message of "21 Jun 2005 11:13:07 -0400") References: <1119366787.4477.26.camel@localhost.localdomain> Message-ID: <52d5qfwtrz.fsf@topspin.com> Hal> Hi Roland, With the latest mthca changes (this test worked Hal> prior to these changes), I get the following running kdapl in Hal> "loopback": You're running out of memory: Hal> Jun 21 08:56:09 localhost kernel: DT_Mdep_Thread_: page Hal> allocation failure. order:0, mode:0x20 An order 0 allocation failing means the kernel can't even find a single free page. However, the mode of 0x20 means this is effectively a GFP_ATOMIC allocation. I just checked in a change that converts from using pci_alloc_consistent(), which is always GFP_ATOMIC, to dma_alloc_coherent(..., GFP_KERNEL). This should help with this symptom. I'm not sure what recent changes could have exposed this problem with your test. - R. From mst at mellanox.co.il Tue Jun 21 10:35:15 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 21 Jun 2005 20:35:15 +0300 Subject: [openib-general] Re: [PATCH] SDP: use container_of in place of cast to sdp_opt In-Reply-To: <1119368291.29476.7.camel@duffman> References: <1119368291.29476.7.camel@duffman> Message-ID: <20050621173515.GA24512@mellanox.co.il> Quoting r. Tom Duffy : > Subject: Re: [PATCH] SDP: use container_of in place of cast to sdp_opt > > On Tue, 2005-06-21 at 09:21 +0300, Michael S. Tsirkin wrote: > > As a side note, wouldnt sk_to_sdp and sdp_to_sk be better names for > > sdp_sk and sk_sdp? > > I modeled this code based off of every other socket class in Linux. > They use casts and shortened names. Seems to be not entirely accurate: there is inet_sk but I no sk_inet that I could see. And inet_sk() kind of makes sense since it returns inet_socket, as opposed to sdp_sk() which returns sdp_conn :) As for casts, inet_sock says: /* sk and pinet6 has to be the first two members of inet_sock */ so apparently there are other tricks that force specific layout in inet_sock which we dont have, or need. -- MST From libor at topspin.com Tue Jun 21 10:54:46 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 21 Jun 2005 10:54:46 -0700 Subject: [openib-general] Re: [PATCH] SDP: use container_of in place of cast to sdp_opt In-Reply-To: <20050621173515.GA24512@mellanox.co.il>; from mst@mellanox.co.il on Tue, Jun 21, 2005 at 08:35:15PM +0300 References: <1119368291.29476.7.camel@duffman> <20050621173515.GA24512@mellanox.co.il> Message-ID: <20050621105446.C17704@topspin.com> On Tue, Jun 21, 2005 at 08:35:15PM +0300, Michael S. Tsirkin wrote: > Quoting r. Tom Duffy : > > Subject: Re: [PATCH] SDP: use container_of in place of cast to sdp_opt > > > > On Tue, 2005-06-21 at 09:21 +0300, Michael S. Tsirkin wrote: > > > As a side note, wouldnt sk_to_sdp and sdp_to_sk be better names for > > > sdp_sk and sk_sdp? > > > > I modeled this code based off of every other socket class in Linux. > > They use casts and shortened names. > > Seems to be not entirely accurate: there is inet_sk but I no sk_inet > that I could see. And inet_sk() kind of makes sense since it returns > inet_socket, as opposed to sdp_sk() which returns sdp_conn :) > > As for casts, inet_sock says: > /* sk and pinet6 has to be the first two members of inet_sock */ > so apparently there are other tricks that force specific layout in inet_sock > which we dont have, or need. sk_alloc() asssumes that 'struct sock' is at the head 'struct inet_sock' which needs to be at the head of the protocol specific structure. -Libor From halr at voltaire.com Tue Jun 21 11:58:42 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 21 Jun 2005 14:58:42 -0400 Subject: [openib-general] SDP can cause IPoIB module to be unable to be removed Message-ID: <1119380321.4477.142.camel@localhost.localdomain> Hi Libor, After using SDP, and unloading SDP, I then attempted to unload IPoIB and got the following: unregister_netdevice: waiting for ib0 to become free. Usage count = 1 The simplest way I found to recreate this is: 1. Bring up IPoIB and then SDP 2. Run tcp.aio.x -t (no server/receiver) 3. Wait for connection refused 4. Unload SDP and then IPoIB -- Hal From jcarr at linuxmachines.com Tue Jun 21 12:47:58 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Tue, 21 Jun 2005 12:47:58 -0700 Subject: [openib-general] A new simple ulp (SPTS) Message-ID: <42B86EEE.4020804@linuxmachines.com> Here is a simple ulp to test transfering pages between machines on an infiniband network. I modified Sean's cmtest to be a standalone thread that listens for incoming connections (ib_cm_spts.ko). A seperate module (spts) registers itself so that incoming connections can be initialized. This code at least managed to work for connecting a small number of nodes together (less than 10). (n)(n-1)/2 connections. However, something still goes wrong after MAX_CQ + 1 transfers, just like the cmtest code. If anyone has any ideas on what is wrong with what I am doing here that would be helpful. It must be something really simple I just don't understand about the IB stack. Specifically, when I create the cq I pass in SPTS_IB_CQ_SIZE: node->recv_cq = ib_create_cq(node->device, spts_comp_handler, spts_event_handler, node, SPTS_IB_CQ_SIZE); and then I set qp_attr.cap.max_send_wr = SPTS_IB_CQ_SIZE. After that initilization, I fill the qp using ib_post_recv(). When I get to the SPTS_IB_CQ_SIZE + 1 message to send, everything stops transfering. I'm not sure what the correct method to "clear" out entries so more buffers can be posted to recieve into. Enjoy, Jeff Signed-off-by: Jeff Carr -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cmspts.diff URL: From roland at topspin.com Tue Jun 21 12:50:03 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 21 Jun 2005 12:50:03 -0700 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <42B86EEE.4020804@linuxmachines.com> (Jeff Carr's message of "Tue, 21 Jun 2005 12:47:58 -0700") References: <42B86EEE.4020804@linuxmachines.com> Message-ID: <527jgnv6ac.fsf@topspin.com> Jeff> However, something still goes wrong after MAX_CQ + 1 Jeff> transfers, just like the cmtest code. If anyone has any Jeff> ideas on what is wrong with what I am doing here that would Jeff> be helpful. It must be something really simple I just don't Jeff> understand about the IB stack. You create the QP so that CQ entries are only created when requested in the send work request: + qp_attr.sq_sig_type = IB_SIGNAL_REQ_WR; but then all of your send work requests have the signal flag unset: + send_wr.send_flags = IB_SEND_SOLICITED; Since no CQ entry is generated for any of the send requests, there is no way for any software on the host to know that the send requests have completed. So no send requests can be freed, and once you have filled your send queue, the driver will refuse to post any more send requests. What happens if you try replacing the send_flags line with the one you have commented out? + // send_wr.send_flags = IB_SEND_SIGNALED; - R. From jcarr at linuxmachines.com Tue Jun 21 13:08:33 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Tue, 21 Jun 2005 13:08:33 -0700 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <527jgnv6ac.fsf@topspin.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> Message-ID: <42B873C1.9090109@linuxmachines.com> On 06/21/05 12:50, Roland Dreier wrote: > Jeff> However, something still goes wrong after MAX_CQ + 1 > Jeff> transfers, just like the cmtest code. If anyone has any > Jeff> ideas on what is wrong with what I am doing here that would > Jeff> be helpful. It must be something really simple I just don't > Jeff> understand about the IB stack. > > You create the QP so that CQ entries are only created when requested > in the send work request: > > + qp_attr.sq_sig_type = IB_SIGNAL_REQ_WR; > > but then all of your send work requests have the signal flag unset: > > + send_wr.send_flags = IB_SEND_SOLICITED; > > Since no CQ entry is generated for any of the send requests, there is > no way for any software on the host to know that the send requests > have completed. So no send requests can be freed, and once you have > filled your send queue, the driver will refuse to post any more send > requests. > > What happens if you try replacing the send_flags line with the one you > have commented out? > > + // send_wr.send_flags = IB_SEND_SIGNALED; Yes, a better way of asking my question would be to say I tried to find documentation on the meaning of IB_SIGNAL_REQ_WR & IB_SEND_SOLICITED vs IB_SEND_SIGNALED but there are no comments in the code anywhere :) It still isn't completely clear to me, but your comments here help. I did find: enum ib_sig_type { IB_SIGNAL_ALL_WR, IB_SIGNAL_REQ_WR }; And wondered what is is that the difference between these are. I have tried both IB_SEND_SIGNALED & IB_SEND_SOLICITED but the behavior seems to be the same. I'm doing more experiementation with this to try to figure it out, but I figured it would be worthwhile to post the code and ask. Thanks for your feedback, Jeff From roland at topspin.com Tue Jun 21 14:23:37 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 21 Jun 2005 14:23:37 -0700 Subject: [openib-general] [ABI CHANGE] Userspace verbs ABI change Message-ID: <52zmtjtndy.fsf@topspin.com> I've just committed changes some changes to both the kernel and userspace side of userspace verbs support. The kernel-user ABI has changed, which means that for example, old userspace binaries will not work with a new kernel. In other words, if you update your svn tree, make sure to update it all and rebuild both userspace and your kernel. I _hope_ this will be the last such ABI change required. I am planning on sending userspace verbs support upstream for review and (we hope) merging as soon as I finish a few minor cleanups and produce a patch series. Once uverbs is merged in the upstream kernel, I will always change IB_USER_VERBS_ABI_VERSION when an incompatible change is made, so this sort of breakage will be avoided. Thanks, Roland From iod00d at hp.com Tue Jun 21 23:55:10 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 21 Jun 2005 23:55:10 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050604181237.GA18586@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> Message-ID: <20050622065510.GC17224@esmail.cup.hp.com> On Sat, Jun 04, 2005 at 09:12:37PM +0300, Michael S. Tsirkin wrote: > > grundler at gsyprf3$ taskset 1 ./rdma_lat -C -n 100000 > > libibverbs: Warning: no userspace device-specific driver found for uverbs0 > > driver search path: /usr/lib/infiniband > > No IB devices found > > Segmentation fault After going two more rounds, I'm still getting the same error. At this point I admit defeat and hope that someone could just tell me the last bit of linker magic or env variable so it Just Works. Or maybe a "make static" target in the Makefile....*sigh* ... > libibverbs uses dlsym(openib_driver_init) to get entry point into low level > driver. yup and that part seems fine. The dl is finding libmthca.so (copy of mthca.so) but it's still not happy with something. > > dlsym documentation says: > > If the executable was linked with > the flag "-rdynamic", then the global symbols in the executable will > also be used to resolve references in a dynamically loaded library. > > So you need to link with -rdynamic. > Maybe you also shall list mthca.a on linker's command line, > and pass in -u openib_driver_init, I dont remember, exactly. It wasn't clear to me if the -rdynamic helped. It seems like the right thing to do. strace showed me that the dynamic linker was trying to load "/usr/lib/libc.so.6.1" after it had loaded "/usr/lib/infiniband/libmthca.so". But that doesn't exist. Proper name in /lib/libc.so.6.1. My bad. I had omitted "/lib" from export LD_LIBRARY_PATH=/usr/lib:/usr/lib/infiniband Don't need to set LD_LIBRARY_PATH. unset that and got the same behavior. Here's the tail end of the strace output: ... open("/sys/class/infiniband_verbs/uverbs0/ibdev", O_RDONLY) = 3 read(3, "mthca0\n", 16384) = 7 close(3) = 0 open("/proc/mounts", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2000000000000000 read(3, "rootfs / rootfs rw 0 0\n/dev/root"..., 1024) = 320 close(3) = 0 munmap(0x2000000000000000, 65536) = 0 lstat("/sys/class/infiniband/mthca0", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 write(2, "libibverbs: Warning: no userspac"..., 96) = 96 This doesn't make any sense: grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ls -l /sys/class/infiniband/mthca0/ total 0 lrwxrwxrwx 1 root root 0 Jun 21 23:30 device -> ../../../devices/pci0000:80/0000:80:01.0/0000:81:00.0 lrwxrwxrwx 1 root root 0 Jun 21 23:30 driver -> ../../../bus/pci/drivers/ib_mthca -r--r--r-- 1 root root 16384 Jun 21 23:30 fw_ver -r--r--r-- 1 root root 16384 Jun 21 23:30 hca_type -r--r--r-- 1 root root 16384 Jun 21 23:30 hw_rev -r--r--r-- 1 root root 16384 Jun 21 23:30 node_guid -r--r--r-- 1 root root 16384 Jun 21 23:30 node_type drwxr-xr-x 4 root root 0 Jun 21 00:39 ports -r--r--r-- 1 root root 16384 Jun 21 23:30 sys_image_guid grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ Whatever libibverbs might want from .../infiniband/mthca0/ it didn't even look for it. write(2, "/usr/lib/infiniband\n", 20) = 20 open("/proc/cpuinfo", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2000000000000000 read(3, "processor : 0\nvendor : Genu"..., 1024) = 508 read(3, "", 1024) = 0 close(3) = 0 munmap(0x2000000000000000, 65536) = 0 getpid() = 12794 gettimeofday({1119421686, 336816}, NULL) = 0 write(2, "No IB devices found\n", 20) = 20 exit_group(1) = ? grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ Any other ideas? Remember, the goal of this exercise is to track down symbols in libibverbs described by pfmon31 Data EAR. thanks, grant From mst at mellanox.co.il Wed Jun 22 00:19:28 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 22 Jun 2005 10:19:28 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050622065510.GC17224@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> Message-ID: <20050622071928.GA32485@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [PATCH] rdma_lat-09 and results > > On Sat, Jun 04, 2005 at 09:12:37PM +0300, Michael S. Tsirkin wrote: > > > grundler at gsyprf3$ taskset 1 ./rdma_lat -C -n 100000 > > > libibverbs: Warning: no userspace device-specific driver found for uverbs0 > > > driver search path: /usr/lib/infiniband > > > No IB devices found > > > Segmentation fault > > After going two more rounds, I'm still getting the same error. Yes, I think we need to make it easier to build static libraries. > At this point I admit defeat and hope that someone could just tell > me the last bit of linker magic or env variable so it Just Works. Here's what works for me: >cc -rdynamic -u openib_driver_init -Wall -O2 -g -D_GNU_SOURCE rdma_bw.c get_clock.c /usr/local/lib/infiniband/mthca.a /usr/local/lib/libibverbs.a -o rdma_bw -ldl -lsysfs -lpthread >ldd rdma_bw libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaaabc1000) libsysfs.so.1 => /lib64/libsysfs.so.1 (0x00002aaaaacc5000) libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x00002aaaaadcf000) libc.so.6 => /lib64/tls/libc.so.6 (0x00002aaaaaee3000) /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000) >./rdma_bw local address: LID 0x02, QPN 0xc0406, PSN 0xd994f5 RKey 0x480030 VAddr 0x00000000513000 > Or maybe a "make static" target in the Makefile....*sigh* Problem with this is, for this I need to figure out where is mthca.a installed. -- MST From arne.redlich at xiranet.com Wed Jun 22 01:39:43 2005 From: arne.redlich at xiranet.com (Arne Redlich) Date: Wed, 22 Jun 2005 10:39:43 +0200 Subject: [openib-general] SDP sk_data_ready() callback Message-ID: <1119429583.8208.28.camel@confield> Hi, I'm trying to use SDP from within the kernel. My problem is that the code relies on sk_data_ready() (this callback is modified to wake up a Rx thread before executing the original function), but sk_data_ready() apparently never gets called. Is there any way to fix this? Thanks, Arne -- Arne Redlich Xiranet Communications GmbH From Nigel.Rochford at HarperCollins.co.uk Wed Jun 22 02:10:41 2005 From: Nigel.Rochford at HarperCollins.co.uk (Rochford, Nigel) Date: Wed, 22 Jun 2005 10:10:41 +0100 Subject: [openib-general] Mmap, 32 bit machine, limit Message-ID: Hi, I'm using the c function "mmap" to memory map binary files into an int**. (on a 32 bit machine). There are a total of 22 files - the maximum of which is about 450MB. Since 4 bytes per int, this will imply 112M addresses will be needed. So: array[0] = file1 array[1] = file2 array[21] = file22 In total, these 22 files come to about 3.4GB - roughly 850M addresses. However, the function crashes and gives out the message ENOMEM after mapping about 2.6GB. Does anyone know what's going wrong??? None of it makes sense to me because: (a) if problem was that 2^31 max addresses, then would crash surely after mapping 2^31bytes (less than 2.6) (b) problem doesn't seem to be lack of swap space. Is there a maximum memory limit in mapping? Nigel Rochford Analyst Programmer +44 306 3563 ********************************************************************** This message is intended only for the use of the individual(s) to which it is addressed and may contain information that is privileged and confidential. If you are not the intended recipient, you are hereby notified that you have received this transmission in error; any review, dissemination, distribution or copying of this transmission is strictly prohibited. If you have received this communication in error, please notify us immediately by reply e-mail and delete this message and all of its attachments. ********************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From MULI at il.ibm.com Wed Jun 22 03:32:41 2005 From: MULI at il.ibm.com (Muli Ben-Yehuda) Date: Wed, 22 Jun 2005 13:32:41 +0300 Subject: [openib-general] Mmap, 32 bit machine, limit In-Reply-To: Message-ID: > In total, these 22 files come to about 3.4GB - roughly 850M addresses. > However, the function crashes and gives out the message ENOMEM after mapping > about 2.6GB. On 32 bit x86 machines, with the stock Linux kernel address space arrangement, you have only 3GB of usable address space (because the kernel is mapped at the top 1GB). Since some address space is taken up by your program code, data and stack (and any shared libraries that it's using), that leaves something less than 3 GB. If you really need more than 3GB of usable address space, either switch to a 64 bit machine, or use multiple processes (each has its own address space) and some form of IPC between them. Cheers, Muli From mst at mellanox.co.il Wed Jun 22 05:07:44 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 22 Jun 2005 15:07:44 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050622071928.GA32485@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> Message-ID: <20050622120743.GZ20041@mellanox.co.il> Quoting r. Michael S. Tsirkin : > Subject: Re: [PATCH] rdma_lat-09 and results > > Quoting r. Grant Grundler : > > Subject: Re: [PATCH] rdma_lat-09 and results > > > > On Sat, Jun 04, 2005 at 09:12:37PM +0300, Michael S. Tsirkin wrote: > > > > grundler at gsyprf3$ taskset 1 ./rdma_lat -C -n 100000 > > > > libibverbs: Warning: no userspace device-specific driver found for uverbs0 > > > > driver search path: /usr/lib/infiniband > > > > No IB devices found > > > > Segmentation fault > > > > After going two more rounds, I'm still getting the same error. > > Yes, I think we need to make it easier to build static libraries. > > > At this point I admit defeat and hope that someone could just tell > > me the last bit of linker magic or env variable so it Just Works. > > > Here's what works for me: > > >cc -rdynamic -u openib_driver_init -Wall -O2 -g -D_GNU_SOURCE rdma_bw.c get_clock.c /usr/local/lib/infiniband/mthca.a /usr/local/lib/libibverbs.a -o rdma_bw -ldl -lsysfs -lpthread OK, I just tried that on your machine. You just have to replace /use/local/lib with /usr/lib. It links fine and starts running. There's some issue with the stack which results in $ ibv_pingpong Couldn't get context for mthca0 but that's probably some setup issue you'll be able to figure out. Roland, is there some way to get the plugin directory out of the libibverbs library? Thanks, -- MST From mst at mellanox.co.il Wed Jun 22 07:12:38 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 22 Jun 2005 17:12:38 +0300 Subject: [openib-general] [PATCH] ib_verbs.h ib_send_wr imm_data endianness Message-ID: <20050622141238.GC20041@mellanox.co.il> Mark imm_data as big endian in ib_send_wr, same as we do in ib_wc. Signed-off-by: Michael S. Tsirkin Index: infiniband/include/ib_verbs.h =================================================================== --- infiniband/include/ib_verbs.h (revision 2670) +++ infiniband/include/ib_verbs.h (working copy) @@ -556,7 +556,7 @@ struct ib_send_wr { int num_sge; enum ib_wr_opcode opcode; int send_flags; - u32 imm_data; + __be32 imm_data; union { struct { u64 remote_addr; -- MST From itamar at mellanox.co.il Wed Jun 22 07:36:35 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 22 Jun 2005 17:36:35 +0300 Subject: [openib-general] [PATCH][kdapl] remove NULL check for input params Message-ID: <20050622143635.GA24010@mellanox.co.il> Apply This patch on top of the the previous patch set (5 patches) dapl_cr.c | 23 ---------- dapl_ep.c | 134 ------------------------------------------------------------- dapl_evd.c | 36 ---------------- dapl_ia.c | 24 ---------- dapl_lmr.c | 37 ---------------- dapl_pz.c | 36 ---------------- dapl_rmr.c | 29 ------------- dapl_sp.c | 86 --------------------------------------- dapl_srq.c | 58 -------------------------- 9 files changed, 4 insertions(+), 459 deletions(-) Delete Null input param check in dapl functions. Signed-off-by: Itamar Rabenstein diff -Nurp -X dontdiff dat-provider_del_hca/dapl_cr.c dat-provider/dapl_cr.c --- dat-provider_del_hca/dapl_cr.c Mon Jun 20 08:40:28 2005 +++ dat-provider/dapl_cr.c Wed Jun 22 11:37:50 2005 @@ -84,11 +84,6 @@ u32 dapl_cr_accept(struct dat_cr *dat_cr dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_accept (%p, %p, %d, %p)\n", dat_cr, dat_ep, priv_size, priv_data); - if (!dat_cr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); - goto bail; - } - cr = (struct dapl_cr *)dat_cr; /* @@ -569,16 +564,6 @@ u32 dapl_cr_query(struct dat_cr *dat_cr, dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", dat_cr, cr_param); - if (dat_cr == NULL) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); - goto bail; - } - - if (NULL == cr_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - cr = (struct dapl_cr *)dat_cr; /* obtain the remote IP address */ @@ -590,7 +575,6 @@ u32 dapl_cr_query(struct dat_cr *dat_cr, status = DAT_SUCCESS; -bail: return status; } @@ -609,11 +593,6 @@ u32 dapl_cr_reject(struct dat_cr *dat_cr dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", dat_cr); - if (!dat_cr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); - goto bail; - } - cr = (struct dapl_cr *)dat_cr; /* @@ -654,7 +633,5 @@ u32 dapl_cr_reject(struct dat_cr *dat_cr dapl_cr_free(cr); } - -bail: return status; } diff -Nurp -X dontdiff dat-provider_del_hca/dapl_ep.c dat-provider/dapl_ep.c --- dat-provider_del_hca/dapl_ep.c Mon Jun 20 08:40:28 2005 +++ dat-provider/dapl_ep.c Wed Jun 22 11:38:46 2005 @@ -175,12 +175,6 @@ static u32 dapl_ep_post_send_req(struct struct dapl_cookie *cookie; u32 dat_status; - if (!ep) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - ep_ptr = (struct dapl_ep *)ep; /* @@ -301,15 +295,6 @@ u32 dapl_ep_create(struct dat_ia *ia, st ia_ptr = (struct dapl_ia *)ia; /* - * Verify parameters - */ - if (!ia_ptr) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - - /* * Verify non-required parameters. * N.B. Assumption: any parameter that can be * modified by dat_ep_modify() is not strictly @@ -342,15 +327,6 @@ u32 dapl_ep_create(struct dat_ia *ia, st goto bail; } - if (ep == NULL) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); - goto bail; - } - if ((unsigned long)ep_attr & 3) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG6); - goto bail; - } - /* * Qualify EP Attributes are legal and make sense. Note that if one * or both of the DTO handles are NULL, then the corresponding @@ -495,15 +471,6 @@ u32 dapl_ep_create_with_srq(struct dat_i ia_ptr = (struct dapl_ia *)ia; /* - * Verify parameters - */ - if (!ia_ptr) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - - /* * Verify non-required parameters. * N.B. Assumption: any parameter that can be * modified by dat_ep_modify() is not strictly @@ -537,25 +504,6 @@ u32 dapl_ep_create_with_srq(struct dat_i } /* - * Verify the SRQ handle. It is an error to invoke this call with - * a NULL handle - */ - if (!srq) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - - if (ep == NULL) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); - goto bail; - } - if ((unsigned long)ep_attr & 3) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG6); - goto bail; - } - - /* * Qualify EP Attributes are legal and make sense. Note that if one * or both of the DTO handles are NULL, then the corresponding * max_*_dtos must 0 as the user will not be able to post dto ops on @@ -703,15 +651,6 @@ u32 dapl_ep_free(struct dat_ep *ep) ep_ptr = (struct dapl_ep *)ep; param = &ep_ptr->param; - /* - * Verify parameter & state - */ - if (!ep_ptr) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - if (ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED || ep_ptr->param.ep_state == DAT_EP_STATE_PASSIVE_CONNECTION_PENDING || ep_ptr->param.ep_state == DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) @@ -834,12 +773,6 @@ u32 dapl_ep_connect(struct dat_ep *ep, * Verify parameter & state. The connection handle must be good * at this point. */ - if (!ep_ptr) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - if (!ep_ptr->param.connect_evd) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); @@ -1036,16 +969,6 @@ u32 dapl_ep_dup_connect(struct dat_ep *e ep_dup_ptr = (struct dapl_ep *)ep_dup; - /* - * Verify the dup handle, which must be connected. All other - * parameters will be verified by dapl_ep_connect - */ - if (!ep_dup) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - /* Can't do a connection in 0 time, reject outright */ if (timeout == 0) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -1096,15 +1019,6 @@ u32 dapl_ep_disconnect(struct dat_ep *da ep = (struct dapl_ep *)dat_ep; /* - * Verify parameter & state - */ - if (!ep) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - - /* * Do the verification of parameters and the state change * atomically. */ @@ -1875,11 +1789,6 @@ u32 dapl_ep_post_recv(struct dat_ep *ep, ep, num_segments, local_iov, user_cookie.as_64, completion_flags); - if (!ep) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - ep_ptr = (struct dapl_ep *) ep; /* @@ -1950,19 +1859,6 @@ u32 dapl_ep_query(struct dat_ep *dat_ep, ep = (struct dapl_ep *)dat_ep; /* - * Verify parameter & state - */ - if (!ep) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - - if (ep_param == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - /* * Fill in according to user request * * N.B. Just slam all values into the user structure, there @@ -1978,7 +1874,6 @@ u32 dapl_ep_query(struct dat_ep *dat_ep, (struct sockaddr *)&ep->remote_ia_address; *ep_param = ep->param; -bail: return status; } @@ -1993,18 +1888,8 @@ u32 dapl_ep_recv_query(struct dat_ep *ep ep_ptr = (struct dapl_ep *)ep; - /* - * Verify parameter & state - */ - if (!ep_ptr) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); -bail: return dat_status; } @@ -2016,15 +1901,6 @@ u32 dapl_ep_reset(struct dat_ep *ep) ep_ptr = (struct dapl_ep *)ep; - /* - * Verify parameter & state - */ - if (!ep_ptr) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED && ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECTED) { dat_status = @@ -2053,18 +1929,8 @@ u32 dapl_ep_set_watermark(struct dat_ep ep_ptr = (struct dapl_ep *)ep; - /* - * Verify parameter & state - */ - if (!ep_ptr) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - dat_status = DAT_NOT_IMPLEMENTED; -bail: return dat_status; } diff -Nurp -X dontdiff dat-provider_del_hca/dapl_evd.c dat-provider/dapl_evd.c --- dat-provider_del_hca/dapl_evd.c Mon Jun 20 08:40:28 2005 +++ dat-provider/dapl_evd.c Wed Jun 22 11:39:25 2005 @@ -1013,11 +1013,6 @@ u32 dapl_evd_kcreate(struct dat_ia *ia_h evd = NULL; *evd_handle = NULL; - if (!ia_handle) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (min_qlen <= 0) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; @@ -1060,11 +1055,6 @@ u32 dapl_evd_free(struct dat_evd *evd_ha evd = (struct dapl_evd *)evd_handle; - if (!evd_handle) { - status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - if (atomic_read(&evd->evd_ref_count) != 0) { status = DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EVD_IN_USE); @@ -1090,11 +1080,6 @@ u32 dapl_evd_kquery(struct dat_evd *evd_ struct dapl_evd *evd; u32 status = DAT_SUCCESS; - if (NULL == evd_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - /* Note: the spec. allows for events to be directed to a NULL EVD */ /* with handle of type NULL. See 6.3.1 */ if (NULL == evd_handle) { @@ -1138,15 +1123,9 @@ u32 dapl_evd_modify_upcall(struct dat_ev evd = (struct dapl_evd *)evd_handle; - if (!evd_handle) { - status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - evd->upcall_policy = upcall_policy; evd->upcall = *upcall; -bail: return status; } @@ -1195,16 +1174,6 @@ u32 dapl_evd_dequeue(struct dat_evd *evd evd = (struct dapl_evd *)evd_handle; - if (!evd_handle) { - status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - - if (event == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - /* * We need to dequeue under lock, as the IB OS Access API * restricts us from having multiple threads in CQ poll, and the @@ -1255,11 +1224,6 @@ u32 dapl_evd_resize(struct dat_evd *evd_ dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", evd_handle, evd_qlen); - if (!evd_handle) { - status = DAT_ERROR(DAT_INVALID_HANDLE, 0); - goto bail; - } - evd = (struct dapl_evd *)evd_handle; ia = evd->common.owner_ia; diff -Nurp -X dontdiff dat-provider_del_hca/dapl_ia.c dat-provider/dapl_ia.c --- dat-provider_del_hca/dapl_ia.c Mon Jun 20 08:40:28 2005 +++ dat-provider/dapl_ia.c Wed Jun 22 11:40:02 2005 @@ -611,16 +611,6 @@ u32 dapl_ia_open(const char *name, int a goto bail; } - /* ia_ptr and async_evd cannot be NULL */ - if (ia_ptr == NULL) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - if (async_evd == NULL) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - /* initialize the caller's param */ *ia_ptr = NULL; @@ -710,12 +700,6 @@ u32 dapl_ia_close(struct dat_ia *ia_ptr, ia = (struct dapl_ia *)ia_ptr; - if (!ia) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) dat_status = dapl_ia_abrupt_close(ia); else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) @@ -723,7 +707,6 @@ u32 dapl_ia_close(struct dat_ia *ia_ptr, else dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); -bail: return dat_status; } @@ -783,12 +766,6 @@ u32 dapl_ia_query(struct dat_ia *ia_ptr, ia = (struct dapl_ia *)ia_ptr; - if (!ia) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (NULL != async_evd) { *async_evd = (struct dat_evd *)ia->async_error_evd; } @@ -862,7 +839,6 @@ u32 dapl_ia_query(struct dat_ia *ia_ptr, #endif /* DAPL_MERGE_CM_DTO */ } -bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_ia_query () returns 0x%x\n", dat_status); diff -Nurp -X dontdiff dat-provider_del_hca/dapl_lmr.c dat-provider/dapl_lmr.c --- dat-provider_del_hca/dapl_lmr.c Mon Jun 20 08:40:28 2005 +++ dat-provider/dapl_lmr.c Wed Jun 22 11:45:13 2005 @@ -241,15 +241,6 @@ u32 dapl_lmr_kcreate(struct dat_ia *ia, "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", ia, mem_type); - if (!ia) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (!pz) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - dapl_ia = (struct dapl_ia *)ia; dapl_pz = (struct dapl_pz *)pz; @@ -309,11 +300,6 @@ u32 dapl_lmr_free(struct dat_lmr *lmr) dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr); - if (!lmr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); - goto error; - } - dapl_lmr = (struct dapl_lmr *)lmr; switch (dapl_lmr->param.mem_type) { @@ -343,7 +329,6 @@ u32 dapl_lmr_free(struct dat_lmr *lmr) status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); break; } -error: return status; } @@ -355,19 +340,9 @@ u32 dapl_lmr_query(struct dat_lmr *lmr, dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", lmr, lmr_param); - if (!lmr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); - goto error; - } - if (NULL == lmr_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto error; - } - dapl_lmr = (struct dapl_lmr *)lmr; memcpy(lmr_param, &dapl_lmr->param, sizeof *lmr_param); status = DAT_SUCCESS; -error: return status; } @@ -388,12 +363,6 @@ u32 dapl_lmr_sync_rdma_read(struct dat_i ia_ptr = (struct dapl_ia *)ia; - if (!ia_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto error; - } - -error: return status; } @@ -409,11 +378,5 @@ u32 dapl_lmr_sync_rdma_write(struct dat_ ia_ptr = (struct dapl_ia *)ia; - if (!ia_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto error; - } - -error: return status; } diff -Nurp -X dontdiff dat-provider_del_hca/dapl_pz.c dat-provider/dapl_pz.c --- dat-provider_del_hca/dapl_pz.c Mon Jun 20 08:40:28 2005 +++ dat-provider/dapl_pz.c Wed Jun 22 11:41:13 2005 @@ -75,40 +75,23 @@ u32 dapl_pz_create(struct dat_ia *ia, st dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", ia, pz); - if (!ia) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto error1; - } - if (NULL == pz) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto error1; - } - dapl_ia = (struct dapl_ia *)ia; dapl_pz = dapl_pz_alloc(dapl_ia); - if (!dapl_pz) { - status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto error1; - } - dapl_pz->pd = ib_alloc_pd(dapl_ia->hca->ib_hca_handle); if (IS_ERR(dapl_pz->pd)) { ib_status = PTR_ERR(dapl_pz->pd); dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_alloc_pd failed: %X\n", ib_status); status = dapl_ib_status_convert(ib_status); - goto error2; + goto error; } *pz = (struct dat_pz *)dapl_pz; return DAT_SUCCESS; -error2: +error: dapl_pz_dealloc(dapl_pz); - -error1: *pz = NULL; return status; } @@ -121,11 +104,6 @@ u32 dapl_pz_free(struct dat_pz *pz) dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz); - if (!pz) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto error; - } - dapl_pz = (struct dapl_pz *)pz; if (0 != atomic_read(&dapl_pz->pz_ref_count)) { @@ -157,19 +135,9 @@ u32 dapl_pz_query(struct dat_pz *pz, str dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query(%p, %x, %p)\n", pz, pz_param); - if (!pz) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto error; - } - if (NULL == pz_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto error; - } - dapl_pz = (struct dapl_pz *)pz; pz_param->ia = (struct dat_ia *)dapl_pz->common.owner_ia; return DAT_SUCCESS; -error: return status; } diff -Nurp -X dontdiff dat-provider_del_hca/dapl_rmr.c dat-provider/dapl_rmr.c --- dat-provider_del_hca/dapl_rmr.c Mon Jun 20 08:40:28 2005 +++ dat-provider/dapl_rmr.c Wed Jun 22 11:41:36 2005 @@ -354,15 +354,6 @@ u32 dapl_rmr_bind(struct dat_rmr *rmr_ha struct dapl_lmr *lmr; struct dapl_ep *ep_ptr; - if (!rmr_handle) - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - - if (!lmr_handle) - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); - - if (!ep) - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - rmr = (struct dapl_rmr *)rmr_handle; lmr = (struct dapl_lmr *)lmr_handle; ep_ptr = (struct dapl_ep *)ep; @@ -399,11 +390,6 @@ u32 dapl_rmr_create(struct dat_pz *pz, s struct dapl_rmr *dapl_rmr; u32 status = DAT_SUCCESS; - if (!pz) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - dapl_pz = (struct dapl_pz *)pz; dapl_rmr = dapl_rmr_alloc(dapl_pz); @@ -451,11 +437,6 @@ u32 dapl_rmr_free(struct dat_rmr *rmr_ha struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (!rmr_handle) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - goto bail; - } - rmr = (struct dapl_rmr *)rmr_handle; /* @@ -485,19 +466,9 @@ u32 dapl_rmr_query(struct dat_rmr *rmr_h struct dapl_rmr *rmr; u32 status = DAT_SUCCESS; - if (!rmr_handle) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - goto bail; - } - if (NULL == rmr_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - rmr = (struct dapl_rmr *)rmr_handle; memcpy(rmr_param, &rmr->param, sizeof *rmr_param); -bail: return status; } diff -Nurp -X dontdiff dat-provider_del_hca/dapl_sp.c dat-provider/dapl_sp.c --- dat-provider_del_hca/dapl_sp.c Mon Jun 20 08:40:28 2005 +++ dat-provider/dapl_sp.c Wed Jun 22 11:41:57 2005 @@ -145,25 +145,6 @@ u32 dapl_rsp_create(struct dat_ia *dat_i ">>> dapl_rsp_free conn_qual: %x EP: %p\n", conn_qual, dat_ep); - if (!dat_ia) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (!dat_ep) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - if (!dat_evd) { - status = DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (rsp == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - ia = (struct dapl_ia *)dat_ia; ep = (struct dapl_ep *)dat_ep; @@ -263,14 +244,8 @@ u32 dapl_rsp_free(struct dat_sp *dat_rsp u32 status = DAT_SUCCESS; sp = (struct dapl_sp *)dat_rsp; - /* - * Verify handle - */ + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", dat_rsp); - if (!sp) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); - goto bail; - } ia = sp->common.owner_ia; @@ -341,16 +316,6 @@ u32 dapl_rsp_query(struct dat_sp *dat_rs struct dapl_sp *sp; u32 status; - if (!dat_rsp) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); - goto bail; - } - - if (NULL == rsp_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - sp = (struct dapl_sp *)dat_rsp; /* @@ -363,7 +328,6 @@ u32 dapl_rsp_query(struct dat_sp *dat_rs status = DAT_SUCCESS; -bail: return status; } @@ -390,25 +354,6 @@ u32 dapl_psp_create_any(struct dat_ia *d "dapl_psp_create_anny(%p, %p, %p, %X, %p)\n", dat_ia, conn_qual, dat_evd, psp_flags, dat_psp); - if (!dat_ia) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (!dat_evd) { - status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (!dat_psp) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - if (conn_qual == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - ia = (struct dapl_ia *)dat_ia; evd = (struct dapl_evd *)dat_evd; if (!(evd->evd_flags & DAT_EVD_CR_FLAG)) { @@ -510,21 +455,6 @@ u32 dapl_psp_create(struct dat_ia *dat_i dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_psp_create(%p, %X, %p, %X, %p)\n", dat_ia, conn_qual, dat_evd, psp_flags, dat_psp); - if (!dat_ia) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (!dat_evd) { - status = DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (!dat_psp) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - ia = (struct dapl_ia *)dat_ia; evd = (struct dapl_evd *)dat_evd; if (!(evd->evd_flags & DAT_EVD_CR_FLAG)) { @@ -628,15 +558,8 @@ u32 dapl_psp_free(struct dat_sp *dat_psp u32 status = DAT_SUCCESS; sp = (struct dapl_sp *)dat_psp; - /* - * Verify handle - */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", dat_psp); - if (!sp) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); - goto bail; - } + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", dat_psp); ia = sp->common.owner_ia; /* @@ -712,11 +635,6 @@ u32 dapl_psp_query(struct dat_sp *dat_ps goto bail; } - if (NULL == psp_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - sp = (struct dapl_sp *)dat_psp; psp_param->ia = sp->ia; diff -Nurp -X dontdiff dat-provider_del_hca/dapl_srq.c dat-provider/dapl_srq.c --- dat-provider_del_hca/dapl_srq.c Mon Jun 20 08:40:27 2005 +++ dat-provider/dapl_srq.c Wed Jun 22 11:42:23 2005 @@ -149,30 +149,6 @@ u32 dapl_srq_create(struct dat_ia *ia, s ia_ptr = (struct dapl_ia *)ia; - /* - * Verify parameters - */ - if (!ia_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - - /* - * Verify non-required parameters. - * N.B. Assumption: any parameter that can be - * modified by dat_ep_modify() is not strictly - * required when the EP is created - */ - if (srq == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - if ((unsigned long)srq_attr & 3) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - /* Allocate SRQ */ srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); if (srq_ptr == NULL) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, @@ -228,14 +204,6 @@ u32 dapl_srq_free(struct dat_srq *srq) srq_ptr = (struct dapl_srq *)srq; param = &srq_ptr->param; - /* - * Verify parameter & state - */ - if (!srq_ptr) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - if (atomic_read(&srq_ptr->srq_ref_count) != 0) { /* * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does @@ -313,11 +281,6 @@ u32 dapl_srq_post_recv(struct dat_srq *s "dapl_srq_post_recv (%p, %d, %p, %P)\n", srq, num_segments, local_iov, user_cookie.as_64); - if (!srq) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - srq_ptr = (struct dapl_srq *)srq; /* @@ -360,15 +323,6 @@ u32 dapl_srq_query(struct dat_srq *srq, dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", srq, srq_param); - if (!srq) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - if (srq_param == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - srq_ptr = (struct dapl_srq *)srq; /* @@ -379,7 +333,6 @@ u32 dapl_srq_query(struct dat_srq *srq, *srq_param = srq_ptr->param; -bail: return status; } @@ -412,11 +365,6 @@ u32 dapl_srq_resize(struct dat_srq *srq, dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", srq, srq_max_recv_dto); - if (!srq) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - srq_ptr = (struct dapl_srq *)srq; ia_ptr = srq_ptr->common.owner_ia; @@ -466,17 +414,11 @@ u32 dapl_srq_set_lw(struct dat_srq *srq, dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq, low_watermark); - if (!srq) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - srq_ptr = (struct dapl_srq *)srq; /* XXX Put implementation here XXX */ /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); -bail: return status; } -- Itamar From tduffy at sun.com Wed Jun 22 08:25:59 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 22 Jun 2005 08:25:59 -0700 Subject: [openib-general] [PATCH][kdapl] remove NULL check for input params In-Reply-To: <20050622143635.GA24010@mellanox.co.il> References: <20050622143635.GA24010@mellanox.co.il> Message-ID: <1119453959.26537.9.camel@duffman> On Wed, 2005-06-22 at 17:36 +0300, Itamar Rabenstein wrote: > diff -Nurp -X dontdiff dat-provider_del_hca/dapl_pz.c dat-provider/dapl_pz.c > --- dat-provider_del_hca/dapl_pz.c Mon Jun 20 08:40:28 2005 > +++ dat-provider/dapl_pz.c Wed Jun 22 11:41:13 2005 > @@ -75,40 +75,23 @@ u32 dapl_pz_create(struct dat_ia *ia, st > dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", > ia, pz); > > - if (!ia) { > - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); > - goto error1; > - } > - if (NULL == pz) { > - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); > - goto error1; > - } > - > dapl_ia = (struct dapl_ia *)ia; > > dapl_pz = dapl_pz_alloc(dapl_ia); > - if (!dapl_pz) { > - status = > - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); > - goto error1; > - } > - This is an alloc check, should stay in. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Wed Jun 22 08:57:30 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 22 Jun 2005 08:57:30 -0700 Subject: [openib-general] [PATCHv2] SDP: rename sdp_opt to sdp_sock Message-ID: <1119455850.26537.12.camel@duffman> This patch renames the sdp_opt struct sdp_sock as this is what all the other socket classes do. Signed-off-by: Tom Duffy Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_write.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_write.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_write.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_event_write - RDMA write event handler */ -int sdp_event_write(struct sdp_opt *conn, struct ib_wc *comp) +int sdp_event_write(struct sdp_sock *conn, struct ib_wc *comp) { struct sdpc_iocb *iocb; struct sdpc_buff *buff; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -38,7 +38,7 @@ /* * Specific MID handler functions. (RECV) */ -static int sdp_rcvd_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_disconnect(struct sdp_sock *conn, struct sdpc_buff *buff) { int result = 0; int band; @@ -99,7 +99,7 @@ error: return result; } -static int sdp_rcvd_abort(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_abort(struct sdp_sock *conn, struct sdpc_buff *buff) { int result = 0; @@ -123,7 +123,7 @@ static int sdp_rcvd_abort(struct sdp_opt return result; } -static int sdp_rcvd_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_send_sm(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; @@ -164,7 +164,7 @@ static int sdp_rcvd_send_sm(struct sdp_o return 0; } -static int sdp_rcvd_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_rdma_wr(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_rwch *rwch; struct sdpc_iocb *iocb; @@ -214,7 +214,7 @@ static int sdp_rcvd_rdma_wr(struct sdp_o return 0; } -static int sdp_rcvd_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_rdma_rd(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_rrch *rrch; struct sdpc_iocb *iocb; @@ -285,7 +285,7 @@ static int sdp_rcvd_rdma_rd(struct sdp_o return 0; } -static int sdp_rcvd_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_mode_change(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_mch *mch; int result; @@ -381,7 +381,7 @@ error: return result; } -static int sdp_rcvd_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_src_cancel(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_advt *advt; int result; @@ -470,7 +470,7 @@ done: return result; } -static int sdp_rcvd_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_snk_cancel(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_advt *advt; s32 counter; @@ -543,7 +543,7 @@ done: /* * sdp_rcvd_snk_cancel_ack - sink cancel confirmantion */ -static int sdp_rcvd_snk_cancel_ack(struct sdp_opt *conn, +static int sdp_rcvd_snk_cancel_ack(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; @@ -573,7 +573,7 @@ static int sdp_rcvd_snk_cancel_ack(struc /* * sdp_rcvd_resize_buff_ack - buffer size change request */ -static int sdp_rcvd_resize_buff_ack(struct sdp_opt *conn, +static int sdp_rcvd_resize_buff_ack(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_crbh *crbh; @@ -603,7 +603,7 @@ error: return result; } -static int sdp_rcvd_suspend(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_suspend(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_sch *sch; @@ -615,12 +615,12 @@ static int sdp_rcvd_suspend(struct sdp_o return 0; } -static int sdp_rcvd_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_suspend_ack(struct sdp_sock *conn, struct sdpc_buff *buff) { return 0; } -static int sdp_rcvd_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_snk_avail(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_snkah *snkah; struct sdpc_advt *advt; @@ -745,7 +745,7 @@ consume: return result; } -static int sdp_rcvd_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_src_avail(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_srcah *srcah; struct sdpc_advt *advt; @@ -914,7 +914,7 @@ done: /* * sdp_rcvd_data - SDP data message event received */ -static int sdp_rcvd_data(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_data(struct sdp_sock *conn, struct sdpc_buff *buff) { int ret_val; @@ -959,7 +959,7 @@ static int sdp_rcvd_data(struct sdp_opt /* * sdp_rcvd_unsupported - Valid messages we're not expecting */ -static int sdp_rcvd_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_unsupported(struct sdp_sock *conn, struct sdpc_buff *buff) { /* * Since the gateway only initates RDMA's but is never a target, and @@ -1023,7 +1023,7 @@ static sdp_event_cb_func recv_event_func /* * sdp_event_recv - recv event demultiplexing into sdp messages */ -int sdp_event_recv(struct sdp_opt *conn, struct ib_wc *comp) +int sdp_event_recv(struct sdp_sock *conn, struct ib_wc *comp) { sdp_event_cb_func dispatch_func; struct sdpc_buff *buff; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -130,7 +130,7 @@ MODULE_PARM_DESC(sdp_debug_level, */ void sdp_inet_wake_send(struct sock *sk) { - struct sdp_opt *conn = sdp_sk(sk); + struct sdp_sock *conn = sdp_sk(sk); if (!sk) return; @@ -212,7 +212,7 @@ void sdp_inet_wake_urg(struct sock *sk) /* * sdp_inet_disconnect - disconnect a connection */ -static int sdp_inet_disconnect(struct sdp_opt *conn) +static int sdp_inet_disconnect(struct sdp_sock *conn) { int result = 0; /* @@ -294,7 +294,7 @@ error: */ static int sdp_inet_release(struct socket *sock) { - struct sdp_opt *conn; + struct sdp_sock *conn; struct sock *sk = sock->sk; int result; long timeout; @@ -424,7 +424,7 @@ static int sdp_inet_bind(struct socket * { struct sockaddr_in *addr = (struct sockaddr_in *)uaddr; struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; unsigned int addr_result = RTN_UNSPEC; u16 bind_port; int result; @@ -516,7 +516,7 @@ static int sdp_inet_connect(struct socke { struct sockaddr_in *addr = (struct sockaddr_in *)uaddr; struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; long timeout; int result; @@ -679,7 +679,7 @@ done: static int sdp_inet_listen(struct socket *sock, int backlog) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; int result; sk = sock->sk; @@ -738,8 +738,8 @@ static int sdp_inet_accept(struct socket { struct sock *listen_sk; struct sock *accept_sk = NULL; - struct sdp_opt *listen_conn; - struct sdp_opt *accept_conn = NULL; + struct sdp_sock *listen_conn; + struct sdp_sock *accept_conn = NULL; int result; long timeout; @@ -866,7 +866,7 @@ static int sdp_inet_getname(struct socke { struct sockaddr_in *addr = (struct sockaddr_in *)uaddr; struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; sk = sock->sk; conn = sdp_sk(sk); @@ -901,7 +901,7 @@ static unsigned int sdp_inet_poll(struct poll_table *wait) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; unsigned int mask = 0; /* @@ -990,7 +990,7 @@ static int sdp_inet_ioctl(struct socket unsigned long arg) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_buff *buff; int result = 0; int value; @@ -1113,7 +1113,7 @@ static int sdp_inet_setopt(struct socket char __user *optval, int optlen) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; int value; int result = 0; @@ -1178,7 +1178,7 @@ static int sdp_inet_getopt(struct socket char __user *optval, int __user *optlen) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; int value; int len; @@ -1239,7 +1239,7 @@ static int sdp_inet_getopt(struct socket static int sdp_inet_shutdown(struct socket *sock, int flag) { int result = 0; - struct sdp_opt *conn; + struct sdp_sock *conn; conn = sdp_sk(sock->sk); @@ -1326,7 +1326,7 @@ static struct proto_ops lnx_stream_ops = */ static int sdp_inet_create(struct socket *sock, int protocol) { - struct sdp_opt *conn; + struct sdp_sock *conn; sdp_dbg_ctrl(NULL, "SOCKET: type <%d> proto <%d> state <%u:%08lx>", sock->type, protocol, sock->state, sock->flags); @@ -1379,7 +1379,7 @@ static struct net_proto_family sdp_proto struct proto sdp_sk_proto = { .name = "SDP", .owner = THIS_MODULE, - .obj_size = sizeof(struct sdp_opt), + .obj_size = sizeof(struct sdp_sock), }; /* Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_proto.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_proto.h (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_proto.h (working copy) @@ -108,11 +108,11 @@ int sdp_proc_dump_buff_pool(char *buffer /* * Wall between userspace protocol and SDP protocol proper */ -void sdp_conn_abort(struct sdp_opt *conn); +void sdp_conn_abort(struct sdp_sock *conn); -void sdp_conn_inet_error(struct sdp_opt *conn, int error); +void sdp_conn_inet_error(struct sdp_sock *conn, int error); -int sdp_recv_buff(struct sdp_opt *conn, struct sdpc_buff *buff); +int sdp_recv_buff(struct sdp_sock *conn, struct sdpc_buff *buff); /* * Zcopy advertisment managment @@ -166,7 +166,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_ void sdp_iocb_q_remove(struct sdpc_iocb *iocb); -int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_opt *conn); +int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_sock *conn); void sdp_iocb_release(struct sdpc_iocb *iocb); @@ -258,13 +258,13 @@ int sdp_proc_dump_device(char *buffer, off_t start_index, long *end_index); -int sdp_conn_table_remove(struct sdp_opt *conn); +int sdp_conn_table_remove(struct sdp_sock *conn); -struct sdp_opt *sdp_conn_table_lookup(s32 entry); +struct sdp_sock *sdp_conn_table_lookup(s32 entry); -struct sdp_opt *sdp_conn_alloc(int priority); +struct sdp_sock *sdp_conn_alloc(int priority); -int sdp_conn_alloc_ib(struct sdp_opt *conn, +int sdp_conn_alloc_ib(struct sdp_sock *conn, struct ib_device *device, u8 hw_port, u16 pkey); @@ -282,39 +282,39 @@ void sdp_inet_wake_urg(struct sock *sk); /* * port/queue managment */ -int sdp_inet_accept_q_put(struct sdp_opt *listen_conn, - struct sdp_opt *accept_conn); +int sdp_inet_accept_q_put(struct sdp_sock *listen_conn, + struct sdp_sock *accept_conn); -struct sdp_opt *sdp_inet_accept_q_get(struct sdp_opt *listen_conn); +struct sdp_sock *sdp_inet_accept_q_get(struct sdp_sock *listen_conn); -int sdp_inet_accept_q_remove(struct sdp_opt *accept_conn); +int sdp_inet_accept_q_remove(struct sdp_sock *accept_conn); -int sdp_inet_listen_start(struct sdp_opt *listen_conn); +int sdp_inet_listen_start(struct sdp_sock *listen_conn); -int sdp_inet_listen_stop(struct sdp_opt *listen_conn); +int sdp_inet_listen_stop(struct sdp_sock *listen_conn); -struct sdp_opt *sdp_inet_listen_lookup(u32 addr, u16 port); +struct sdp_sock *sdp_inet_listen_lookup(u32 addr, u16 port); -int sdp_inet_port_get(struct sdp_opt *conn, u16 port); +int sdp_inet_port_get(struct sdp_sock *conn, u16 port); -int sdp_inet_port_put(struct sdp_opt *conn); +int sdp_inet_port_put(struct sdp_sock *conn); -int sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child); +int sdp_inet_port_inherit(struct sdp_sock *parent, struct sdp_sock *child); /* * active connect functions */ -int sdp_cm_connect(struct sdp_opt *conn); +int sdp_cm_connect(struct sdp_sock *conn); int sdp_cm_rep_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn); + struct sdp_sock *conn); -void sdp_cm_actv_error(struct sdp_opt *conn, int error); +void sdp_cm_actv_error(struct sdp_sock *conn, int error); /* * passive connect functions */ -int sdp_cm_pass_establish(struct sdp_opt *conn); +int sdp_cm_pass_establish(struct sdp_sock *conn); int sdp_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event); @@ -322,36 +322,36 @@ int sdp_cm_req_handler(struct ib_cm_id * /* * post functions */ -int sdp_recv_flush(struct sdp_opt *conn); +int sdp_recv_flush(struct sdp_sock *conn); -int sdp_send_flush(struct sdp_opt *conn); +int sdp_send_flush(struct sdp_sock *conn); -int sdp_send_ctrl_ack(struct sdp_opt *conn); +int sdp_send_ctrl_ack(struct sdp_sock *conn); -int sdp_send_ctrl_disconnect(struct sdp_opt *conn); +int sdp_send_ctrl_disconnect(struct sdp_sock *conn); -int sdp_send_ctrl_abort(struct sdp_opt *conn); +int sdp_send_ctrl_abort(struct sdp_sock *conn); -int sdp_send_ctrl_send_sm(struct sdp_opt *conn); +int sdp_send_ctrl_send_sm(struct sdp_sock *conn); -int sdp_send_ctrl_snk_avail(struct sdp_opt *conn, +int sdp_send_ctrl_snk_avail(struct sdp_sock *conn, u32 size, u32 rkey, u64 addr); -int sdp_send_ctrl_resize_buff_ack(struct sdp_opt *conn, u32 size); +int sdp_send_ctrl_resize_buff_ack(struct sdp_sock *conn, u32 size); -int sdp_send_ctrl_rdma_rd(struct sdp_opt *conn, s32 size); +int sdp_send_ctrl_rdma_rd(struct sdp_sock *conn, s32 size); -int sdp_send_ctrl_rdma_wr(struct sdp_opt *conn, u32 size); +int sdp_send_ctrl_rdma_wr(struct sdp_sock *conn, u32 size); -int sdp_send_ctrl_mode_ch(struct sdp_opt *conn, u8 mode); +int sdp_send_ctrl_mode_ch(struct sdp_sock *conn, u8 mode); -int sdp_send_ctrl_src_cancel(struct sdp_opt *conn); +int sdp_send_ctrl_src_cancel(struct sdp_sock *conn); -int sdp_send_ctrl_snk_cancel(struct sdp_opt *conn); +int sdp_send_ctrl_snk_cancel(struct sdp_sock *conn); -int sdp_send_ctrl_snk_cancel_ack(struct sdp_opt *conn); +int sdp_send_ctrl_snk_cancel_ack(struct sdp_sock *conn); /* * inet functions @@ -360,20 +360,20 @@ int sdp_send_ctrl_snk_cancel_ack(struct /* * event functions */ -int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_opt *conn); +int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_sock *conn); void sdp_cq_event_handler(struct ib_cq *cq, void *arg); int sdp_cm_event_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event); -int sdp_event_recv(struct sdp_opt *conn, struct ib_wc *comp); +int sdp_event_recv(struct sdp_sock *conn, struct ib_wc *comp); -int sdp_event_send(struct sdp_opt *conn, struct ib_wc *comp); +int sdp_event_send(struct sdp_sock *conn, struct ib_wc *comp); -int sdp_event_read(struct sdp_opt *conn, struct ib_wc *comp); +int sdp_event_read(struct sdp_sock *conn, struct ib_wc *comp); -int sdp_event_write(struct sdp_opt *conn, struct ib_wc *comp); +int sdp_event_write(struct sdp_sock *conn, struct ib_wc *comp); /* * DATA transport @@ -389,11 +389,11 @@ int sdp_inet_recv(struct kiocb *iocb, size_t size, int flags); -void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error); +void sdp_iocb_q_cancel_all_read(struct sdp_sock *conn, ssize_t error); -void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error); +void sdp_iocb_q_cancel_all_write(struct sdp_sock *conn, ssize_t error); -void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error); +void sdp_iocb_q_cancel_all(struct sdp_sock *conn, ssize_t error); /* * link address information @@ -423,7 +423,7 @@ void sdp_link_addr_cleanup(void); /* * Event handling function, demultiplexed base on Message ID */ -typedef int (*sdp_event_cb_func)(struct sdp_opt *conn, +typedef int (*sdp_event_cb_func)(struct sdp_sock *conn, struct sdpc_buff *buff); /* @@ -458,7 +458,7 @@ extern int sdp_debug_level; #define sdp_conn_dbg(level, type, conn, format, arg...) \ do { \ - struct sdp_opt *x = (conn); \ + struct sdp_sock *x = (conn); \ if (x) { \ sdp_dbg_out(level, type, \ "<%d> <%04x> " format, \ @@ -532,7 +532,7 @@ do { /* * sdp_inet_write_space - writable space on send side */ -static inline int sdp_inet_write_space(struct sdp_opt *conn, int urg) +static inline int sdp_inet_write_space(struct sdp_sock *conn, int urg) { int size; /* @@ -555,7 +555,7 @@ static inline int sdp_inet_write_space(s /* * sdp_inet_writable - return non-zero if socket is writable */ -static inline int sdp_inet_writable(struct sdp_opt *conn) +static inline int sdp_inet_writable(struct sdp_sock *conn) { if (conn->send_buf > 0) return (sdp_inet_write_space(conn, 0) < @@ -567,7 +567,7 @@ static inline int sdp_inet_writable(stru /* * sdp_conn_stat_dump - dump stats to the log */ -static inline void sdp_conn_stat_dump(struct sdp_opt *conn) +static inline void sdp_conn_stat_dump(struct sdp_sock *conn) { #ifdef _SDP_CONN_STATS_REC int counter; @@ -589,7 +589,7 @@ static inline void sdp_conn_stat_dump(st /* * sdp_conn_state_dump - dump state information to the log */ -static inline void sdp_conn_state_dump(struct sdp_opt *conn) +static inline void sdp_conn_state_dump(struct sdp_sock *conn) { #ifdef _SDP_CONN_STATE_REC int counter; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_event_read_advt - RDMA read event handler for source advertisments */ -static int sdp_event_read_advt(struct sdp_opt *conn, struct ib_wc *comp) +static int sdp_event_read_advt(struct sdp_sock *conn, struct ib_wc *comp) { struct sdpc_advt *advt; int result; @@ -106,7 +106,7 @@ error: /* * sdp_event_read - RDMA read event handler */ -int sdp_event_read(struct sdp_opt *conn, struct ib_wc *comp) +int sdp_event_read(struct sdp_sock *conn, struct ib_wc *comp) { struct sdpc_iocb *iocb; struct sdpc_buff *buff; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_send_buff_post - Post a buffer send on a SDP connection */ -static int sdp_send_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_send_buff_post(struct sdp_sock *conn, struct sdpc_buff *buff) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -178,7 +178,7 @@ done: /* * sdp_send_data_buff_post - Post data for buffered transmission */ -static int sdp_send_data_buff_post(struct sdp_opt *conn, +static int sdp_send_data_buff_post(struct sdp_sock *conn, struct sdpc_buff *buff) { int result; @@ -282,7 +282,7 @@ static int sdp_send_data_buff_post(struc /* * sdp_send_data_buff_snk - Post data for buffered transmission */ -static int sdp_send_data_buff_snk(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_send_data_buff_snk(struct sdp_sock *conn, struct sdpc_buff *buff) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -407,7 +407,7 @@ error: /* * sdp_send_data_iocb_snk - process a zcopy write advert in the data path */ -static int sdp_send_data_iocb_snk(struct sdp_opt *conn, struct sdpc_iocb *iocb) +static int sdp_send_data_iocb_snk(struct sdp_sock *conn, struct sdpc_iocb *iocb) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -521,7 +521,7 @@ error: /* * sdp_send_data_iocb_src - send a zcopy read advert in the data path */ -static int sdp_send_data_iocb_src(struct sdp_opt *conn, struct sdpc_iocb *iocb) +static int sdp_send_data_iocb_src(struct sdp_sock *conn, struct sdpc_iocb *iocb) { struct msg_hdr_srcah *src_ah; struct sdpc_buff *buff; @@ -737,7 +737,7 @@ static int sdp_send_iocb_buff_write(stru /* * sdp_send_data_iocb_buff - write multiple SDP buffers from an iocb */ -static int sdp_send_data_iocb_buff(struct sdp_opt *conn, struct sdpc_iocb *iocb) +static int sdp_send_data_iocb_buff(struct sdp_sock *conn, struct sdpc_iocb *iocb) { struct sdpc_buff *buff; int result; @@ -805,7 +805,7 @@ error: /* * sdp_send_data_iocb - Post IOCB data for transmission */ -static int sdp_send_data_iocb(struct sdp_opt *conn, struct sdpc_iocb *iocb) +static int sdp_send_data_iocb(struct sdp_sock *conn, struct sdpc_iocb *iocb) { int result = ENOBUFS; @@ -883,7 +883,7 @@ done: /* * sdp_send_data_queue_test - send data buffer if conditions are met */ -static int sdp_send_data_queue_test(struct sdp_opt *conn, +static int sdp_send_data_queue_test(struct sdp_sock *conn, struct sdpc_desc *element) { int result; @@ -914,7 +914,7 @@ static int sdp_send_data_queue_test(stru /* * sdp_send_data_queue_flush - Flush data from send queue, to send post */ -static int sdp_send_data_queue_flush(struct sdp_opt *conn) +static int sdp_send_data_queue_flush(struct sdp_sock *conn) { struct sdpc_desc *element; int result = 0; @@ -941,7 +941,7 @@ static int sdp_send_data_queue_flush(str /* * sdp_send_data_queue - send using the data queue if necessary */ -static int sdp_send_data_queue(struct sdp_opt *conn, struct sdpc_desc *element) +static int sdp_send_data_queue(struct sdp_sock *conn, struct sdpc_desc *element) { int result = 0; @@ -986,7 +986,7 @@ done: /* * sdp_send_data_buff_get - get an appropriate write buffer for send */ -static inline struct sdpc_buff *sdp_send_data_buff_get(struct sdp_opt *conn) +static inline struct sdpc_buff *sdp_send_data_buff_get(struct sdp_sock *conn) { struct sdpc_buff *buff; @@ -1011,7 +1011,7 @@ static inline struct sdpc_buff *sdp_send /* * sdp_send_data_buff_put - place a buffer into the send queue */ -static inline int sdp_send_data_buff_put(struct sdp_opt *conn, +static inline int sdp_send_data_buff_put(struct sdp_sock *conn, struct sdpc_buff *buff, int size, int urg) { @@ -1066,7 +1066,7 @@ static inline int sdp_send_data_buff_put /* * sdp_send_ctrl_buff_test - determine if it's OK to post a control msg */ -static int sdp_send_ctrl_buff_test(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_send_ctrl_buff_test(struct sdp_sock *conn, struct sdpc_buff *buff) { int result = 0; @@ -1092,7 +1092,7 @@ error: /* * sdp_send_ctrl_buff_flush - Flush control buffers, to send post */ -static int sdp_send_ctrl_buff_flush(struct sdp_opt *conn) +static int sdp_send_ctrl_buff_flush(struct sdp_sock *conn) { struct sdpc_desc *element; int result = 0; @@ -1119,7 +1119,7 @@ static int sdp_send_ctrl_buff_flush(stru /* * sdp_send_ctrl_buff_buffered - Send a buffered control message */ -static int sdp_send_ctrl_buff_buffered(struct sdp_opt *conn, +static int sdp_send_ctrl_buff_buffered(struct sdp_sock *conn, struct sdpc_buff *buff) { int result = 0; @@ -1149,7 +1149,7 @@ error: /* * sdp_send_ctrl_buff - Create and Send a buffered control message */ -static int sdp_send_ctrl_buff(struct sdp_opt *conn, u8 mid, int se, int sig) +static int sdp_send_ctrl_buff(struct sdp_sock *conn, u8 mid, int se, int sig) { int result = 0; struct sdpc_buff *buff; @@ -1201,7 +1201,7 @@ static int sdp_send_ctrl_buff(struct sdp /* * do_send_ctrl_disconnect - Send a disconnect request */ -static int do_send_ctrl_disconnect(struct sdp_opt *conn) +static int do_send_ctrl_disconnect(struct sdp_sock *conn) { int result = 0; struct sdpc_buff *buff; @@ -1247,7 +1247,7 @@ error: /* * sdp_send_ctrl_disconnect - potentially send a disconnect request */ -int sdp_send_ctrl_disconnect(struct sdp_opt *conn) +int sdp_send_ctrl_disconnect(struct sdp_sock *conn) { int result = 0; /* @@ -1270,7 +1270,7 @@ int sdp_send_ctrl_disconnect(struct sdp_ /* * sdp_send_ctrl_ack - Send a gratuitous Ack */ -int sdp_send_ctrl_ack(struct sdp_opt *conn) +int sdp_send_ctrl_ack(struct sdp_sock *conn) { /* * The gratuitous ack is not really and ack, but an update of the @@ -1292,7 +1292,7 @@ int sdp_send_ctrl_ack(struct sdp_opt *co /* * sdp_send_ctrl_send_sm - Send a request for buffered mode */ -int sdp_send_ctrl_send_sm(struct sdp_opt *conn) +int sdp_send_ctrl_send_sm(struct sdp_sock *conn) { return sdp_send_ctrl_buff(conn, SDP_MID_SEND_SM, 1, 1); } @@ -1300,7 +1300,7 @@ int sdp_send_ctrl_send_sm(struct sdp_opt /* * sdp_send_ctrl_src_cancel - Send a source cancel */ -int sdp_send_ctrl_src_cancel(struct sdp_opt *conn) +int sdp_send_ctrl_src_cancel(struct sdp_sock *conn) { return sdp_send_ctrl_buff(conn, SDP_MID_SRC_CANCEL, 1, 1); } @@ -1308,7 +1308,7 @@ int sdp_send_ctrl_src_cancel(struct sdp_ /* * sdp_send_ctrl_snk_cancel - Send a sink cancel */ -int sdp_send_ctrl_snk_cancel(struct sdp_opt *conn) +int sdp_send_ctrl_snk_cancel(struct sdp_sock *conn) { return sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL, 1, 1); } @@ -1316,7 +1316,7 @@ int sdp_send_ctrl_snk_cancel(struct sdp_ /* * sdp_send_ctrl_snk_cancel_ack - Send an ack for a sink cancel */ -int sdp_send_ctrl_snk_cancel_ack(struct sdp_opt *conn) +int sdp_send_ctrl_snk_cancel_ack(struct sdp_sock *conn) { return sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL_ACK, 1, 1); } @@ -1324,7 +1324,7 @@ int sdp_send_ctrl_snk_cancel_ack(struct /* * sdp_send_ctrl_abort - Send an abort message */ -int sdp_send_ctrl_abort(struct sdp_opt *conn) +int sdp_send_ctrl_abort(struct sdp_sock *conn) { /* * send @@ -1335,7 +1335,7 @@ int sdp_send_ctrl_abort(struct sdp_opt * /* * sdp_send_ctrl_resize_buff_ack - Send an ack for a buffer size change */ -int sdp_send_ctrl_resize_buff_ack(struct sdp_opt *conn, u32 size) +int sdp_send_ctrl_resize_buff_ack(struct sdp_sock *conn, u32 size) { struct msg_hdr_crbah *crbah; struct sdpc_buff *buff; @@ -1386,7 +1386,7 @@ error: /* * sdp_send_ctrl_rdma_rd - Send an rdma read completion */ -int sdp_send_ctrl_rdma_rd(struct sdp_opt *conn, s32 size) +int sdp_send_ctrl_rdma_rd(struct sdp_sock *conn, s32 size) { struct msg_hdr_rrch *rrch; struct sdpc_buff *buff; @@ -1455,7 +1455,7 @@ error: /* * sdp_send_ctrl_rdma_wr - Send an rdma write completion */ -int sdp_send_ctrl_rdma_wr(struct sdp_opt *conn, u32 size) +int sdp_send_ctrl_rdma_wr(struct sdp_sock *conn, u32 size) { struct msg_hdr_rwch *rwch; struct sdpc_buff *buff; @@ -1512,7 +1512,7 @@ error: /* * sdp_send_ctrl_snk_avail - Send a sink available message */ -int sdp_send_ctrl_snk_avail(struct sdp_opt *conn, u32 size, u32 rkey, u64 addr) +int sdp_send_ctrl_snk_avail(struct sdp_sock *conn, u32 size, u32 rkey, u64 addr) { struct msg_hdr_snkah *snkah; struct sdpc_buff *buff; @@ -1573,7 +1573,7 @@ error: /* * sdp_send_ctrl_mode_ch - Send a mode change command */ -int sdp_send_ctrl_mode_ch(struct sdp_opt *conn, u8 mode) +int sdp_send_ctrl_mode_ch(struct sdp_sock *conn, u8 mode) { struct msg_hdr_mch *mch; struct sdpc_buff *buff; @@ -1679,7 +1679,7 @@ static int sdp_write_src_lookup(struct s static int sdp_inet_write_cancel(struct kiocb *req, struct io_event *ev) { struct sock_iocb *si = kiocb_to_siocb(req); - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_iocb *iocb; int result = 0; @@ -1812,7 +1812,7 @@ done: /* * sdp_send_flush_advt - Flush passive sink advertisments */ -static int sdp_send_flush_advt(struct sdp_opt *conn) +static int sdp_send_flush_advt(struct sdp_sock *conn) { struct sdpc_advt *advt; int result; @@ -1850,7 +1850,7 @@ static int sdp_send_flush_advt(struct sd /* * sdp_send_flush - Flush buffers from send queue, in to send post */ -int sdp_send_flush(struct sdp_opt *conn) +int sdp_send_flush(struct sdp_sock *conn) { int result = 0; @@ -1912,7 +1912,7 @@ int sdp_inet_send(struct kiocb *req, str size_t size) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_buff *buff; struct sdpc_iocb *iocb; int result = 0; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (working copy) @@ -38,7 +38,7 @@ /* * Connection establishment functions */ -void sdp_cm_actv_error(struct sdp_opt *conn, int error) +void sdp_cm_actv_error(struct sdp_sock *conn, int error) { int result; struct sock *sk; @@ -105,7 +105,7 @@ void sdp_cm_actv_error(struct sdp_opt *c /* * sdp_cm_actv_establish - process an accepted connection request. */ -static int sdp_cm_actv_establish(struct sdp_opt *conn) +static int sdp_cm_actv_establish(struct sdp_sock *conn) { struct ib_qp_attr *qp_attr; int attr_mask = 0; @@ -263,7 +263,7 @@ static int sdp_cm_hello_ack_check(struct * sdp_cm_rep_handler - handler for active connection open completion */ int sdp_cm_rep_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { struct sdp_msg_hello_ack *hello_ack; int result = -ECONNRESET; @@ -343,7 +343,7 @@ static void sdp_cm_path_complete(u64 id, { struct ib_cm_req_param param; struct sdp_msg_hello *hello_msg; - struct sdp_opt *conn = (struct sdp_opt *) arg; + struct sdp_sock *conn = (struct sdp_sock *) arg; struct sdpc_buff *buff; int result = 0; /* @@ -513,7 +513,7 @@ done: /* * sdp_cm_connect - initiate a SDP connection with a hello message. */ -int sdp_cm_connect(struct sdp_opt *conn) +int sdp_cm_connect(struct sdp_sock *conn) { int result; /* Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -74,7 +74,7 @@ static u32 sdp_psn_generate(void) return psn; } -void sdp_conn_inet_error(struct sdp_opt *conn, int error) +void sdp_conn_inet_error(struct sdp_sock *conn, int error) { struct sock *sk; @@ -99,7 +99,7 @@ void sdp_conn_inet_error(struct sdp_opt sdp_inet_wake_error(sk); } -void sdp_conn_abort(struct sdp_opt *conn) +void sdp_conn_abort(struct sdp_sock *conn) { int result; int error = ECONNRESET; @@ -175,10 +175,10 @@ void sdp_conn_abort(struct sdp_opt *conn /* * sdp_inet_accept_q_put - put a conn into a listen conn's accept Q. */ -int sdp_inet_accept_q_put(struct sdp_opt *listen_conn, - struct sdp_opt *accept_conn) +int sdp_inet_accept_q_put(struct sdp_sock *listen_conn, + struct sdp_sock *accept_conn) { - struct sdp_opt *next_conn; + struct sdp_sock *next_conn; if (listen_conn->parent || accept_conn->parent || @@ -206,10 +206,10 @@ int sdp_inet_accept_q_put(struct sdp_opt /* * sdp_inet_accept_q_get - get a conn from a listen conn's accept Q. */ -struct sdp_opt *sdp_inet_accept_q_get(struct sdp_opt *listen_conn) +struct sdp_sock *sdp_inet_accept_q_get(struct sdp_sock *listen_conn) { - struct sdp_opt *prev_conn; - struct sdp_opt *accept_conn; + struct sdp_sock *prev_conn; + struct sdp_sock *accept_conn; if (listen_conn->parent || !listen_conn->accept_next || @@ -245,10 +245,10 @@ struct sdp_opt *sdp_inet_accept_q_get(st /* * sdp_inet_accept_q_remove - remove a conn from a conn's accept Q. */ -int sdp_inet_accept_q_remove(struct sdp_opt *accept_conn) +int sdp_inet_accept_q_remove(struct sdp_sock *accept_conn) { - struct sdp_opt *next_conn; - struct sdp_opt *prev_conn; + struct sdp_sock *next_conn; + struct sdp_sock *prev_conn; if (!accept_conn->parent) return -EFAULT; @@ -282,7 +282,7 @@ int sdp_inet_accept_q_remove(struct sdp_ /* * sdp_inet_listen_start - start listening for new connections on a socket */ -int sdp_inet_listen_start(struct sdp_opt *conn) +int sdp_inet_listen_start(struct sdp_sock *conn) { unsigned long flags; @@ -315,9 +315,9 @@ int sdp_inet_listen_start(struct sdp_opt /* * sdp_inet_listen_stop - stop listening for new connections on a socket */ -int sdp_inet_listen_stop(struct sdp_opt *listen_conn) +int sdp_inet_listen_stop(struct sdp_sock *listen_conn) { - struct sdp_opt *accept_conn; + struct sdp_sock *accept_conn; unsigned long flags; if (listen_conn->state != SDP_CONN_ST_LISTEN) { @@ -368,9 +368,9 @@ int sdp_inet_listen_stop(struct sdp_opt /* * sdp_inet_listen_lookup - lookup a connection in the listen list */ -struct sdp_opt *sdp_inet_listen_lookup(u32 addr, u16 port) +struct sdp_sock *sdp_inet_listen_lookup(u32 addr, u16 port) { - struct sdp_opt *conn; + struct sdp_sock *conn; unsigned long flags; /* * table lock @@ -393,11 +393,11 @@ struct sdp_opt *sdp_inet_listen_lookup(u /* * sdp_inet_port_get - bind a socket to a port. */ -int sdp_inet_port_get(struct sdp_opt *conn, u16 port) +int sdp_inet_port_get(struct sdp_sock *conn, u16 port) { struct sock *sk; struct sock *srch; - struct sdp_opt *look; + struct sdp_sock *look; s32 counter; s32 low_port; s32 top_port; @@ -515,7 +515,7 @@ done: /* * sdp_inet_port_put - unbind a socket from a port. */ -int sdp_inet_port_put(struct sdp_opt *conn) +int sdp_inet_port_put(struct sdp_sock *conn) { unsigned long flags; @@ -544,7 +544,7 @@ int sdp_inet_port_put(struct sdp_opt *co /* * sdp_inet_port_inherit - inherit a port from another socket (accept) */ -int sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child) +int sdp_inet_port_inherit(struct sdp_sock *parent, struct sdp_sock *child) { int result; unsigned long flags; @@ -580,7 +580,7 @@ done: /* * sdp_conn_table_insert - insert a connection into the connection table */ -static int sdp_conn_table_insert(struct sdp_opt *conn) +static int sdp_conn_table_insert(struct sdp_sock *conn) { u32 counter; int result = -ENOMEM; @@ -624,7 +624,7 @@ static int sdp_conn_table_insert(struct /* * sdp_conn_table_remove - remove a connection from the connection table */ -int sdp_conn_table_remove(struct sdp_opt *conn) +int sdp_conn_table_remove(struct sdp_sock *conn) { int result = 0; unsigned long flags; @@ -661,9 +661,9 @@ done: /* * sdp_conn_table_lookup - look up connection in the connection table */ -struct sdp_opt *sdp_conn_table_lookup(s32 entry) +struct sdp_sock *sdp_conn_table_lookup(s32 entry) { - struct sdp_opt *conn; + struct sdp_sock *conn; unsigned long flags; /* * lock table @@ -711,7 +711,7 @@ static void sdp_desc_q_cancel_iocb(struc } } -void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error) +void sdp_iocb_q_cancel_all_read(struct sdp_sock *conn, ssize_t error) { sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error); sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error); @@ -719,7 +719,7 @@ void sdp_iocb_q_cancel_all_read(struct s sdp_desc_q_cancel_iocb(&conn->r_src, error); } -void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error) +void sdp_iocb_q_cancel_all_write(struct sdp_sock *conn, ssize_t error) { sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error); @@ -727,7 +727,7 @@ void sdp_iocb_q_cancel_all_write(struct sdp_desc_q_cancel_iocb(&conn->w_snk, error); } -void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error) +void sdp_iocb_q_cancel_all(struct sdp_sock *conn, ssize_t error) { sdp_iocb_q_cancel_all_read(conn, error); sdp_iocb_q_cancel_all_write(conn, error); @@ -740,7 +740,7 @@ void sdp_iocb_q_cancel_all(struct sdp_op /* * sdp_conn_destruct - final destructor for connection. */ -void sdp_conn_destruct(struct sdp_opt *conn) +void sdp_conn_destruct(struct sdp_sock *conn) { int dump = 0; int result; @@ -852,7 +852,7 @@ void sdp_conn_destruct(struct sdp_opt *c /* * sdp_conn_internal_lock - lock the connection (use only from macro) */ -void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags) +void sdp_conn_internal_lock(struct sdp_sock *conn, unsigned long *flags) { DECLARE_WAITQUEUE(wait, current); unsigned long f = *flags; @@ -876,7 +876,7 @@ void sdp_conn_internal_lock(struct sdp_o /* * sdp_conn_relock - test the connection (use only from macro) */ -void sdp_conn_relock(struct sdp_opt *conn) +void sdp_conn_relock(struct sdp_sock *conn) { unsigned long flags; struct ib_wc entry; @@ -940,7 +940,7 @@ void sdp_conn_relock(struct sdp_opt *con /* * sdp_conn_cq_drain - drain one of the the connection's CQs */ -int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn) +int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_sock *conn) { struct ib_wc entry; int result; @@ -992,7 +992,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, /* * sdp_conn_internal_unlock - lock the connection (use only from macro) */ -void sdp_conn_internal_unlock(struct sdp_opt *conn) +void sdp_conn_internal_unlock(struct sdp_sock *conn) { int calls = 0; /* @@ -1010,7 +1010,7 @@ void sdp_conn_internal_unlock(struct sdp /* * sdp_conn_lock_init - initialize connection lock */ -static void sdp_conn_lock_init(struct sdp_opt *conn) +static void sdp_conn_lock_init(struct sdp_sock *conn) { spin_lock_init(&(conn->lock.slock)); conn->lock.users = 0; @@ -1021,7 +1021,7 @@ static void sdp_conn_lock_init(struct sd /* * sdp_conn_alloc_ib - allocate IB structures for a new connection. */ -int sdp_conn_alloc_ib(struct sdp_opt *conn, struct ib_device *device, +int sdp_conn_alloc_ib(struct sdp_sock *conn, struct ib_device *device, u8 hw_port, u16 pkey) { struct ib_qp_init_attr *init_attr; @@ -1196,9 +1196,9 @@ extern struct proto sdp_sk_proto; /* * sdp_conn_alloc - allocate a new socket, and init. */ -struct sdp_opt *sdp_conn_alloc(int priority) +struct sdp_sock *sdp_conn_alloc(int priority) { - struct sdp_opt *conn; + struct sdp_sock *conn; struct sock *sk; int result; @@ -1393,7 +1393,7 @@ error: int sdp_proc_dump_conn_main(char *buffer, int max_size, off_t start_index, long *end_index) { - struct sdp_opt *conn; + struct sdp_sock *conn; off_t counter = 0; int offset = 0; u64 s_guid; @@ -1498,7 +1498,7 @@ int sdp_proc_dump_conn_data(char *buffer long *end_index) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; off_t counter = 0; int offset = 0; unsigned long flags; @@ -1591,7 +1591,7 @@ done: int sdp_proc_dump_conn_rdma(char *buffer, int max_size, off_t start_index, long *end_index) { - struct sdp_opt *conn; + struct sdp_sock *conn; off_t counter = 0; int offset = 0; unsigned long flags; @@ -1668,7 +1668,7 @@ done: int sdp_proc_dump_conn_sopt(char *buffer, int max_size, off_t start_index, long *end_index) { - struct sdp_opt *conn; + struct sdp_sock *conn; off_t counter = 0; int offset = 0; unsigned long flags; @@ -1986,7 +1986,7 @@ int sdp_conn_table_init(int proto_family goto error_size; } - byte_size = conn_size * sizeof(struct sdp_opt *); + byte_size = conn_size * sizeof(struct sdp_sock *); page_size = (byte_size >> 12) + ((0xfff & byte_size) > 0 ? 1 : 0); for (dev_root_s.sk_ordr = 0; (1 << dev_root_s.sk_ordr) < page_size; dev_root_s.sk_ordr++) ; @@ -2058,7 +2058,7 @@ void sdp_conn_table_clear(void) { sdp_dbg_init("Deleting connection tables."); #if 0 - struct sdp_opt *conn; + struct sdp_sock *conn; /* * drain all the connections */ Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_post_recv_buff - post a single buffers for data recv */ -static int sdp_post_recv_buff(struct sdp_opt *conn) +static int sdp_post_recv_buff(struct sdp_sock *conn) { struct ib_recv_wr receive_param = { NULL }; struct ib_recv_wr *bad_wr; @@ -111,7 +111,7 @@ error: /* * sdp_post_rdma_buff - post a single buffers for rdma read on a conn */ -static int sdp_post_rdma_buff(struct sdp_opt *conn) +static int sdp_post_rdma_buff(struct sdp_sock *conn) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -202,7 +202,7 @@ drop: /* * sdp_post_rdma_iocb_src - post a iocb for rdma read on a conn */ -static int sdp_post_rdma_iocb_src(struct sdp_opt *conn) +static int sdp_post_rdma_iocb_src(struct sdp_sock *conn) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -311,7 +311,7 @@ error: /* * sdp_post_rdma_iocb_snk - post a iocb for rdma read on a conn */ -static int sdp_post_rdma_iocb_snk(struct sdp_opt *conn) +static int sdp_post_rdma_iocb_snk(struct sdp_sock *conn) { struct sdpc_iocb *iocb; int result = 0; @@ -379,7 +379,7 @@ error: /* * sdp_post_rdma - post a rdma based requests for a connection */ -static int sdp_post_rdma(struct sdp_opt *conn) +static int sdp_post_rdma(struct sdp_sock *conn) { int result = 0; @@ -458,7 +458,7 @@ done: /* * sdp_recv_flush - post a certain number of buffers on a connection */ -int sdp_recv_flush(struct sdp_opt *conn) +int sdp_recv_flush(struct sdp_sock *conn) { int result = 0; int counter; @@ -625,7 +625,7 @@ static int sdp_read_buff_iocb(struct sdp /* * sdp_recv_buff_iocb_active - Ease AIO read pending pressure */ -static int sdp_recv_buff_iocb_active(struct sdp_opt *conn, +static int sdp_recv_buff_iocb_active(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; @@ -671,7 +671,7 @@ static int sdp_recv_buff_iocb_active(str /* * sdp_recv_buff_iocb_pending - Ease AIO read pending pressure */ -static int sdp_recv_buff_iocb_pending(struct sdp_opt *conn, +static int sdp_recv_buff_iocb_pending(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; @@ -725,7 +725,7 @@ static int sdp_recv_buff_iocb_pending(st /* * sdp_recv_buff - Process a new buffer based on queue type */ -int sdp_recv_buff(struct sdp_opt *conn, struct sdpc_buff *buff) +int sdp_recv_buff(struct sdp_sock *conn, struct sdpc_buff *buff) { int result; int buffered; @@ -835,7 +835,7 @@ static int sdp_read_src_lookup(struct sd static int sdp_inet_read_cancel(struct kiocb *req, struct io_event *ev) { struct sock_iocb *si = kiocb_to_siocb(req); - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_iocb *iocb; int result = 0; @@ -992,7 +992,7 @@ static int sdp_inet_recv_urg_trav(struct static int sdp_inet_recv_urg(struct sock *sk, struct msghdr *msg, int size, int flags) { - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_buff *buff; int result = 0; u8 value; @@ -1061,7 +1061,7 @@ int sdp_inet_recv(struct kiocb *req, st size_t size, int flags) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_iocb *iocb; struct sdpc_buff *buff; struct sdpc_buff *head = NULL; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -109,12 +109,12 @@ enum sdp_mode { */ #define SDP_MSG_EVENT_TABLE_SIZE 0x20 -static inline struct sdp_opt *sdp_sk(struct sock *sk) +static inline struct sdp_sock *sdp_sk(struct sock *sk) { - return (struct sdp_opt *)sk; + return (struct sdp_sock *)sk; } -static inline struct sock *sk_sdp(struct sdp_opt *conn) +static inline struct sock *sk_sdp(struct sdp_sock *conn) { return (struct sock *)conn; } @@ -181,9 +181,9 @@ struct sdp_conn_lock { /* * SDP Connection structure. */ -struct sdp_opt { +struct sdp_sock { /* - * inet_sock must be first member of sdp_opt + * inet_sock must be first member of sdp_sock * NOTE: this depends on inet_sock having struct sock as its * first member */ @@ -343,17 +343,17 @@ struct sdp_opt { /* * table managment */ - struct sdp_opt *lstn_next; /* next conn in the chain */ - struct sdp_opt **lstn_p_next; /* previous next conn in the chain */ + struct sdp_sock *lstn_next; /* next conn in the chain */ + struct sdp_sock **lstn_p_next; /* previous next conn in the chain */ - struct sdp_opt *bind_next; /* next conn in the chain */ - struct sdp_opt **bind_p_next; /* previous next conn in the chain */ + struct sdp_sock *bind_next; /* next conn in the chain */ + struct sdp_sock **bind_p_next; /* previous next conn in the chain */ /* * listen/accept managment */ - struct sdp_opt *parent; /* listening socket queuing. */ - struct sdp_opt *accept_next; /* sockets waiting for acceptance. */ - struct sdp_opt *accept_prev; /* sockets waiting for acceptance. */ + struct sdp_sock *parent; /* listening socket queuing. */ + struct sdp_sock *accept_next; /* sockets waiting for acceptance. */ + struct sdp_sock *accept_prev; /* sockets waiting for acceptance. */ /* * OS info */ @@ -434,18 +434,18 @@ struct sdp_opt { /* * SDP connection lock */ -extern void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags); -extern void sdp_conn_internal_unlock(struct sdp_opt *conn); -extern void sdp_conn_relock(struct sdp_opt *conn); -extern void sdp_conn_destruct(struct sdp_opt *conn); -extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn); +extern void sdp_conn_internal_lock(struct sdp_sock *conn, unsigned long *flags); +extern void sdp_conn_internal_unlock(struct sdp_sock *conn); +extern void sdp_conn_relock(struct sdp_sock *conn); +extern void sdp_conn_destruct(struct sdp_sock *conn); +extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_sock *conn); #define SDP_CONN_LOCK_IRQ(conn, flags) \ spin_lock_irqsave(&((conn)->lock.slock), flags) #define SDP_CONN_UNLOCK_IRQ(conn, flags) \ spin_unlock_irqrestore(&((conn)->lock.slock), flags) -static inline void sdp_conn_lock(struct sdp_opt *conn) +static inline void sdp_conn_lock(struct sdp_sock *conn) { unsigned long flags; @@ -461,7 +461,7 @@ static inline void sdp_conn_lock(struct spin_unlock_irqrestore(&(conn->lock.slock), flags); } -static inline void sdp_conn_unlock(struct sdp_opt *conn) +static inline void sdp_conn_unlock(struct sdp_sock *conn) { unsigned long flags; @@ -480,12 +480,12 @@ static inline void sdp_conn_unlock(struc /* * connection reference counting. */ -static inline void sdp_conn_hold(struct sdp_opt *conn) +static inline void sdp_conn_hold(struct sdp_sock *conn) { atomic_inc(&conn->refcnt); } -static inline void sdp_conn_put(struct sdp_opt *conn) +static inline void sdp_conn_put(struct sdp_sock *conn) { if (atomic_dec_and_test(&conn->refcnt)) sdp_conn_destruct(conn); @@ -494,7 +494,7 @@ static inline void sdp_conn_put(struct s /* * sdp_conn_error - get the connections error value destructively */ -static inline int sdp_conn_error(struct sdp_opt *conn) +static inline int sdp_conn_error(struct sdp_sock *conn) { /* * The connection error parameter is set and read under the connection Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (working copy) @@ -38,7 +38,7 @@ /* * handle incoming passive connection establishment. (RTU) */ -int sdp_cm_pass_establish(struct sdp_opt *conn) +int sdp_cm_pass_establish(struct sdp_sock *conn) { struct ib_qp_attr *qp_attr; int attr_mask = 0; @@ -101,7 +101,7 @@ done: /* * Functions to handle incoming passive connection requests. (REQ) */ -static int sdp_cm_accept(struct sdp_opt *conn) +static int sdp_cm_accept(struct sdp_sock *conn) { struct ib_cm_rep_param param; struct sdp_msg_hello_ack *hello_ack; @@ -230,9 +230,9 @@ error: return result; } -static int sdp_cm_listen_lookup(struct sdp_opt *conn) +static int sdp_cm_listen_lookup(struct sdp_sock *conn) { - struct sdp_opt *listen_conn; + struct sdp_sock *listen_conn; struct sock *listen_sk; struct sock *sk; int result; @@ -416,7 +416,7 @@ static int sdp_cm_hello_check(struct sdp int sdp_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) { struct sdp_msg_hello *msg_hello = event->private_data; - struct sdp_opt *conn; + struct sdp_sock *conn; int result; sdp_dbg_ctrl(NULL, Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_sent.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_sent.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_sent.c (working copy) @@ -39,7 +39,7 @@ * Specific MID handler functions. (SEND) */ -static int sdp_sent_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_sent_disconnect(struct sdp_sock *conn, struct sdpc_buff *buff) { int result; @@ -95,7 +95,7 @@ error: return result; } -static int sdp_sent_abort(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_sent_abort(struct sdp_sock *conn, struct sdpc_buff *buff) { int result; @@ -115,7 +115,7 @@ static int sdp_sent_abort(struct sdp_opt /* * sdp_event_send - send event handler */ -int sdp_event_send(struct sdp_opt *conn, struct ib_wc *comp) +int sdp_event_send(struct sdp_sock *conn, struct ib_wc *comp) { struct sdpc_buff *head = NULL; struct sdpc_buff *buff; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c (working copy) @@ -413,7 +413,7 @@ static void sdp_mem_lock_cleanup(void) /* * sdp_iocb_register - register an IOCBs memory for advertisment */ -int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_opt *conn) +int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_sock *conn) { int result; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_event.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_event.c (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_event.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_cq_event_locked - main per QP event handler */ -int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_opt *conn) +int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_sock *conn) { int result = 0; @@ -127,7 +127,7 @@ done: void sdp_cq_event_handler(struct ib_cq *cq, void *arg) { s32 hashent = (unsigned long)arg; - struct sdp_opt *conn; + struct sdp_sock *conn; u16 event; unsigned long flags; @@ -183,7 +183,7 @@ void sdp_cq_event_handler(struct ib_cq * sdp_conn_put(conn); } -static void sdp_cm_to_error(struct sdp_opt *conn) +static void sdp_cm_to_error(struct sdp_sock *conn) { sdp_conn_inet_error(conn, ECONNRESET); conn->cm_id = NULL; @@ -194,7 +194,7 @@ static void sdp_cm_to_error(struct sdp_o */ static int sdp_cm_idle(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { sdp_dbg_ctrl(conn, "CM IDLE. commID <%08x> event <%d> status <%d>", cm_id->local_id, event->event, event->param.send_status); @@ -234,7 +234,7 @@ static int sdp_cm_idle(struct ib_cm_id * static int sdp_cm_established(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { int result = 0; @@ -292,7 +292,7 @@ error: } static int sdp_cm_dreq_rcvd(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { int result = 0; @@ -313,7 +313,7 @@ static int sdp_cm_dreq_rcvd(struct ib_cm * sdp_cm_timewait - handler for connection Time Wait completion */ static int sdp_cm_timewait(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { int result = 0; @@ -381,7 +381,7 @@ static int sdp_cm_timewait(struct ib_cm_ int sdp_cm_event_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) { s32 hashent = (unsigned long)cm_id->context; - struct sdp_opt *conn = NULL; + struct sdp_sock *conn = NULL; int result = 0; sdp_dbg_ctrl(NULL, "CM state <%d> event <%d> commID <%08x> ID <%d>", Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (revision 2670) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (working copy) @@ -182,12 +182,12 @@ struct sdev_root { u32 sk_ordr; /* order size of region. */ u32 sk_rover; /* next potential available space. */ u32 sk_entry; /* number of socket table entries. */ - struct sdp_opt **sk_array; /* array of sockets. */ + struct sdp_sock **sk_array; /* array of sockets. */ /* * connection managment */ - struct sdp_opt *listen_list; /* list of listening connections */ - struct sdp_opt *bind_list; /* connections bound to a port. */ + struct sdp_sock *listen_list; /* list of listening connections */ + struct sdp_sock *bind_list; /* connections bound to a port. */ /* * list locks */ From jlentini at netapp.com Wed Jun 22 08:59:33 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 11:59:33 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [AT] Initial changes to work with user AT In-Reply-To: <1118936120.4506.1267.camel@localhost.localdomain> References: <1118936120.4506.1267.camel@localhost.localdomain> Message-ID: Committed in revision 2671 On Thu, 16 Jun 2005, Hal Rosenstock wrote: halr> Initial changes to work with user AT. halr> Also, some other changes based on code inspection. halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: at.c halr> =================================================================== halr> --- at.c (revision 2610) halr> +++ at.c (working copy) halr> @@ -118,7 +118,7 @@ halr> int sa_id; halr> }; halr> halr> -struct async pending_reqs; /* dummy head for cyclic list */ halr> +static struct async pending_reqs; /* dummy head for cyclic list */ halr> halr> struct ib_at_src { halr> u32 ip; halr> @@ -286,7 +286,7 @@ halr> halr> spin_lock_irqsave(&pending_reqs.lock, flags); halr> new_id = ++req_id; halr> - if (!new_id) halr> + if (!new_id) /* 0 is not used as req_id (reserved value) */ halr> new_id = ++req_id; halr> spin_unlock_irqrestore(&pending_reqs.lock, flags); halr> halr> @@ -319,9 +319,13 @@ halr> break; halr> default: halr> WARN("bad async req type %d", pend->type); halr> + if (pend->sa_query) { halr> + ib_sa_cancel_query(pend->sa_id, pend->sa_query); halr> + pend->sa_query = NULL; halr> + } halr> pend->status = IB_AT_STATUS_INVALID; halr> pend->type = IBAT_REQ_NONE; halr> - pend->sa_query = NULL; halr> + break; halr> } halr> } halr> halr> @@ -377,16 +381,16 @@ halr> halr> DEBUG("pend %p nrec %d async %p", pend, nrec, q); halr> halr> + if (pend->sa_query) { halr> + ib_sa_cancel_query(pend->sa_id, pend->sa_query); halr> + pend->sa_query = NULL; halr> + } halr> + halr> if (pend->status != IB_AT_STATUS_PENDING) halr> WARN("pend %p already completed? status %d", pend, pend->status); halr> halr> pend->status = nrec < 0 ? IB_AT_STATUS_ERROR : IB_AT_STATUS_COMPLETED; halr> halr> - if (pend->sa_query) { halr> - ib_sa_cancel_query(pend->sa_id, pend->sa_query); halr> - pend->sa_query = NULL; halr> - } halr> - halr> if (q) halr> spin_lock_irqsave(&q->lock, flags); halr> halr> @@ -464,9 +468,12 @@ halr> halr> DEBUG("free async %p req %p", async, req); halr> halr> + if (req->pend.sa_query) { halr> + ib_sa_cancel_query(req->pend.sa_id, req->pend.sa_query); halr> + req->pend.sa_query = NULL; halr> + } halr> req->pend.status = IB_AT_STATUS_INVALID; halr> req->pend.type = IBAT_REQ_NONE; halr> - req->pend.sa_query = NULL; halr> halr> kmem_cache_free(route_req_cache, req); halr> } halr> @@ -477,9 +484,12 @@ halr> halr> DEBUG("free async %p req %p", async, req); halr> halr> + if (req->pend.sa_query) { halr> + ib_sa_cancel_query(req->pend.sa_id, req->pend.sa_query); halr> + req->pend.sa_query = NULL; halr> + } halr> req->pend.status = IB_AT_STATUS_INVALID; halr> req->pend.type = IBAT_REQ_NONE; halr> - req->pend.sa_query = NULL; halr> halr> kmem_cache_free(path_req_cache, req); halr> } halr> @@ -494,7 +504,7 @@ halr> DEBUG("lookup in q %p pending %p", q, new); halr> spin_lock_irqsave(&q->lock, flags); halr> for (a = q->next; a != q; a = a->next) { halr> - DEBUG("%d %d", a->type, type); halr> + DEBUG("req type %d %d", a->type, type); halr> if (a->type == type && same_fn(a, new)) halr> break; halr> } halr> @@ -508,7 +518,7 @@ halr> unsigned long flags; halr> struct async *a; halr> halr> - DEBUG("lookup in q %p id %llx", q, id); halr> + DEBUG("lookup in q %p id 0x%llx", q, id); halr> spin_lock_irqsave(&q->lock, flags); halr> for (a = q->next; a != q; a = a->next) halr> if (a->id == id) halr> @@ -551,8 +561,7 @@ halr> { halr> int n = min(npath, nelem); halr> halr> - DEBUG("fill ib_sa_path_rec %p output %d records", out, n); halr> - halr> + DEBUG("output ib_sa_path_rec %p %d records", out, n); halr> memcpy(out, resp, n * sizeof (struct ib_sa_path_rec)); halr> return n; halr> } halr> @@ -589,15 +598,15 @@ halr> halr> DEBUG("req %p status %d", req, status); halr> halr> + req->pend.sa_query = NULL; halr> + halr> if (req->pend.parent) { halr> WARN("for child req %p???", req); halr> return; halr> } halr> halr> - req->pend.sa_query = NULL; halr> - halr> if (status) { halr> - DEBUG("status %d - check if should retry", status); halr> + DEBUG("status %d - checking if should retry", status); halr> if (status == -ETIMEDOUT && halr> jiffies - req->pend.start < IB_AT_REQ_TIMEOUT) halr> resolve_path(req); halr> @@ -625,7 +634,7 @@ halr> struct path_req *preq; halr> unsigned long flags; halr> halr> - DEBUG("start sweeping"); halr> + DEBUG("start sweep"); halr> halr> spin_lock_irqsave(&pending_reqs.lock, flags); halr> for (pend = pending_reqs.next; pend != &pending_reqs; pend = next) { halr> @@ -638,7 +647,7 @@ halr> halr> DEBUG("examining route req %p pend %p", req, pend); halr> if (jiffies > pend->start + IB_AT_REQ_TIMEOUT) { halr> - DEBUG("req delete <%d.%d.%d.%d> <%lu:%lu>", halr> + DEBUG("delete route <%d.%d.%d.%d> <%lu:%lu>", halr> (req->dst_ip & 0x000000ff), halr> (req->dst_ip & 0x0000ff00) >> 8, halr> (req->dst_ip & 0x00ff0000) >> 16, halr> @@ -653,7 +662,7 @@ halr> halr> DEBUG("examining path req %p pend %p", preq, pend); halr> if (jiffies > pend->start + IB_AT_REQ_TIMEOUT) { halr> - DEBUG("req delete path <%lu:%lu>", halr> + DEBUG("delete path <%lu:%lu>", halr> jiffies, pend->start); halr> halr> req_end(pend, -ETIMEDOUT, NULL); halr> @@ -661,6 +670,7 @@ halr> break; halr> default: halr> WARN("unknown async req type %d", pend->type); halr> + break; halr> } halr> } halr> halr> @@ -722,7 +732,8 @@ halr> &req->pend.sa_query); halr> halr> if (req->pend.sa_id < 0) { halr> - WARN("ib_sa_path_rec_get %d", req->pend.sa_id); halr> + WARN("ib_sa_path_rec_get failed %d", req->pend.sa_id); halr> + req->pend.sa_query = NULL; halr> return req->pend.sa_id; halr> } halr> halr> halr> halr> From roland at topspin.com Wed Jun 22 09:00:55 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 09:00:55 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050622120743.GZ20041@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 22 Jun 2005 15:07:44 +0300") References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> Message-ID: <52zmtis7ns.fsf@topspin.com> Michael> Roland, is there some way to get the plugin directory out Michael> of the libibverbs library? Sorry, not sure what you mean. - R. From jlentini at netapp.com Wed Jun 22 09:16:30 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 12:16:30 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove the remaining typedef enums In-Reply-To: <1118946957.21846.14.camel@duffman> References: <1118946957.21846.14.camel@duffman> Message-ID: Committed in revision 2672 with one minor change. I changed the DAPL_ASYNC_HANDLER_TYPE to be an enum dapl_async_handler_type for consistency. On Thu, 16 Jun 2005, Tom Duffy wrote: tduffy> This patch removes the last remaining typedef enums (minus bool tduffy> which /might/ justify it). tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_cookie.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cookie.c (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) tduffy> @@ -262,7 +262,8 @@ bail: tduffy> * DAT_INSUFFICIENT_EMPTY tduffy> * tduffy> */ tduffy> -u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, DAPL_DTO_TYPE type, tduffy> +u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, tduffy> + enum dapl_dto_type type, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> struct dapl_cookie **cookie_ptr) tduffy> { tduffy> Index: linux-kernel/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -523,7 +523,7 @@ u32 dapl_ib_mw_unbind(struct dapl_rmr *r tduffy> * tduffy> */ tduffy> u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, tduffy> - DAPL_ASYNC_HANDLER_TYPE handler_type, tduffy> + enum async_handler_type handler_type, tduffy> ib_async_handler_t callback, void *context) tduffy> { tduffy> int ib_status = 0; tduffy> Index: linux-kernel/dat-provider/dapl_cookie.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cookie.h (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_cookie.h (working copy) tduffy> @@ -48,7 +48,8 @@ extern u32 dapl_rmr_cookie_alloc(struct tduffy> struct dapl_cookie **cookie_ptr); tduffy> tduffy> extern u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, tduffy> - DAPL_DTO_TYPE type, DAT_DTO_COOKIE user_cookie, tduffy> + enum dapl_dto_type type, tduffy> + DAT_DTO_COOKIE user_cookie, tduffy> struct dapl_cookie **cookie_ptr); tduffy> tduffy> extern void dapl_cookie_dealloc(struct dapl_cookie_buffer *buffer, tduffy> Index: linux-kernel/dat-provider/dapl_openib_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_util.h (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_openib_util.h (working copy) tduffy> @@ -36,11 +36,11 @@ tduffy> #include "ib_verbs.h" tduffy> #include "ib_cm.h" tduffy> tduffy> -typedef enum async_handler_type { tduffy> +enum async_handler_type { tduffy> DAPL_ASYNC_UNAFILIATED, tduffy> DAPL_ASYNC_CQ_ERROR, tduffy> DAPL_ASYNC_QP_ERROR tduffy> -} DAPL_ASYNC_HANDLER_TYPE; tduffy> +}; tduffy> tduffy> /* alignment for the bus transfer from the HCA/IB chip to the main memory */ tduffy> #define DAPL_OPTIMAL_ALIGNMENT 256 tduffy> @@ -76,7 +76,7 @@ u32 dapl_ib_accept_connection(struct dap tduffy> u32 dapl_ib_reject_connection(struct dapl_cm_ctx *cm_ctx); tduffy> tduffy> u32 dapl_ib_setup_async_callback(struct dapl_ia *ia, tduffy> - DAPL_ASYNC_HANDLER_TYPE handler_type, tduffy> + enum async_handler_type handler_type, tduffy> ib_async_handler_t callback, void *context); tduffy> tduffy> u32 dapl_ib_cq_alloc(struct dapl_ia *ia, struct dapl_evd *evd, int *cqlen, tduffy> Index: linux-kernel/dat-provider/dapl_ep.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep.c (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_ep.c (working copy) tduffy> @@ -168,7 +168,7 @@ static u32 dapl_ep_post_send_req(struct tduffy> DAT_DTO_COOKIE user_cookie, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> enum dat_completion_flags completion_flags, tduffy> - DAPL_DTO_TYPE dto_type, tduffy> + enum dapl_dto_type dto_type, tduffy> enum ib_wr_opcode op_type) tduffy> { tduffy> struct dapl_ep *ep_ptr; tduffy> Index: linux-kernel/dat-provider/dapl_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.c (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_util.c (working copy) tduffy> @@ -36,7 +36,7 @@ tduffy> tduffy> #ifdef DAPL_DBG tduffy> tduffy> -void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) tduffy> +void dapl_dbg_log(enum dapl_dbg_type type, const char *fmt, ...) tduffy> { tduffy> char buf[1024]; tduffy> va_list args; tduffy> Index: linux-kernel/dat-provider/dapl_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.c (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_evd.c (working copy) tduffy> @@ -557,7 +557,7 @@ static void dapl_evd_wc_to_event(struct tduffy> /* We can not check the following assert for now */ tduffy> if (dto_status == DAT_DTO_SUCCESS) { tduffy> enum ib_wc_opcode ib_opcode = wc->opcode; tduffy> - DAPL_DTO_TYPE dto_type = cookie->val.dto.type; tduffy> + enum dapl_dto_type dto_type = cookie->val.dto.type; tduffy> dapl_os_assert((ib_opcode == IB_WC_SEND && tduffy> dto_type == DAPL_DTO_TYPE_SEND) || tduffy> (ib_opcode == IB_WC_RECV && tduffy> @@ -877,7 +877,7 @@ void dapl_evd_connection_callback(struct tduffy> static void dapl_evd_dto_callback(struct ib_cq *cq, void *user_context) tduffy> { tduffy> struct dapl_evd *evd; tduffy> - DAPL_EVD_STATE state; tduffy> + enum dapl_evd_state state; tduffy> u32 status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, "dapl_evd_dto_callback(%p, %p)\n", tduffy> @@ -888,7 +888,7 @@ static void dapl_evd_dto_callback(struct tduffy> dapl_os_assert(evd->cq == cq); tduffy> tduffy> /* Read once. */ tduffy> - state = *(volatile DAPL_EVD_STATE *)&evd->evd_state; tduffy> + state = *(volatile enum dapl_evd_state *)&evd->evd_state; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_EVD, tduffy> "dapl_evd_dto_callback: CQ %p, state %x\n", tduffy> Index: linux-kernel/dat-provider/dapl_sp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp.c (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_sp.c (working copy) tduffy> @@ -624,7 +624,7 @@ u32 dapl_psp_free(struct dat_sp *dat_psp tduffy> { tduffy> struct dapl_ia *ia; tduffy> struct dapl_sp *sp; tduffy> - DAPL_SP_STATE save_state; tduffy> + enum dapl_sp_state save_state; tduffy> u32 status = DAT_SUCCESS; tduffy> tduffy> sp = (struct dapl_sp *)dat_psp; tduffy> Index: linux-kernel/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.h (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl_util.h (working copy) tduffy> @@ -167,7 +167,7 @@ static inline long dapl_os_mod64(uint64_ tduffy> * Use these bits to enable various tracing/debug options. Each bit tduffy> * represents debugging in a particular subsystem or area of the code. tduffy> */ tduffy> -typedef enum { tduffy> +enum dapl_dbg_type { tduffy> DAPL_DBG_TYPE_ERR = 0x0001, tduffy> DAPL_DBG_TYPE_WARN = 0x0002, tduffy> DAPL_DBG_TYPE_EVD = 0x0004, tduffy> @@ -180,12 +180,12 @@ typedef enum { tduffy> DAPL_DBG_TYPE_RTN = 0x0200, tduffy> DAPL_DBG_TYPE_EXCEPTION = 0x0400, tduffy> DAPL_DBG_TYPE_SRQ = 0x0800 tduffy> -} DAPL_DBG_TYPE; tduffy> +}; tduffy> tduffy> typedef int DAPL_DBG_MASK; tduffy> tduffy> #if defined(DAPL_DBG) tduffy> -extern void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...); tduffy> +extern void dapl_dbg_log(enum dapl_dbg_type type, const char *fmt, ...); tduffy> #else /* !DAPL_DBG */ tduffy> #define dapl_dbg_log(...) tduffy> #endif /* DAPL_DBG */ tduffy> Index: linux-kernel/dat-provider/dapl.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl.h (revision 2640) tduffy> +++ linux-kernel/dat-provider/dapl.h (working copy) tduffy> @@ -49,17 +49,17 @@ tduffy> * * tduffy> *********************************************************************/ tduffy> tduffy> -typedef enum dapl_evd_state { tduffy> +enum dapl_evd_state { tduffy> DAPL_EVD_STATE_OPEN, tduffy> DAPL_EVD_STATE_DEAD = 0xDEAD tduffy> -} DAPL_EVD_STATE; tduffy> +}; tduffy> tduffy> -typedef enum dapl_evd_completion { tduffy> +enum dapl_evd_completion { tduffy> DAPL_EVD_STATE_INIT, tduffy> DAPL_EVD_STATE_SOLICITED_WAIT, tduffy> DAPL_EVD_STATE_THRESHOLD, tduffy> DAPL_EVD_STATE_UNSIGNALLED tduffy> -} DAPL_EVD_COMPLETION; tduffy> +}; tduffy> tduffy> /********************************************************************* tduffy> * * tduffy> @@ -161,7 +161,7 @@ struct dapl_evd { tduffy> struct dat_evd evd; tduffy> struct dapl_common common; tduffy> struct list_head list; tduffy> - DAPL_EVD_STATE evd_state; tduffy> + enum dapl_evd_state evd_state; tduffy> enum dat_evd_flags evd_flags; tduffy> tduffy> /* Derived from evd_flags; see dapls_evd_internal_create. */ tduffy> @@ -265,19 +265,19 @@ struct dapl_rmr { tduffy> }; tduffy> tduffy> /* SP types, indicating the state and queue */ tduffy> -typedef enum dapl_sp_state { tduffy> +enum dapl_sp_state { tduffy> DAPL_SP_STATE_FREE, tduffy> DAPL_SP_STATE_PSP_LISTENING, tduffy> DAPL_SP_STATE_PSP_PENDING, tduffy> DAPL_SP_STATE_RSP_LISTENING, tduffy> DAPL_SP_STATE_RSP_PENDING tduffy> -} DAPL_SP_STATE; tduffy> +}; tduffy> tduffy> struct dapl_sp { tduffy> struct dat_sp sp; tduffy> struct dapl_common common; tduffy> struct list_head list; tduffy> - DAPL_SP_STATE state; /* type and queue of the SP */ tduffy> + enum dapl_sp_state state; /* type and queue of the SP */ tduffy> tduffy> /* PSP/RSP PARAM fields */ tduffy> struct dat_ia *ia; tduffy> @@ -313,22 +313,22 @@ struct dapl_cr { tduffy> struct dapl_sp *sp; tduffy> }; tduffy> tduffy> -typedef enum dapl_dto_type { tduffy> +enum dapl_dto_type { tduffy> DAPL_DTO_TYPE_SEND, tduffy> DAPL_DTO_TYPE_RECV, tduffy> DAPL_DTO_TYPE_RDMA_WRITE, tduffy> DAPL_DTO_TYPE_RDMA_READ, tduffy> -} DAPL_DTO_TYPE; tduffy> +}; tduffy> tduffy> -typedef enum dapl_cookie_type { tduffy> +enum dapl_cookie_type { tduffy> DAPL_COOKIE_TYPE_NULL, tduffy> DAPL_COOKIE_TYPE_DTO, tduffy> DAPL_COOKIE_TYPE_RMR, tduffy> -} DAPL_COOKIE_TYPE; tduffy> +}; tduffy> tduffy> /* struct dapl_dto_cookie used as context for DTO WQEs */ tduffy> struct dapl_dto_cookie { tduffy> - DAPL_DTO_TYPE type; tduffy> + enum dapl_dto_type type; tduffy> DAT_DTO_COOKIE cookie; tduffy> int size; /* used for SEND and RDMA write */ tduffy> }; tduffy> @@ -341,7 +341,7 @@ struct dapl_rmr_cookie { tduffy> tduffy> /* struct dapl_cookie used as context for WQEs */ tduffy> struct dapl_cookie { tduffy> - DAPL_COOKIE_TYPE type; /* Must be first, to define struct. */ tduffy> + enum dapl_cookie_type type; /* Must be first, to define struct. */ tduffy> struct dapl_ep *ep; tduffy> int index; tduffy> union { tduffy> Index: linux-kernel/dat/core.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/core.c (revision 2640) tduffy> +++ linux-kernel/dat/core.c (working copy) tduffy> @@ -43,7 +43,7 @@ static DAT_DBG_MASK g_dbg_mask = DAT_DBG tduffy> module_param_named(dbg_mask, g_dbg_mask, int, 0644); tduffy> MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug message types."); tduffy> tduffy> -void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...) tduffy> +void dat_dbg_print(enum dat_dbg_type type, const char *fmt, ...) tduffy> { tduffy> static char buf[1024]; tduffy> tduffy> Index: linux-kernel/dat/core.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/core.h (revision 2640) tduffy> +++ linux-kernel/dat/core.h (working copy) tduffy> @@ -45,17 +45,17 @@ tduffy> tduffy> #define DAT_IS_BAD_POINTER(p) ( NULL == (p) ) tduffy> tduffy> -typedef enum { tduffy> +enum dat_dbg_type { tduffy> DAT_DBG_TYPE_ERROR = (1 << 0), tduffy> DAT_DBG_TYPE_GENERIC = (1 << 1), tduffy> DAT_DBG_TYPE_DR = (1 << 2), tduffy> DAT_DBG_TYPE_PROVIDER_API = (1 << 3), tduffy> DAT_DBG_TYPE_CONSUMER_API = (1 << 4), tduffy> DAT_DBG_TYPE_ALL = 0x1f tduffy> -} DAT_DBG_TYPE; tduffy> +}; tduffy> tduffy> typedef int DAT_DBG_MASK; tduffy> tduffy> -extern void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...); tduffy> +extern void dat_dbg_print(enum dat_dbg_type type, const char *fmt, ...); tduffy> tduffy> #endif /* CORE_H */ tduffy> From halr at voltaire.com Wed Jun 22 09:15:20 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 22 Jun 2005 12:15:20 -0400 Subject: [openib-general] [PATCH] [AT} [TRIVIAL]: Fix typo in header file Message-ID: <1119456919.4480.1.camel@hal.voltaire.com> Fix typo in header file Signed-off-by: Hal Rosenstock Index: ib_at.h =================================================================== --- ib_at.h (revision 2671) +++ ib_at.h (working copy) @@ -125,7 +125,7 @@ * @async_comp: asynchronous callback structure - optional * * Resolve the specified dst_ip to a &struct ib_route structure. - * src_ip can be provide to force specific output interface. + * src_ip can be provided to force specific output interface. * flags can be used to select resolving method; currently IB-ARP or ATS. * * See ib_at_completion structure documentation for asynchronous From roland at topspin.com Wed Jun 22 09:22:57 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 09:22:57 -0700 Subject: [openib-general] [PATCH] [AT} [TRIVIAL]: Fix typo in header file In-Reply-To: <1119456919.4480.1.camel@hal.voltaire.com> (Hal Rosenstock's message of "22 Jun 2005 12:15:20 -0400") References: <1119456919.4480.1.camel@hal.voltaire.com> Message-ID: <52r7eus6n2.fsf@topspin.com> > - * src_ip can be provide to force specific output interface. > + * src_ip can be provided to force specific output interface. The typo fix is fine but how does srp_ip force a specific output interface? It's perfectly valid to configure every local network interface with the same IP address. - R. From iod00d at hp.com Wed Jun 22 09:28:56 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 22 Jun 2005 09:28:56 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050622071928.GA32485@mellanox.co.il> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> Message-ID: <20050622162856.GA18883@esmail.cup.hp.com> On Wed, Jun 22, 2005 at 10:19:28AM +0300, Michael S. Tsirkin wrote: > Here's what works for me: > > cc -rdynamic -u openib_driver_init -Wall -O2 -g -D_GNU_SOURCE rdma_bw.c get_clock.c /usr/local/lib/infiniband/mthca.a /usr/local/lib/libibverbs.a -o rdma_bw -ldl -lsysfs -lpthread Yes - as you noted "/usr/local/lib/" needed to be "/usr/lib". The debian packages use the latter. I was able to build with the above command line and SVN 2670 bits. But uverbs now seems broken with normal dynamic builds too. More details below. > > Or maybe a "make static" target in the Makefile....*sigh* > > Problem with this is, for this I need to figure out where > is mthca.a installed. Understood...maybe just hard code it in the Makefile for now? That's at least a start. Use something like "OPENIBLIBSDIR = /usr/local/lib/infiniband"? I also don't know how to express if target == static then MORECFLAGS = -rdynamic... endif Anyway....after rebuilding everything to deal with the ABI event and rebooting to 2.6.12 kernel, I couldn't collect pfmon31 output: grundler at iota:/usr/src/openib_gen2/src/userspace/perftest$ ./rdma_bw Couldn't get context for mthca0 This error message doesn't tell me how to fix it. :^( I'm not getting any dmesg output for this either. I should be at SVN 2670 with everything after updating and following the rebuild sequence below: cd $SRC/userspace/libibverbs dpkg-buildpackage -rfakeroot -uc -us cd .. sudo dpkg -i libibverbs-dev_0.1.0-1_ia64.deb \ libibverbs1_0.1.0-1_ia64.deb cd libmthca dpkg-buildpackage -rfakeroot -uc -us cd .. sudo dpkg -i libmthca1_0.1.0-1_ia64.deb libmthca-dev_0.1.0-1_ia64.deb cd perftest make perftest has no "install" target and doesn't need one. Matt is there someplace appropriate to add the above recipe? I ended up with: grundler at iota:/usr/src/openib_gen2/src/userspace$ ls -lrt *.deb -rw-r--r-- 1 grundler grundler 15254 Jun 22 07:54 libibverbs1_0.1.0-1_ia64.deb -rw-r--r-- 1 grundler grundler 24060 Jun 22 07:54 libibverbs-dev_0.1.0-1_ia64.deb -rw-r--r-- 1 grundler grundler 21758 Jun 22 07:54 ibverbs-examples_0.1.0-1_ia64.deb -rw-r--r-- 1 grundler grundler 18380 Jun 22 08:01 libmthca1_0.1.0-1_ia64.deb -rw-r--r-- 1 grundler grundler 179284 Jun 22 08:01 libmthca-dev_0.1.0-1_ia64.deb google suggests I need to create /dev/infiniband nodes but I already have the following: grundler at iota:/usr/src/linux-2.6$ ls -l /dev/infiniband/ total 0 crw-rw-rw- 1 root root 231, 255 May 3 15:48 ucm crw-rw-rw- 1 root root 231, 128 May 3 12:10 uverbs0 crw-rw-rw- 1 root root 231, 129 May 3 12:10 uverbs1 I'm guessing uverbs b0rkage because I can ping other HCAs: grundler at iota:/usr/src/linux-2.6$ /sbin/ifconfig ib0 ib0 Link encap:UNSPEC HWaddr 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:10.0.0.30 Bcast:10.0.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:7 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:532 (532.0 b) TX bytes:620 (620.0 b) grundler at iota:/usr/src/linux-2.6$ ping 10.0.0.51 PING 10.0.0.51 (10.0.0.51): 56 data bytes 64 bytes from 10.0.0.51: icmp_seq=0 ttl=64 time=0.5 ms 64 bytes from 10.0.0.51: icmp_seq=1 ttl=64 time=0.0 ms ... Any clue what stupid thing I've done now? Or is this really fallout from the ABI event? (new bug?) thanks, grant From iod00d at hp.com Wed Jun 22 09:29:48 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 22 Jun 2005 09:29:48 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <52zmtis7ns.fsf@topspin.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> Message-ID: <20050622162948.GB18883@esmail.cup.hp.com> On Wed, Jun 22, 2005 at 09:00:55AM -0700, Roland Dreier wrote: > Michael> Roland, is there some way to get the plugin directory out > Michael> of the libibverbs library? > > Sorry, not sure what you mean. we need to know where mthca.a can be found. Ie where is libibverbs dlopen() looking for it? grant From roland at topspin.com Wed Jun 22 09:30:02 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 09:30:02 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050622162948.GB18883@esmail.cup.hp.com> (Grant Grundler's message of "Wed, 22 Jun 2005 09:29:48 -0700") References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622162948.GB18883@esmail.cup.hp.com> Message-ID: <52ll52s6b9.fsf@topspin.com> Grant> we need to know where mthca.a can be found. Ie where is Grant> libibverbs dlopen() looking for it? ??? dlopen() can't load .a files. So the only way it will be found is if you staticly link it into your executable. - R. From jcarr at linuxmachines.com Wed Jun 22 09:43:50 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Wed, 22 Jun 2005 09:43:50 -0700 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <527jgnv6ac.fsf@topspin.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> Message-ID: <42B99546.2070703@linuxmachines.com> On 06/21/2005 12:50 PM, Roland Dreier wrote: > What happens if you try replacing the send_flags line with the one you > have commented out? > > + // send_wr.send_flags = IB_SEND_SIGNALED; Thanks, you are correct. IB_SEND_SIGNALED gives me the behavior I was expecting. Jeff From iod00d at hp.com Wed Jun 22 09:44:16 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 22 Jun 2005 09:44:16 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <52ll52s6b9.fsf@topspin.com> References: <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622162948.GB18883@esmail.cup.hp.com> <52ll52s6b9.fsf@topspin.com> Message-ID: <20050622164416.GD18883@esmail.cup.hp.com> On Wed, Jun 22, 2005 at 09:30:02AM -0700, Roland Dreier wrote: > Grant> we need to know where mthca.a can be found. Ie where is > Grant> libibverbs dlopen() looking for it? > > ??? dlopen() can't load .a files. So the only way it will be found is > if you staticly link it into your executable. Sorry - *we* are looking for the .a file. dlopen() will be looking for the .so in (hopefully) the same directory. grant From jlentini at netapp.com Wed Jun 22 09:47:07 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 12:47:07 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] Update dat_rmr_bind API In-Reply-To: <20050620084255.GA7060@mellanox.co.il> References: <20050620084255.GA7060@mellanox.co.il> Message-ID: Committed in revision 2674. On Mon, 20 Jun 2005, Itamar Rabenstein wrote: itamar> Update dat_rmr_bind function API (added lmr_handle as input param) itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: test/dapltest/test/dapl_bpool.c itamar> =================================================================== itamar> --- test/dapltest/test/dapl_bpool.c (revision 2656) itamar> +++ test/dapltest/test/dapl_bpool.c (working copy) itamar> @@ -237,6 +237,7 @@ itamar> bpool_ptr->reg_addr, bpool_ptr->reg_size)); itamar> itamar> ret = dat_rmr_bind ( bpool_ptr->rmr_handle, itamar> + bpool_ptr->lmr, itamar> &iov, itamar> mflags, itamar> bpool_ptr->ep, itamar> Index: test/dapltest/kdapl/kdapl_tdep_user.c itamar> =================================================================== itamar> --- test/dapltest/kdapl/kdapl_tdep_user.c (revision 2656) itamar> +++ test/dapltest/kdapl/kdapl_tdep_user.c (working copy) itamar> @@ -76,7 +76,7 @@ itamar> } itamar> if (params_ptr->test_type == TRANSACTION_TEST) { itamar> print_ioctl.cookie = ioctl (fd, KDAPL_IOCTL_GET_STAT_T, &Client_Stats_T); itamar> - DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_iterations, itamar> + DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_threads, itamar> params_ptr->u.Transaction_Cmd.eps_per_thread); itamar> } itamar> itamar> Index: dat-provider/dapl_rmr.c itamar> =================================================================== itamar> --- dat-provider/dapl_rmr.c (revision 2656) itamar> +++ dat-provider/dapl_rmr.c (working copy) itamar> @@ -149,26 +149,18 @@ itamar> } itamar> itamar> static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, itamar> - const struct dat_lmr_triplet *lmr_triplet, itamar> + struct dapl_lmr *lmr, itamar> + const struct dat_lmr_triplet *lmr_triplet, itamar> enum dat_mem_priv_flags mem_priv, itamar> struct dapl_ep *ep_ptr, itamar> DAT_RMR_COOKIE user_cookie, itamar> enum dat_completion_flags completion_flags, itamar> DAT_RMR_CONTEXT *rmr_context) itamar> { itamar> - struct dapl_lmr *lmr; itamar> struct dapl_cookie *cookie; itamar> u32 status; itamar> boolean_t is_signaled; itamar> itamar> - status = dapl_hash_search(rmr->common.owner_ia->hca->lmr_hash_table, itamar> - lmr_triplet->lmr_context, itamar> - (DAPL_HASH_DATA *) &lmr); itamar> - if (DAT_SUCCESS != status) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); itamar> - goto bail; itamar> - } itamar> - itamar> /* itamar> * if the ep in unconnected return an error. IB requires that the itamar> * QP be connected to change a memory window binding since: itamar> @@ -353,27 +345,32 @@ itamar> * Output: itamar> */ itamar> u32 dapl_rmr_bind(struct dat_rmr *rmr_handle, itamar> - const struct dat_lmr_triplet *lmr_triplet, itamar> + struct dat_lmr *lmr_handle, const struct dat_lmr_triplet *lmr_triplet, itamar> enum dat_mem_priv_flags mem_priv, struct dat_ep *ep, itamar> DAT_RMR_COOKIE user_cookie, itamar> enum dat_completion_flags completion_flags, itamar> DAT_RMR_CONTEXT *rmr_context) itamar> { itamar> struct dapl_rmr *rmr; itamar> + struct dapl_lmr *lmr; itamar> struct dapl_ep *ep_ptr; itamar> itamar> if (!rmr_handle) itamar> return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); itamar> itamar> + if (!lmr_handle) itamar> + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); itamar> + itamar> if (!ep) itamar> return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> itamar> rmr = (struct dapl_rmr *)rmr_handle; itamar> + lmr = (struct dapl_lmr *)lmr_handle; itamar> ep_ptr = (struct dapl_ep *)ep; itamar> itamar> /* if the rmr should be bound */ itamar> if (0 != lmr_triplet->segment_length) itamar> - return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, itamar> + return dapl_rmr_bind_fuse(rmr, lmr, lmr_triplet, mem_priv, ep_ptr, itamar> user_cookie, completion_flags, itamar> rmr_context); itamar> else /* the rmr should be unbound */ itamar> Index: dat-provider/dapl.h itamar> =================================================================== itamar> --- dat-provider/dapl.h (revision 2656) itamar> +++ dat-provider/dapl.h (working copy) itamar> @@ -543,6 +543,7 @@ itamar> struct dat_rmr_param *rmr_args); itamar> itamar> extern u32 dapl_rmr_bind(struct dat_rmr *rmr, itamar> + struct dat_lmr *lmr, itamar> const struct dat_lmr_triplet *lmr_triplet, itamar> enum dat_mem_priv_flags mem_priv, itamar> struct dat_ep *ep, itamar> Index: dat/dat.h itamar> =================================================================== itamar> --- dat/dat.h (revision 2656) itamar> +++ dat/dat.h (working copy) itamar> @@ -1144,7 +1144,8 @@ itamar> itamar> typedef u32 (*DAT_RMR_QUERY_FUNC)(struct dat_rmr *, struct dat_rmr_param *); itamar> itamar> -typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, const struct dat_lmr_triplet *, itamar> +typedef u32 (*DAT_RMR_BIND_FUNC)(struct dat_rmr *, struct dat_lmr *, itamar> + const struct dat_lmr_triplet *, itamar> enum dat_mem_priv_flags, struct dat_ep *, itamar> DAT_RMR_COOKIE, enum dat_completion_flags, itamar> DAT_RMR_CONTEXT *); itamar> @@ -1617,23 +1618,24 @@ itamar> return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); itamar> } itamar> itamar> -static inline u32 dat_rmr_query(struct dat_rmr * rmr, struct dat_rmr_param *param) itamar> +static inline u32 dat_rmr_query(struct dat_rmr *rmr, struct dat_rmr_param *param) itamar> { itamar> return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); itamar> } itamar> itamar> -static inline u32 dat_rmr_bind(struct dat_rmr * rmr, itamar> - const struct dat_lmr_triplet *iov, itamar> +static inline u32 dat_rmr_bind(struct dat_rmr *rmr, itamar> + struct dat_lmr *lmr, itamar> + const struct dat_lmr_triplet *iov, itamar> enum dat_mem_priv_flags mem_flags, itamar> struct dat_ep *ep, DAT_RMR_COOKIE cookie, itamar> enum dat_completion_flags comp_flags, itamar> DAT_RMR_CONTEXT *context) itamar> { itamar> - return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, iov, mem_flags, ep, itamar> - cookie, comp_flags, context); itamar> + return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, lmr, iov, mem_flags, itamar> + ep, cookie, comp_flags, context); itamar> } itamar> itamar> -static inline u32 dat_rmr_free(struct dat_rmr * rmr) itamar> +static inline u32 dat_rmr_free(struct dat_rmr *rmr) itamar> { itamar> return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); itamar> } itamar> -- itamar> Itamar itamar> From roland at topspin.com Wed Jun 22 09:47:27 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 09:47:27 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050622164416.GD18883@esmail.cup.hp.com> (Grant Grundler's message of "Wed, 22 Jun 2005 09:44:16 -0700") References: <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622162948.GB18883@esmail.cup.hp.com> <52ll52s6b9.fsf@topspin.com> <20050622164416.GD18883@esmail.cup.hp.com> Message-ID: <52fyvas5i8.fsf@topspin.com> Grant> Sorry - *we* are looking for the .a file. dlopen() will be Grant> looking for the .so in (hopefully) the same directory. By default the .so plugins are installed in $(libdir)/infiniband. libibverbs looks there plus any other directories in OPENIB_DRIVER_PATH. - R. From jlentini at netapp.com Wed Jun 22 09:53:49 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 12:53:49 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] Remove dapl_hash (there is no need for the hash) In-Reply-To: <20050620084450.GA7070@mellanox.co.il> References: <20050620084450.GA7070@mellanox.co.il> Message-ID: Committed in revision 2675. On Mon, 20 Jun 2005, Itamar Rabenstein wrote: itamar> Remove dapl_hash (there is no need for the hash) itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> diff -Nurp -X dontdiff dat-provider_bind/Makefile dat-provider/Makefile itamar> --- dat-provider_bind/Makefile Sun Jun 19 15:17:54 2005 itamar> +++ dat-provider/Makefile Sun Jun 19 15:24:03 2005 itamar> @@ -20,7 +20,6 @@ PROVIDER_MODULES := \ itamar> dapl_cr \ itamar> dapl_ep \ itamar> dapl_evd \ itamar> - dapl_hash \ itamar> dapl_hca_util \ itamar> dapl_ia \ itamar> dapl_lmr \ itamar> diff -Nurp -X dontdiff dat-provider_bind/dapl.h dat-provider/dapl.h itamar> --- dat-provider_bind/dapl.h Sun Jun 19 15:17:54 2005 itamar> +++ dat-provider/dapl.h Sun Jun 19 15:24:06 2005 itamar> @@ -84,9 +84,6 @@ typedef enum dapl_evd_completion { itamar> itamar> typedef void (*ib_async_handler_t) (struct ib_event *, void *); itamar> itamar> -typedef u64 DAPL_HASH_KEY; itamar> -typedef void *DAPL_HASH_DATA; itamar> - itamar> /********************************************************************* itamar> * * itamar> * Structures * itamar> @@ -125,8 +122,6 @@ struct dapl_hca { itamar> u8 port_num; itamar> struct ib_device *ib_hca_handle; itamar> struct ib_cq *null_cq; /* CQ with 0 entries */ itamar> - /* Memory Subsystem Support */ itamar> - struct dapl_hash_table *lmr_hash_table; itamar> /* Limits & useful HCA attributes */ itamar> struct dat_ia_attr ia_attr; itamar> struct dat_ep_attr ep_attr; itamar> diff -Nurp -X dontdiff dat-provider_bind/dapl_hash.c dat-provider/dapl_hash.c itamar> --- dat-provider_bind/dapl_hash.c Sun Jun 19 15:17:54 2005 itamar> +++ dat-provider/dapl_hash.c Thu Jan 1 02:00:00 1970 itamar> @@ -1,462 +0,0 @@ itamar> -/* itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. itamar> - * itamar> - * This Software is licensed under one of the following licenses: itamar> - * itamar> - * 1) under the terms of the "Common Public License 1.0" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/cpl.php. itamar> - * itamar> - * 2) under the terms of the "The BSD License" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/bsd-license.php. itamar> - * itamar> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a itamar> - * copy of which is available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/gpl-license.php. itamar> - * itamar> - * Licensee has the right to choose one of the above licenses. itamar> - * itamar> - * Redistributions of source code must retain the above copyright itamar> - * notice and one of the license notices. itamar> - * itamar> - * Redistributions in binary form must reproduce both the above copyright itamar> - * notice, one of the license notices in the documentation itamar> - * and/or other materials provided with the distribution. itamar> - */ itamar> - itamar> -/* itamar> - * Provides a generic hash table with chaining. itamar> - * itamar> - * $Id: dapl_hash.c 2640 2005-06-16 16:22:46Z jlentini $ itamar> - */ itamar> - itamar> -#include "dapl_hash.h" itamar> - itamar> -/********************************************************************* itamar> - * * itamar> - * Structures * itamar> - * * itamar> - *********************************************************************/ itamar> - itamar> -/* itamar> - * A hash table element itamar> - */ itamar> -typedef struct DAPL_HASH_ELEM { itamar> - struct DAPL_HASH_ELEM *next_element; itamar> - DAPL_HASH_KEY key; itamar> - void *datum; itamar> -} DAPL_HASH_ELEM; itamar> - itamar> -/* itamar> - * The hash table itamar> - */ itamar> -struct dapl_hash_table { itamar> - unsigned long num_entries; itamar> - unsigned long tbl_size; itamar> - DAPL_HASH_ELEM *table; itamar> - spinlock_t lock; itamar> - unsigned long flags; itamar> - /* itamar> - * statistics - we tally on insert operations, counting itamar> - * the number of entries in the whole hash table, as itamar> - * well as the length of chains we walk to insert. This itamar> - * ignores empty buckets, giving us data on overall table itamar> - * occupancy, as well as max/average chain length for itamar> - * the buckets used. If our hash function results in itamar> - * hot buckets, this will show it. itamar> - */ itamar> - uint64_t hash_tbl_inserts; /* total inserts ops */ itamar> - uint64_t hash_tbl_max; /* max in entire table */ itamar> - uint64_t hash_tbl_total; /* total in table */ itamar> - uint64_t hash_chn_max; /* longest chain */ itamar> - uint64_t hash_chn_total; /* total non-0 lenghts */ itamar> -}; itamar> - itamar> -/********************************************************************* itamar> - * * itamar> - * Defines * itamar> - * * itamar> - *********************************************************************/ itamar> - itamar> -/* datum value in empty table slots (use 0UL or ~0UL as appropriate) */ itamar> -#define NO_DATUM_VALUE ((void *) 0UL) itamar> -#define NO_DATUM(value) ((value) == NO_DATUM_VALUE) itamar> - itamar> -/* Lookup macro (which falls back to function to rehash) */ itamar> -#define DAPL_HASHLOOKUP( p_table, in_key, out_datum, bucket_head) \ itamar> - do { \ itamar> - DAPL_HASH_KEY save_key = in_key; \ itamar> - DAPL_HASH_ELEM *element = \ itamar> - &((p_table)->table)[DAPL_DOHASH(in_key,(p_table)->tbl_size)]; \ itamar> - in_key = save_key; \ itamar> - if (NO_DATUM(element->datum)) { \ itamar> - (bucket_head) = (void *)0; \ itamar> - } else if (element->key == (DAPL_HASH_KEY) (in_key)) { \ itamar> - (out_datum) = element->datum; \ itamar> - (bucket_head) = (void *)element; \ itamar> - } else if (element->next_element) { \ itamar> - dapl_hash_rehash(element, \ itamar> - (in_key), \ itamar> - (void **)&(out_datum), \ itamar> - (DAPL_HASH_ELEM **)&(bucket_head)); \ itamar> - } else { \ itamar> - (bucket_head) = (void *)0; \ itamar> - }\ itamar> - } while (0) itamar> - itamar> -/********************************************************************* itamar> - * * itamar> - * Internal Functions * itamar> - * * itamar> - *********************************************************************/ itamar> - itamar> -/* itamar> - * Rehash the key (used by add and lookup functions) itamar> - * itamar> - * Inputs: element element to rehash key itamar> - * key, datum datum for key head itamar> - * head head for list itamar> - */ itamar> -static void itamar> -dapl_hash_rehash(DAPL_HASH_ELEM * element, itamar> - DAPL_HASH_KEY key, void **datum, DAPL_HASH_ELEM ** head) itamar> -{ itamar> - /* itamar> - * assume we looked at the contents of element already, itamar> - * and start with the next element. itamar> - */ itamar> - dapl_os_assert(element->next_element); itamar> - dapl_os_assert(!NO_DATUM(element->datum)); itamar> - itamar> - *head = element; itamar> - while (1) { itamar> - element = element->next_element; itamar> - if (!element) { itamar> - break; itamar> - } itamar> - if (element->key == key) { itamar> - *datum = element->datum; itamar> - return; itamar> - } itamar> - } itamar> - *head = NULL; itamar> -} itamar> - itamar> -/* itamar> - * Add a new key to the hash table itamar> - * itamar> - * Inputs: itamar> - * table, key and datum to be added itamar> - * allow_dup - TRUE if dups are allowed itamar> - * Outputs: itamar> - * report_dup - should you care to know itamar> - * Returns: itamar> - * TRUE on success itamar> - */ itamar> -static boolean_t itamar> -dapl_hash_add(struct dapl_hash_table *table, DAPL_HASH_KEY key, void *datum, itamar> - boolean_t allow_dup, boolean_t *report_dup) itamar> -{ itamar> - void *olddatum; itamar> - DAPL_HASH_KEY hashValue, save_key = key; itamar> - DAPL_HASH_ELEM *found; itamar> - boolean_t status = FALSE; itamar> - unsigned int chain_len = 0; itamar> - itamar> - if (report_dup) { itamar> - (*report_dup) = FALSE; itamar> - } itamar> - itamar> - if (NO_DATUM(datum)) { itamar> - /* itamar> - * Reserved value used for datum itamar> - */ itamar> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, itamar> - "dapl_hash_add() called with magic NO_DATA " itamar> - "value (%p) used as datum!\n", datum); itamar> - return FALSE; itamar> - } itamar> - itamar> - DAPL_HASHLOOKUP(table, key, olddatum, found); itamar> - if (found) { itamar> - /* itamar> - * key exists already itamar> - */ itamar> - if (report_dup) { itamar> - *report_dup = TRUE; itamar> - } itamar> - itamar> - if (!allow_dup) { itamar> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, itamar> - "dapl_hash_add() called with duplicate " itamar> - "key (" F64x ")\n", key); itamar> - return FALSE; itamar> - } itamar> - } itamar> - itamar> - hashValue = DAPL_DOHASH(key, table->tbl_size); itamar> - key = save_key; itamar> - if (NO_DATUM(table->table[hashValue].datum)) { itamar> - /* itamar> - * Empty head - just fill it in itamar> - */ itamar> - table->table[hashValue].key = key; itamar> - table->table[hashValue].datum = datum; itamar> - table->table[hashValue].next_element = NULL; itamar> - table->num_entries++; itamar> - status = TRUE; itamar> - } else { itamar> - DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, itamar> - GFP_ATOMIC); itamar> - /* itamar> - * Add an element to the end of the chain itamar> - */ itamar> - if (newelement) { itamar> - DAPL_HASH_ELEM *lastelement; itamar> - newelement->key = key; itamar> - newelement->datum = datum; itamar> - newelement->next_element = NULL; itamar> - for (lastelement = &table->table[hashValue]; itamar> - lastelement->next_element; itamar> - lastelement = lastelement->next_element) { itamar> - /* Walk to the end of the chain */ itamar> - chain_len++; itamar> - } itamar> - lastelement->next_element = newelement; itamar> - table->num_entries++; itamar> - status = TRUE; itamar> - } else itamar> - status = FALSE; itamar> - } itamar> - itamar> - /* itamar> - * Tally up our counters. chain_len is one less than current chain itamar> - * length. itamar> - */ itamar> - chain_len++; itamar> - table->hash_tbl_inserts++; itamar> - table->hash_tbl_total += table->num_entries; itamar> - table->hash_chn_total += chain_len; itamar> - if (table->num_entries > table->hash_tbl_max) { itamar> - table->hash_tbl_max = table->num_entries; itamar> - } itamar> - if (chain_len > table->hash_chn_max) { itamar> - table->hash_chn_max = chain_len; itamar> - } itamar> - itamar> - return status; itamar> -} itamar> - itamar> -/* itamar> - * Remove element from hash bucket itamar> - * itamar> - * Inputs: itamar> - * element, key to be deleted itamar> - * Returns: itamar> - * TRUE on success itamar> - */ itamar> -static boolean_t itamar> -dapl_hash_delete_element(DAPL_HASH_ELEM * element, itamar> - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_datum) itamar> -{ itamar> - DAPL_HASH_ELEM *curelement; itamar> - DAPL_HASH_ELEM *lastelement; itamar> - itamar> - lastelement = NULL; itamar> - for (curelement = element; itamar> - curelement; itamar> - lastelement = curelement, curelement = curelement->next_element) { itamar> - if (curelement->key == key) { itamar> - if (p_datum) { itamar> - *p_datum = curelement->datum; itamar> - } itamar> - if (lastelement) { itamar> - /* itamar> - * curelement was malloc'd; free it itamar> - */ itamar> - lastelement->next_element = itamar> - curelement->next_element; itamar> - kfree(curelement); itamar> - } else { itamar> - /* itamar> - * curelement is static list head itamar> - */ itamar> - DAPL_HASH_ELEM *n = curelement->next_element; itamar> - if (n) { itamar> - /* itamar> - * If there is a next element, copy its contents into the itamar> - * head and free the original next element. itamar> - */ itamar> - curelement->key = n->key; itamar> - curelement->datum = n->datum; itamar> - curelement->next_element = itamar> - n->next_element; itamar> - kfree(n); itamar> - } else { itamar> - curelement->datum = NO_DATUM_VALUE; itamar> - } itamar> - } itamar> - break; itamar> - } itamar> - } itamar> - itamar> - return (curelement != NULL ? TRUE : FALSE); itamar> -} itamar> - itamar> -/********************************************************************* itamar> - * * itamar> - * External Functions * itamar> - * * itamar> - *********************************************************************/ itamar> - itamar> -/* itamar> - * Create a new hash table with at least 'table_size' hash buckets. itamar> - */ itamar> -u32 dapl_hash_create(int table_size, struct dapl_hash_table **pp_table) itamar> -{ itamar> - struct dapl_hash_table *p_table; itamar> - int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); itamar> - u32 dat_status = DAT_SUCCESS; itamar> - int i; itamar> - itamar> - dapl_os_assert(pp_table); itamar> - itamar> - /* Allocate hash table */ itamar> - p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); itamar> - if (!p_table) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); itamar> - goto bail; itamar> - } itamar> - itamar> - /* Init hash table, allocate and init and buckets */ itamar> - memset(p_table, 0, sizeof *p_table); itamar> - p_table->tbl_size = table_size; itamar> - p_table->table = kmalloc(table_length, GFP_ATOMIC); itamar> - if (!p_table->table) { itamar> - kfree(p_table); itamar> - dat_status = itamar> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); itamar> - goto bail; itamar> - } itamar> - itamar> - spin_lock_init(&p_table->lock); itamar> - for (i = 0; i < table_size; i++) { itamar> - p_table->table[i].datum = NO_DATUM_VALUE; itamar> - p_table->table[i].key = 0; itamar> - p_table->table[i].next_element = NULL; itamar> - } itamar> - itamar> - *pp_table = p_table; itamar> - itamar> -bail: itamar> - return DAT_SUCCESS; itamar> -} itamar> - itamar> -/* itamar> - * Destroy a hash table itamar> - */ itamar> -u32 dapl_hash_free(struct dapl_hash_table *p_table) itamar> -{ itamar> - dapl_os_assert(p_table && p_table->table); itamar> - itamar> - /* no need to destroy p_table->lock */ itamar> - kfree(p_table->table); itamar> - kfree(p_table); itamar> - itamar> - return DAT_SUCCESS; itamar> -} itamar> - itamar> -/* itamar> - * Returns the number of elements stored in the table itamar> - */ itamar> - itamar> -u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size) itamar> -{ itamar> - dapl_os_assert(p_table && p_size); itamar> - itamar> - *p_size = p_table->num_entries; itamar> - itamar> - return DAT_SUCCESS; itamar> -} itamar> - itamar> -/* itamar> - * Inserts the specified data into the table with the given key. itamar> - * Duplicates are not expected, and return in error, having done nothing. itamar> - */ itamar> - itamar> -u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, itamar> - DAPL_HASH_DATA data) itamar> -{ itamar> - u32 dat_status = DAT_SUCCESS; itamar> - itamar> - dapl_os_assert(p_table); itamar> - itamar> - spin_lock_irqsave(&p_table->lock, p_table->flags); itamar> - if (!dapl_hash_add(p_table, key, data, FALSE, NULL)) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); itamar> - } itamar> - spin_unlock_irqrestore(&p_table->lock, p_table->flags); itamar> - itamar> - return dat_status; itamar> -} itamar> - itamar> -/* itamar> - * Searches for the given key. If found, itamar> - * DAT_SUCCESS is returned and the associated itamar> - * data is returned in the DAPL_HASH_DATA itamar> - * pointer if that pointer is not NULL. itamar> - */ itamar> -u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, itamar> - DAPL_HASH_DATA *p_data) itamar> -{ itamar> - u32 dat_status; itamar> - void *olddatum; itamar> - DAPL_HASH_ELEM *found; itamar> - itamar> - dapl_os_assert(p_table); itamar> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, 0); itamar> - itamar> - spin_lock_irqsave(&p_table->lock, p_table->flags); itamar> - DAPL_HASHLOOKUP(p_table, key, olddatum, found); itamar> - spin_unlock_irqrestore(&p_table->lock, p_table->flags); itamar> - itamar> - if (found) { itamar> - if (p_data) { itamar> - *p_data = olddatum; itamar> - } itamar> - dat_status = DAT_SUCCESS; itamar> - } itamar> - itamar> - return dat_status; itamar> -} itamar> - itamar> -u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, itamar> - DAPL_HASH_DATA *p_data) itamar> -{ itamar> - u32 dat_status; itamar> - DAPL_HASH_KEY hashValue, save_key = key; itamar> - itamar> - dapl_os_assert(p_table); itamar> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, 0); itamar> - itamar> - if (p_table->num_entries == 0) { itamar> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, itamar> - "dapl_hash_remove () called on empty hash table!\n"); itamar> - return dat_status; itamar> - } itamar> - itamar> - hashValue = DAPL_DOHASH(key, p_table->tbl_size); itamar> - key = save_key; itamar> - spin_lock_irqsave(&p_table->lock, p_table->flags); itamar> - if (dapl_hash_delete_element(&p_table->table[hashValue], key, p_data)) { itamar> - p_table->num_entries--; itamar> - dat_status = DAT_SUCCESS; itamar> - } itamar> - spin_unlock_irqrestore(&p_table->lock, p_table->flags); itamar> - itamar> - return dat_status; itamar> -} itamar> diff -Nurp -X dontdiff dat-provider_bind/dapl_hash.h dat-provider/dapl_hash.h itamar> --- dat-provider_bind/dapl_hash.h Sun Jun 19 15:17:54 2005 itamar> +++ dat-provider/dapl_hash.h Thu Jan 1 02:00:00 1970 itamar> @@ -1,81 +0,0 @@ itamar> -/* itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. itamar> - * itamar> - * This Software is licensed under one of the following licenses: itamar> - * itamar> - * 1) under the terms of the "Common Public License 1.0" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/cpl.php. itamar> - * itamar> - * 2) under the terms of the "The BSD License" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/bsd-license.php. itamar> - * itamar> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a itamar> - * copy of which is available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/gpl-license.php. itamar> - * itamar> - * Licensee has the right to choose one of the above licenses. itamar> - * itamar> - * Redistributions of source code must retain the above copyright itamar> - * notice and one of the license notices. itamar> - * itamar> - * Redistributions in binary form must reproduce both the above copyright itamar> - * notice, one of the license notices in the documentation itamar> - * and/or other materials provided with the distribution. itamar> - */ itamar> - itamar> -/* itamar> - * $Id: dapl_hash.h 2640 2005-06-16 16:22:46Z jlentini $ itamar> - */ itamar> - itamar> -#ifndef DAPL_HASH_H itamar> -#define DAPL_HASH_H itamar> - itamar> -#include "dapl.h" itamar> - itamar> -/********************************************************************* itamar> - * * itamar> - * Defines * itamar> - * * itamar> - *********************************************************************/ itamar> - itamar> -/* itamar> - * Hash table size. itamar> - * itamar> - * Default is small; use the larger sample values for hash tables itamar> - * known to be heavily used. The sample values chosen are the itamar> - * largest primes below 2^8, 2^9, and 2^10. itamar> - */ itamar> -#define DAPL_DEF_HASHSIZE 251 itamar> -#define DAPL_MED_HASHSIZE 509 itamar> -#define DAPL_LRG_HASHSIZE 1021 itamar> - itamar> -#define DAPL_HASH_TABLE_DEFAULT_CAPACITY DAPL_DEF_HASHSIZE itamar> - itamar> -/* The hash function */ itamar> -#define DAPL_DOHASH(key,hashsize) dapl_os_mod64(key,hashsize) itamar> - itamar> -/********************************************************************* itamar> - * * itamar> - * Function Prototypes * itamar> - * * itamar> - *********************************************************************/ itamar> - itamar> -extern u32 dapl_hash_create(int capacity, struct dapl_hash_table **pp_table); itamar> - itamar> -extern u32 dapl_hash_free(struct dapl_hash_table *p_table); itamar> - itamar> -extern u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size); itamar> - itamar> -extern u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, itamar> - DAPL_HASH_DATA data); itamar> - itamar> -extern u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, itamar> - DAPL_HASH_DATA *p_data); itamar> - itamar> -extern u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, itamar> - DAPL_HASH_DATA *p_data); itamar> - itamar> -#endif /* DAPL_HASH_H */ itamar> diff -Nurp -X dontdiff dat-provider_bind/dapl_hca_util.c dat-provider/dapl_hca_util.c itamar> --- dat-provider_bind/dapl_hca_util.c Sun Jun 19 15:17:54 2005 itamar> +++ dat-provider/dapl_hca_util.c Sun Jun 19 15:24:06 2005 itamar> @@ -34,7 +34,6 @@ itamar> #include "dapl_openib_util.h" itamar> #include "dapl_provider.h" itamar> #include "dapl_hca_util.h" itamar> -#include "dapl_hash.h" itamar> itamar> /* itamar> * dapl_hca_alloc itamar> @@ -60,20 +59,13 @@ struct dapl_hca *dapl_hca_alloc(char *na itamar> if (hca) { itamar> memset(hca, 0, sizeof *hca); itamar> itamar> - if (DAT_SUCCESS == itamar> - dapl_hash_create(DAPL_HASH_TABLE_DEFAULT_CAPACITY, itamar> - &hca->lmr_hash_table)) { itamar> - spin_lock_init(&hca->lock); itamar> - INIT_LIST_HEAD(&hca->ia_list); itamar> - itamar> - hca->name = dapl_os_strdup(name); itamar> - hca->ib_hca_handle = device; itamar> - hca->port_num = port; itamar> - if (hca->name == NULL) { itamar> - kfree(hca); itamar> - hca = NULL; itamar> - } itamar> - } else { itamar> + spin_lock_init(&hca->lock); itamar> + INIT_LIST_HEAD(&hca->ia_list); itamar> + itamar> + hca->name = dapl_os_strdup(name); itamar> + hca->ib_hca_handle = device; itamar> + hca->port_num = port; itamar> + if (hca->name == NULL) { itamar> kfree(hca); itamar> hca = NULL; itamar> } itamar> @@ -99,7 +91,6 @@ struct dapl_hca *dapl_hca_alloc(char *na itamar> */ itamar> void dapl_hca_free(struct dapl_hca *hca) itamar> { itamar> - (void)dapl_hash_free(hca->lmr_hash_table); itamar> kfree(hca->name); itamar> kfree(hca); itamar> } itamar> diff -Nurp -X dontdiff dat-provider_bind/dapl_lmr.c dat-provider/dapl_lmr.c itamar> --- dat-provider_bind/dapl_lmr.c Sun Jun 19 15:17:54 2005 itamar> +++ dat-provider/dapl_lmr.c Sun Jun 19 15:24:06 2005 itamar> @@ -32,7 +32,6 @@ itamar> itamar> #include "dapl_openib_util.h" itamar> #include "dapl_ia.h" itamar> -#include "dapl_hash.h" itamar> itamar> static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, itamar> enum dat_mem_type mem_type, itamar> @@ -104,17 +103,6 @@ static inline u32 dapl_lmr_create_virtua itamar> if (DAT_SUCCESS != status) itamar> goto error2; itamar> itamar> - /* if the LMR context is already in the hash table */ itamar> - status = dapl_hash_search(ia->hca->lmr_hash_table, itamar> - new_lmr->param.lmr_context, NULL); itamar> - if (status == DAT_SUCCESS) itamar> - goto error3; itamar> - itamar> - status = dapl_hash_insert(ia->hca->lmr_hash_table, itamar> - new_lmr->param.lmr_context, lmr); itamar> - if (status != DAT_SUCCESS) itamar> - goto error3; itamar> - itamar> atomic_inc(&pz->pz_ref_count); itamar> itamar> if (lmr) itamar> @@ -130,8 +118,6 @@ static inline u32 dapl_lmr_create_virtua itamar> itamar> return DAT_SUCCESS; itamar> itamar> -error3: itamar> - (void)dapl_ib_mr_deregister(new_lmr); itamar> error2: itamar> dapl_lmr_dealloc(new_lmr); itamar> error1: itamar> @@ -168,17 +154,6 @@ static inline u32 dapl_lmr_create_physic itamar> if (DAT_SUCCESS != status) itamar> goto error2; itamar> itamar> - /* if the LMR context is already in the hash table */ itamar> - status = dapl_hash_search(ia->hca->lmr_hash_table, itamar> - new_lmr->param.lmr_context, NULL); itamar> - if (status == DAT_SUCCESS) itamar> - goto error3; itamar> - itamar> - status = dapl_hash_insert(ia->hca->lmr_hash_table, itamar> - new_lmr->param.lmr_context, lmr); itamar> - if (status != DAT_SUCCESS) itamar> - goto error3; itamar> - itamar> atomic_inc(&pz->pz_ref_count); itamar> itamar> if (lmr) itamar> @@ -194,8 +169,6 @@ static inline u32 dapl_lmr_create_physic itamar> itamar> return DAT_SUCCESS; itamar> itamar> -error3: itamar> - (void)dapl_ib_mr_deregister(new_lmr); itamar> error2: itamar> dapl_lmr_dealloc(new_lmr); itamar> error1: itamar> @@ -216,12 +189,6 @@ static inline u32 dapl_lmr_create_lmr(st itamar> DAT_REGION_DESCRIPTION reg_desc; itamar> u32 status; itamar> itamar> - status = dapl_hash_search(ia->hca->lmr_hash_table, itamar> - original_lmr->param.lmr_context, itamar> - (DAPL_HASH_DATA *) &lmr); itamar> - if (status != DAT_SUCCESS) itamar> - goto error1; itamar> - itamar> reg_desc.for_lmr = (struct dat_lmr *) original_lmr; itamar> itamar> new_lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, itamar> @@ -236,17 +203,6 @@ static inline u32 dapl_lmr_create_lmr(st itamar> if (DAT_SUCCESS != status) itamar> goto error2; itamar> itamar> - /* if the LMR context is already in the hash table */ itamar> - status = dapl_hash_search(ia->hca->lmr_hash_table, itamar> - new_lmr->param.lmr_context, NULL); itamar> - if (status == DAT_SUCCESS) itamar> - goto error3; itamar> - itamar> - status = dapl_hash_insert(ia->hca->lmr_hash_table, itamar> - new_lmr->param.lmr_context, lmr); itamar> - if (status != DAT_SUCCESS) itamar> - goto error3; itamar> - itamar> atomic_inc(&pz->pz_ref_count); itamar> itamar> if (lmr) itamar> @@ -263,8 +219,6 @@ static inline u32 dapl_lmr_create_lmr(st itamar> itamar> return DAT_SUCCESS; itamar> itamar> -error3: itamar> - dapl_ib_mr_deregister(new_lmr); itamar> error2: itamar> dapl_lmr_dealloc(new_lmr); itamar> error1: itamar> @@ -372,22 +326,12 @@ u32 dapl_lmr_free(struct dat_lmr *lmr) itamar> if (0 != atomic_read(&dapl_lmr->lmr_ref_count)) itamar> return DAT_INVALID_STATE; itamar> itamar> - status = dapl_hash_remove( itamar> - dapl_lmr->common.owner_ia->hca->lmr_hash_table, itamar> - dapl_lmr->param.lmr_context, NULL); itamar> - if (status != DAT_SUCCESS) itamar> - goto error; itamar> - itamar> status = dapl_ib_mr_deregister(dapl_lmr); itamar> if (status == DAT_SUCCESS) { itamar> pz = (struct dapl_pz *)dapl_lmr->param.pz; itamar> atomic_dec(&pz->pz_ref_count); itamar> dapl_lmr_dealloc(dapl_lmr); itamar> - } else /* failure; put dapl_lmr back in hash table */ itamar> - dapl_hash_insert(dapl_lmr->common.owner_ia-> itamar> - hca->lmr_hash_table, itamar> - dapl_lmr->param.lmr_context, dapl_lmr); itamar> - itamar> + } itamar> break; itamar> } itamar> case DAT_MEM_TYPE_PLATFORM: itamar> diff -Nurp -X dontdiff dat-provider_bind/dapl_rmr.c dat-provider/dapl_rmr.c itamar> --- dat-provider_bind/dapl_rmr.c Sun Jun 19 15:17:54 2005 itamar> +++ dat-provider/dapl_rmr.c Sun Jun 19 15:24:06 2005 itamar> @@ -33,7 +33,6 @@ itamar> #include "dapl.h" itamar> #include "dapl_ep.h" itamar> #include "dapl_ia.h" itamar> -#include "dapl_hash.h" itamar> #include "dapl_cookie.h" itamar> #include "dapl_openib_util.h" itamar> itamar> diff -Nurp -X dontdiff dat-provider_bind/dapl_util.h dat-provider/dapl_util.h itamar> --- dat-provider_bind/dapl_util.h Sun Jun 19 15:17:54 2005 itamar> +++ dat-provider/dapl_util.h Sun Jun 19 15:24:30 2005 itamar> @@ -144,22 +144,6 @@ static inline char *dapl_os_strdup(const itamar> #endif itamar> itamar> /* itamar> - * dapl_os_mod64 itamar> - * itamar> - * Returne the modulo of a 64 bit number. Given that this is running itamar> - * on a 32 bit platform, we need to use the kernel macro to prevent itamar> - * exceptions from killing the machine. itamar> - */ itamar> -static inline long dapl_os_mod64(uint64_t key, unsigned long hashsize) itamar> -{ itamar> -#ifdef __ia64__ itamar> - return ((uint64_t) ((key) % (hashsize))) itamar> -#else itamar> - return do_div(key, hashsize); itamar> -#endif itamar> -} itamar> - itamar> -/* itamar> * Debug Functions itamar> */ itamar> itamar> -- itamar> Itamar itamar> From jlentini at netapp.com Wed Jun 22 10:04:22 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 13:04:22 -0400 (EDT) Subject: [openib-general] [PATCH][kdapl] Small changes in dapl_hca_alloc/dapl_hca_free functions In-Reply-To: <20050620084616.GA7076@mellanox.co.il> References: <20050620084616.GA7076@mellanox.co.il> Message-ID: Committed in revision 2677. On Mon, 20 Jun 2005, Itamar Rabenstein wrote: itamar> Small changes in dapl_hca_alloc/dapl_hca_free function itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> diff -Nurp -X dontdiff dat-provider_hash/dapl_hca_util.c dat-provider/dapl_hca_util.c itamar> --- dat-provider_hash/dapl_hca_util.c Sun Jun 19 16:12:55 2005 itamar> +++ dat-provider/dapl_hca_util.c Sun Jun 19 16:30:07 2005 itamar> @@ -54,23 +54,18 @@ itamar> struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) itamar> { itamar> struct dapl_hca *hca; itamar> - itamar> - hca = kmalloc(sizeof *hca, GFP_ATOMIC); itamar> + int malloc_size = sizeof *hca + strlen(name) + 1; itamar> + itamar> + hca = kmalloc(malloc_size, GFP_ATOMIC); itamar> if (hca) { itamar> - memset(hca, 0, sizeof *hca); itamar> - itamar> + memset(hca, 0, malloc_size); itamar> spin_lock_init(&hca->lock); itamar> INIT_LIST_HEAD(&hca->ia_list); itamar> - itamar> - hca->name = dapl_os_strdup(name); itamar> + hca->name = (char *)hca + sizeof *hca; itamar> + strcpy(hca->name, name); itamar> hca->ib_hca_handle = device; itamar> hca->port_num = port; itamar> - if (hca->name == NULL) { itamar> - kfree(hca); itamar> - hca = NULL; itamar> - } itamar> } itamar> - itamar> return hca; itamar> } itamar> itamar> @@ -91,7 +86,6 @@ struct dapl_hca *dapl_hca_alloc(char *na itamar> */ itamar> void dapl_hca_free(struct dapl_hca *hca) itamar> { itamar> - kfree(hca->name); itamar> kfree(hca); itamar> } itamar> itamar> diff -Nurp -X dontdiff dat-provider_hash/dapl_util.h dat-provider/dapl_util.h itamar> --- dat-provider_hash/dapl_util.h Sun Jun 19 16:12:56 2005 itamar> +++ dat-provider/dapl_util.h Sun Jun 19 16:30:52 2005 itamar> @@ -121,18 +121,6 @@ static inline void *dapl_os_realloc(void itamar> } itamar> itamar> /* itamar> - * String Functions itamar> - */ itamar> - itamar> -static inline char *dapl_os_strdup(const char *str) itamar> -{ itamar> - char *ns = kmalloc(strlen(str) + 1, GFP_KERNEL); itamar> - if (ns) itamar> - strcpy(ns, str); itamar> - return ns; itamar> -} itamar> - itamar> -/* itamar> * *printf format helper. We use the C string constant concatenation itamar> * ability to define 64 bit formats, which unfortunatly are non standard itamar> * in the C compiler world. itamar> -- itamar> Itamar itamar> _______________________________________________ itamar> openib-general mailing list itamar> openib-general at openib.org itamar> http://openib.org/mailman/listinfo/openib-general itamar> itamar> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general itamar> From tduffy at sun.com Wed Jun 22 10:17:09 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 22 Jun 2005 10:17:09 -0700 Subject: [openib-general] [PATCH] kDAPL: remove dat_os_panic() Message-ID: <1119460629.26537.21.camel@duffman> This patch removes the function dapl_os_panic() in favor of calling panic() directly. Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2677) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -681,7 +681,7 @@ static void dapl_evd_cq_async_error_call cause, context); if (!evd) - dapl_os_panic("NULL == context\n"); + panic("NULL == context\n"); async_evd = evd->common.owner_ia->async_error_evd; @@ -689,9 +689,8 @@ static void dapl_evd_cq_async_error_call DAT_ASYNC_ERROR_EVD_OVERFLOW, async_evd->common.owner_ia); - if (status != DAT_SUCCESS) { - dapl_os_panic("async EVD overflow\n"); - } + if (status != DAT_SUCCESS) + panic("async EVD overflow\n"); dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_cq_async_error_callback () returns\n"); @@ -707,10 +706,8 @@ void dapl_evd_un_async_error_callback(st "dapl_evd_un_async_error_callback (%p, %p)\n", cause, context); - if (NULL == context) { - dapl_os_panic("NULL == context\n"); - return; - } + if (NULL == context) + panic("NULL == context\n"); async_evd = (struct dapl_evd *)context; Index: linux-kernel/dat-provider/dapl_util.h =================================================================== --- linux-kernel/dat-provider/dapl_util.h (revision 2677) +++ linux-kernel/dat-provider/dapl_util.h (working copy) @@ -48,12 +48,6 @@ #include #endif -#define dapl_os_panic(fmt, args...) \ - do { \ - printk("PANIC in %s:%i:%s\n", __FILE__, __LINE__, __func__); \ - panic(fmt, ## args); \ - } while(0) - #define dapl_os_assert(expression) \ do { \ if (!(expression)) { \ From jlentini at netapp.com Wed Jun 22 10:22:35 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 13:22:35 -0400 (EDT) Subject: [openib-general] [PATCH][kdapl] Small changes in dapl_hca_all oc/dapl_hca_free functions In-Reply-To: <1119286100.25973.2.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35CD6@mtvex01.mtv.mtl.com> <1119286100.25973.2.camel@duffman> Message-ID: On Mon, 20 Jun 2005, Tom Duffy wrote: > On Mon, 2005-06-20 at 20:00 +0300, Itamar Rabenstein wrote: >> Hi Tom, >> I was just copied to original code from dapl_os_strdup() to >> dapl_hca_alloc(). >> so if it is a problem is it was there in the code before my patch. >> But why do you think we need strncpy and not strcpy ? >> name param is null terminated string. > > It is generally not a good idea to allow an arbitrary sized string in > the kernel. Especially when you are kmalloc()ing the size. If we ignore the mechanism by which this string is copied for a moment, it turns out the hca structure's name field isn't even necessary. It is only used once, in dapl_add_port(). The code looks roughly like this: hca = dapl_hca_alloc(provider_info.ia_name, device, port); status = dapl_ib_open_hca(hca->name, hca); The code allocs an HCA, initializing the hca->name field to provider_info.ia_name and then makes use of this field for the one and only time immediately afterwards. I'll remove the HCA's name field and use provider_info.ia_name in the call to dapl_ib_open_hca() instead. In the future, if there is some reason the HCA structure should contain this information, it should have a pointer to the dat_provider structure which contains the IA name. That will eliminate duplicating this information. james From halr at voltaire.com Wed Jun 22 10:24:34 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 22 Jun 2005 13:24:34 -0400 Subject: [openib-general] [PATCH] [AT} [TRIVIAL]: Fix typo in header file In-Reply-To: <52r7eus6n2.fsf@topspin.com> References: <1119456919.4480.1.camel@hal.voltaire.com> <52r7eus6n2.fsf@topspin.com> Message-ID: <1119460880.4480.59.camel@hal.voltaire.com> On Wed, 2005-06-22 at 12:22, Roland Dreier wrote: > > - * src_ip can be provide to force specific output interface. > > + * src_ip can be provided to force specific output interface. > > The typo fix is fine but how does srp_ip force a specific output > interface? It's perfectly valid to configure every local network > interface with the same IP address. The comment is not accurate. The description would be more accurate to say that it "forces" the use of one of the IP interfaces which share this source IP address (if a route exists to the destination). In the case of IPoIB, there is only one interface per IP address. -- Hal From itamar at mellanox.co.il Wed Jun 22 11:02:51 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 22 Jun 2005 21:02:51 +0300 Subject: [openib-general] [PATCH][kdapl] remove NULL check for input p arams Message-ID: <91DB792C7985D411BEC300B40080D29CC35CF7@mtvex01.mtv.mtl.com> Good catch (;-) James , can you fix it when you apply the patch. (I can not generate a new patch now , I am not at work) Itamar > -----Original Message----- > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Wednesday, June 22, 2005 6:26 PM > To: Itamar Rabenstein > Cc: James.Lentini at netapp.com; openib-general at openib.org > Subject: Re: [openib-general] [PATCH][kdapl] remove NULL > check for input > params > > > On Wed, 2005-06-22 at 17:36 +0300, Itamar Rabenstein wrote: > > diff -Nurp -X dontdiff dat-provider_del_hca/dapl_pz.c > dat-provider/dapl_pz.c > > --- dat-provider_del_hca/dapl_pz.c Mon Jun 20 08:40:28 2005 > > +++ dat-provider/dapl_pz.c Wed Jun 22 11:41:13 2005 > > @@ -75,40 +75,23 @@ u32 dapl_pz_create(struct dat_ia *ia, st > > dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", > > ia, pz); > > > > - if (!ia) { > > - status = DAT_ERROR(DAT_INVALID_HANDLE, > DAT_INVALID_HANDLE_IA); > > - goto error1; > > - } > > - if (NULL == pz) { > > - status = DAT_ERROR(DAT_INVALID_PARAMETER, > DAT_INVALID_ARG2); > > - goto error1; > > - } > > - > > dapl_ia = (struct dapl_ia *)ia; > > > > dapl_pz = dapl_pz_alloc(dapl_ia); > > - if (!dapl_pz) { > > - status = > > - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, > DAT_RESOURCE_MEMORY); > > - goto error1; > > - } > > - > > This is an alloc check, should stay in. > > -tduffy > From jlentini at netapp.com Wed Jun 22 10:51:40 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 13:51:40 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [AT} [TRIVIAL]: Fix typo in header file In-Reply-To: <1119456919.4480.1.camel@hal.voltaire.com> References: <1119456919.4480.1.camel@hal.voltaire.com> Message-ID: Committed in revision 2679. On Wed, 22 Jun 2005, Hal Rosenstock wrote: halr> Fix typo in header file halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: ib_at.h halr> =================================================================== halr> --- ib_at.h (revision 2671) halr> +++ ib_at.h (working copy) halr> @@ -125,7 +125,7 @@ halr> * @async_comp: asynchronous callback structure - optional halr> * halr> * Resolve the specified dst_ip to a &struct ib_route structure. halr> - * src_ip can be provide to force specific output interface. halr> + * src_ip can be provided to force specific output interface. halr> * flags can be used to select resolving method; currently IB-ARP or ATS. halr> * halr> * See ib_at_completion structure documentation for asynchronous halr> halr> From roland at topspin.com Wed Jun 22 10:56:15 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 10:56:15 -0700 Subject: [openib-general] [PATCH] [AT} [TRIVIAL]: Fix typo in header file In-Reply-To: <1119460880.4480.59.camel@hal.voltaire.com> (Hal Rosenstock's message of "22 Jun 2005 13:24:34 -0400") References: <1119456919.4480.1.camel@hal.voltaire.com> <52r7eus6n2.fsf@topspin.com> <1119460880.4480.59.camel@hal.voltaire.com> Message-ID: <52y892qnr4.fsf@topspin.com> Hal> The comment is not accurate. The description would be more Hal> accurate to say that it "forces" the use of one of the IP Hal> interfaces which share this source IP address (if a route Hal> exists to the destination). In the case of IPoIB, there is Hal> only one interface per IP address. Why do you think there's only one interface per IP address? For example, you can easily try the following: # modprobe ib_ipoib # ip addr add dev ib0 192.168.0.2/24 # ip addr add dev ib1 192.168.0.2/24 # ip addr show [...] 7: ib0: mtu 2044 qdisc noop qlen 128 link/[32] 00:02:04:04:fe:80:00:00:00:00:00:00:00:02:c9:01:07:fc:c7:11 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.0.2/24 scope global ib0 8: ib1: mtu 2044 qdisc noop qlen 128 link/[32] 00:02:04:05:fe:80:00:00:00:00:00:00:00:02:c9:01:07:fc:c7:12 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.0.2/24 scope global ib1 - R. From halr at voltaire.com Wed Jun 22 11:08:31 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 22 Jun 2005 14:08:31 -0400 Subject: [openib-general] [PATCH] [AT} [TRIVIAL]: Fix typo in header file In-Reply-To: <52y892qnr4.fsf@topspin.com> References: <1119456919.4480.1.camel@hal.voltaire.com> <52r7eus6n2.fsf@topspin.com> <1119460880.4480.59.camel@hal.voltaire.com> <52y892qnr4.fsf@topspin.com> Message-ID: <1119463710.4480.77.camel@hal.voltaire.com> On Wed, 2005-06-22 at 13:56, Roland Dreier wrote: > Hal> The comment is not accurate. The description would be more > Hal> accurate to say that it "forces" the use of one of the IP > Hal> interfaces which share this source IP address (if a route > Hal> exists to the destination). In the case of IPoIB, there is > Hal> only one interface per IP address. > > Why do you think there's only one interface per IP address? For > example, you can easily try the following: > > # modprobe ib_ipoib > # ip addr add dev ib0 192.168.0.2/24 > # ip addr add dev ib1 192.168.0.2/24 > # ip addr show > [...] > 7: ib0: mtu 2044 qdisc noop qlen 128 > link/[32] 00:02:04:04:fe:80:00:00:00:00:00:00:00:02:c9:01:07:fc:c7:11 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > inet 192.168.0.2/24 scope global ib0 > 8: ib1: mtu 2044 qdisc noop qlen 128 > link/[32] 00:02:04:05:fe:80:00:00:00:00:00:00:00:02:c9:01:07:fc:c7:12 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > inet 192.168.0.2/24 scope global ib1 I was thinking of bonding rather than multihoming with the same address. Aren't there operational issues when configured as you show above ? -- Hal From roland at topspin.com Wed Jun 22 11:17:47 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 11:17:47 -0700 Subject: [openib-general] [PATCH] [AT} [TRIVIAL]: Fix typo in header file In-Reply-To: <1119463710.4480.77.camel@hal.voltaire.com> (Hal Rosenstock's message of "22 Jun 2005 14:08:31 -0400") References: <1119456919.4480.1.camel@hal.voltaire.com> <52r7eus6n2.fsf@topspin.com> <1119460880.4480.59.camel@hal.voltaire.com> <52y892qnr4.fsf@topspin.com> <1119463710.4480.77.camel@hal.voltaire.com> Message-ID: <52is06qmr8.fsf@topspin.com> Hal> Aren't there operational issues when configured as you show above ? Possibly, but it's up to the admin to configure routing properly. My point is simply that it makes no sense to try and map a local IP address to a network interface. If a consumer wants to use a particular interface, then the only sensible way to specify this is to pass the interface. - R. From roland at topspin.com Wed Jun 22 12:00:10 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 12:00:10 -0700 Subject: [openib-general] Re: page allocation failure with kdapl/mthca In-Reply-To: <52d5qfwtrz.fsf@topspin.com> (Roland Dreier's message of "Tue, 21 Jun 2005 09:37:20 -0700") References: <1119366787.4477.26.camel@localhost.localdomain> <52d5qfwtrz.fsf@topspin.com> Message-ID: <52d5qeqksl.fsf@topspin.com> Roland> I just checked in a change that converts from using Roland> pci_alloc_consistent(), which is always GFP_ATOMIC, to Roland> dma_alloc_coherent(..., GFP_KERNEL). This should help Roland> with this symptom. Did this change make any difference for you? - R. From halr at voltaire.com Wed Jun 22 12:15:35 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 22 Jun 2005 15:15:35 -0400 Subject: [openib-general] Re: page allocation failure with kdapl/mthca In-Reply-To: <52d5qeqksl.fsf@topspin.com> References: <1119366787.4477.26.camel@localhost.localdomain> <52d5qfwtrz.fsf@topspin.com> <52d5qeqksl.fsf@topspin.com> Message-ID: <1119467735.4480.95.camel@hal.voltaire.com> On Wed, 2005-06-22 at 15:00, Roland Dreier wrote: > Roland> I just checked in a change that converts from using > Roland> pci_alloc_consistent(), which is always GFP_ATOMIC, to > Roland> dma_alloc_coherent(..., GFP_KERNEL). This should help > Roland> with this symptom. > > Did this change make any difference for you? Yes. That fixed it. Thanks. -- Hal From mst at mellanox.co.il Wed Jun 22 13:17:06 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 22 Jun 2005 23:17:06 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <52zmtis7ns.fsf@topspin.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> Message-ID: <20050622201706.GA2488@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [openib-general] Re: [PATCH] rdma_lat-09 and results > > Michael> Roland, is there some way to get the plugin directory out > Michael> of the libibverbs library? > > Sorry, not sure what you mean. > > - R. > I'd like to add to libibverbs a function returning the path used to find plugins, and a utility calling that function and printing it out. Its easy to do - Roland, OK with you? -- MST From halr at voltaire.com Wed Jun 22 13:17:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 22 Jun 2005 16:17:02 -0400 Subject: [openib-general] [PATCH] [kDAPL} Fix locking problem in some CM callback functions Message-ID: <1119471298.4480.145.camel@hal.voltaire.com> Fix locking problem in some CM callback functions Since modify QP can sleep, dapl_ib_disconnect must be called without holding spinlock. Signed-off-by: Itamar Rabenstein Signed-off-by: Hal Rosenstock Index: dapl_openib_util.h =================================================================== --- dapl_openib_util.h (revision 2681) +++ dapl_openib_util.h (working copy) @@ -125,7 +125,7 @@ void dapl_ib_reinit_ep(struct dapl_ep *ep); -void dapl_ib_disconnect_clean(struct dapl_ep *ep, boolean_t passive); +void dapl_ib_disconnect_clean(struct dapl_ep *ep); u32 dapl_ib_get_async_event(struct ib_event *cause, enum dat_event_number *async_event); Index: dapl_cr.c =================================================================== --- dapl_cr.c (revision 2681) +++ dapl_cr.c (working copy) @@ -397,7 +397,7 @@ struct dapl_ep *ep; struct dapl_evd *evd; u32 status; - + dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, "--> dapl_cr_callback! cm_handle: %p event: %x sp: %p\n", cm_ctx, event, sp); @@ -431,7 +431,6 @@ } status = DAT_INTERNAL_ERROR; /* init to ERR */ - switch (event) { case DAT_CONNECTION_REQUEST_EVENT: /* @@ -479,8 +478,7 @@ /* If someone pulled the plug on the EP or connection, * just exit */ - spin_unlock_irqrestore(&ep->common.lock, - ep->common.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); status = DAT_SUCCESS; /* Set evd = NULL so we don't generate an event below */ evd = NULL; @@ -504,36 +502,22 @@ /* The disconnect has already occurred, we are now * cleaned up and ready to exit */ - spin_unlock_irqrestore(&ep->common.lock, - ep->common.flags); + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); return; } ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, FALSE); spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - + dapl_ib_disconnect_clean(ep); break; case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: - /* - * After posting an accept the requesting node has - * stopped talking. - */ + case DAT_CONNECTION_EVENT_BROKEN: spin_lock_irqsave(&ep->common.lock, ep->common.flags); ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, FALSE); spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - + dapl_ib_disconnect_clean(ep); break; - case DAT_CONNECTION_EVENT_BROKEN: - spin_lock_irqsave(&ep->common.lock, ep->common.flags); - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, FALSE); - spin_unlock_irqrestore(&ep->common.lock, - ep->common.flags); - - break; default: evd = NULL; dapl_os_assert(0); /* shouldn't happen */ Index: dapl_evd.c =================================================================== --- dapl_evd.c (revision 2681) +++ dapl_evd.c (working copy) @@ -757,10 +757,8 @@ * when necessary */ spin_lock_irqsave(&ep->common.lock, ep->common.flags); - switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: - { /* * If we don't have an EP at this point we are very screwed up */ @@ -784,65 +782,28 @@ private_data_size); } spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - break; - } case DAT_CONNECTION_EVENT_DISCONNECTED: - { - /* - * EP is now fully disconnected; initiate any post processing - * to reset the underlying QP and get the EP ready for - * another connection - */ ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->common.lock, - ep->common.flags); - - break; - } - case DAT_CONNECTION_EVENT_PEER_REJECTED: - { - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, TRUE); spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - + dapl_ib_disconnect_clean(ep); break; - } + case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: - { - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - - break; - } case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: - { - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, TRUE); - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - - break; - } case DAT_CONNECTION_EVENT_BROKEN: - { ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep, FALSE); spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); - + dapl_ib_disconnect_clean(ep); break; - } case DAT_CONNECTION_REQUEST_EVENT: default: - { spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); evd = NULL; dapl_os_assert(0); /* shouldn't happen */ break; } - } /* * Post the event Index: dapl_openib_cm.c =================================================================== --- dapl_openib_cm.c (revision 2681) +++ dapl_openib_cm.c (working copy) @@ -553,7 +553,6 @@ * * Input: * ep_ptr pointer to dapl_ep struct - * active Indicates active side of connection * * Output: * none @@ -562,13 +561,12 @@ * void * */ -void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t active) +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr) { int status; dapl_dbg_log(DAPL_DBG_TYPE_CM, - " >>> dapl_ib_disconnect_clean: EP: %p active %d\n", - ep_ptr, active); + " >>> dapl_ib_disconnect_clean: EP: %p\n", ep_ptr); /* * Clean up outstanding connection state From atorrez at lanl.gov Wed Jun 22 13:29:45 2005 From: atorrez at lanl.gov (Alfred Torrez) Date: Wed, 22 Jun 2005 14:29:45 -0600 Subject: [openib-general] User Verbs ibv_ud_pingpong Problem Message-ID: <42B9CA39.6040205@lanl.gov> ibv_ud_pingpong on the client side reports the following (using default size): local address: LID 0x000b, QPN 0x700406, PSN 0x9b207c remote address: LID 0x0007, QPN 0xd40406, PSN 0xc1f79d [ 0] 00700406 [ 4] 00cf36ed [ 8] 0017a224 [ c] 00003800 [10] 026b0000 [14] 00000000 [18] 00007d05 [1c] ff000000 Failed status 2 for wr_id 2 This is 2.6.11.5 kernel, svn version 2631 running on HP X86_64 system, 4X HCA (MT23108) ibv_uc_pingpong and ibv_rc_pingpong work fine. Alfred From halr at voltaire.com Wed Jun 22 13:44:53 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 22 Jun 2005 16:44:53 -0400 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <1118955851.21846.21.camel@duffman> References: <1118774337.22484.16.camel@duffman> <1118775221.24424.21.camel@localhost> <1118955851.21846.21.camel@duffman> Message-ID: <1119473069.4480.167.camel@hal.voltaire.com> On Thu, 2005-06-16 at 17:04, Tom Duffy wrote: > On Tue, 2005-06-14 at 11:53 -0700, Matt Leininger wrote: > > On Tue, 2005-06-14 at 11:38 -0700, Tom Duffy wrote: > > > Can the folks at Sandia please update the http://www.openib.org/doc.html > > > page to point to the wiki site? > > > > > > Also, we should move the FAQ's to WIKI. I can do this if you want. > > > > > I'll add a link to doc.html pointing to the wiki. Once the FAQ's are > > moved to the wiki I can just have doc.html point directly to the wiki. > > I have moved the main FAQ to WIKI and updated it. > > https://openib.org/tiki/tiki-index.php?page=OpenIBFAQ > > You can go ahead and remove all the rest of the *links* on the main > doc.html page. The rest is are pointed to within the Wiki. This is a great start. I have a few nits, etc.: Core ConnectionManager s.b. CommunicationManager UPLs s.b. ULPs They should be broken down by kernel and user space MPI is user space. All the rest are kernel (although I think iSER can run in both). uDAPL should be added to user space. Userspace Libraries user verbs user CM user SA client OpenSM Diagnostics Examples perftest HCA flash tools tvflash mstflint -- Hal From jlentini at netapp.com Wed Jun 22 14:13:03 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 17:13:03 -0400 (EDT) Subject: [openib-general] Re: [PATCH] cleanup dat provider registration In-Reply-To: <20050617163211.GA19071@lst.de> References: <20050617163211.GA19071@lst.de> Message-ID: This is an excellent simplification. Committed in revision 2682 with a few minor modifications: - kept printouts in dat_registry_add_provider, dat_registry_remove_provider, and dat_registry_list_providers - updated printout in dat_ia_close (this wasn't something you changed) - removed parens around sizeof - removed space in front of labels The last two are for consistency with the coding style we've been using. If we've deviated from what is acceptable, let us know. Given this simplification, I can think of a few more changes: - rename api.c to registry.c - remove all "dictionary" references: rename dat_dictionary_search to dat_provider_list_search, rename struct dat_dictionary_entry to struct dat_provider_list_entry, rename dat_dictionary_key_is_equal() to dat_provider_info_is_equal() Anyone have opinions for or against? On Fri, 17 Jun 2005, Christoph Hellwig wrote: hch> This huge patch revamps almost everything under dat, and merges the three hch> layers of external API, dynamic registry and dictinary into a single hch> one, greatly streamlining all of the code. hch> hch> Note that I don't have any RDMA hardware to actually takes this, so it hch> probably needs some testing first. hch> hch> hch> Signed-off-by: Christoph Hellwig hch> hch> Index: linux-kernel/dat/dr.c hch> =================================================================== hch> --- linux-kernel/dat/dr.c (revision 2647) hch> +++ linux-kernel/dat/dr.c (working copy) hch> @@ -1,242 +0,0 @@ hch> -/* hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. hch> - * hch> - * This Software is licensed under one of the following licenses: hch> - * hch> - * 1) under the terms of the "Common Public License 1.0" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/cpl.php. hch> - * hch> - * 2) under the terms of the "The BSD License" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/bsd-license.php. hch> - * hch> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a hch> - * copy of which is available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/gpl-license.php. hch> - * hch> - * Licensee has the right to choose one of the above licenses. hch> - * hch> - * Redistributions of source code must retain the above copyright hch> - * notice and one of the license notices. hch> - * hch> - * Redistributions in binary form must reproduce both the above copyright hch> - * notice, one of the license notices in the documentation hch> - * and/or other materials provided with the distribution. hch> - */ hch> - hch> -/* hch> - * Dynamic Registry (DR) functions and data structures hch> - * hch> - * $Id$ hch> - */ hch> - hch> -#include "core.h" hch> -#include "dictionary.h" hch> -#include "dr.h" hch> - hch> -struct dat_dr_entry { hch> - int ref_count; hch> - struct dat_provider_info info; hch> - DAT_IA_OPEN_FUNC ia_open_func; hch> -}; hch> - hch> -static spinlock_t g_dr_lock = SPIN_LOCK_UNLOCKED; hch> -static struct dat_dictionary g_dr_dictionary = hch> - DAT_DICTIONARY_INIT(g_dr_dictionary); hch> - hch> -void dat_dr_fini(void) hch> -{ hch> - dat_dictionary_destroy(&g_dr_dictionary); hch> -} hch> - hch> -u32 dat_dr_insert(const struct dat_provider_info *info, hch> - const DAT_IA_OPEN_FUNC ia_open_func) hch> -{ hch> - struct dat_dr_entry *data; hch> - struct dat_dictionary_entry *dict_entry; hch> - unsigned long flags; hch> - u32 status; hch> - hch> - data = kmalloc(sizeof *data, GFP_ATOMIC); hch> - if (!data) { hch> - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, hch> - DAT_RESOURCE_MEMORY); hch> - goto bail; hch> - } hch> - hch> - data->ref_count = 0; hch> - data->info = *info; hch> - data->ia_open_func = ia_open_func; hch> - hch> - dict_entry = NULL; hch> - status = dat_dictionary_entry_create(&dict_entry); hch> - if (DAT_SUCCESS != status) hch> - goto bail; hch> - hch> - spin_lock_irqsave(&g_dr_lock, flags); hch> - hch> - status = dat_dictionary_insert(&g_dr_dictionary, info, dict_entry, hch> - data); hch> - hch> - spin_unlock_irqrestore(&g_dr_lock, flags); hch> - hch> -bail: hch> - if (DAT_SUCCESS != status) { hch> - if (NULL != data) hch> - kfree(data); hch> - hch> - if (NULL != dict_entry) hch> - dat_dictionary_entry_destroy(dict_entry); hch> - } hch> - hch> - return status; hch> -} hch> - hch> -u32 dat_dr_remove(const struct dat_provider_info *info) hch> -{ hch> - struct dat_dr_entry *data; hch> - struct dat_dictionary_entry *dict_entry; hch> - unsigned long flags; hch> - u32 status; hch> - hch> - dict_entry = NULL; hch> - spin_lock_irqsave(&g_dr_lock, flags); hch> - hch> - status = dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); hch> - hch> - if (DAT_SUCCESS != status) hch> - goto bail; /* return status from dat_dictionary_search() */ hch> - hch> - if (0 != data->ref_count) { hch> - status = DAT_ERROR(DAT_PROVIDER_IN_USE, 0); hch> - goto bail; hch> - } hch> - hch> - status = dat_dictionary_remove(&g_dr_dictionary, info, &dict_entry, hch> - (void **) &data); hch> - if (DAT_SUCCESS != status) hch> - goto bail; /* return status from dat_dictionary_remove() */ hch> - hch> - kfree(data); hch> - hch> -bail: hch> - spin_unlock_irqrestore(&g_dr_lock, flags); hch> - hch> - if (NULL != dict_entry) hch> - dat_dictionary_entry_destroy(dict_entry); hch> - hch> - return status; hch> -} hch> - hch> -u32 dat_dr_provider_open(const struct dat_provider_info *info, hch> - DAT_IA_OPEN_FUNC *p_ia_open_func) hch> -{ hch> - struct dat_dr_entry *data; hch> - unsigned long flags; hch> - u32 status; hch> - hch> - spin_lock_irqsave(&g_dr_lock, flags); hch> - hch> - status = dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); hch> - hch> - spin_unlock_irqrestore(&g_dr_lock, flags); hch> - hch> - if (DAT_SUCCESS == status) { hch> - data->ref_count++; hch> - *p_ia_open_func = data->ia_open_func; hch> - } hch> - hch> - return status; hch> -} hch> - hch> -u32 dat_dr_provider_close(const struct dat_provider_info *info) hch> -{ hch> - struct dat_dr_entry *data; hch> - unsigned long flags; hch> - u32 status; hch> - hch> - spin_lock_irqsave(&g_dr_lock, flags); hch> - hch> - status = dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); hch> - hch> - spin_unlock_irqrestore(&g_dr_lock, flags); hch> - hch> - if (DAT_SUCCESS == status) hch> - data->ref_count--; hch> - hch> - return status; hch> -} hch> - hch> -u32 dat_dr_size(int *size) hch> -{ hch> - return dat_dictionary_size(&g_dr_dictionary, size); hch> -} hch> - hch> -u32 dat_dr_list(int max_to_return, int *entries_returned, hch> - struct dat_provider_info *(dat_provider_list[])) hch> -{ hch> - struct dat_dr_entry **array = NULL; hch> - int i, array_size; hch> - unsigned long flags; hch> - u32 status = DAT_SUCCESS; hch> - hch> - /* hch> - * The dictionary size may increase between the call to hch> - * dat_dictionary_size() and dat_dictionary_enumerate(). hch> - * Therefore we loop until a successful enumeration is made. hch> - */ hch> - *entries_returned = 0; hch> - for (;;) { hch> - status = dat_dictionary_size(&g_dr_dictionary, &array_size); hch> - if (DAT_SUCCESS != status) hch> - goto bail; hch> - hch> - if (0 == array_size) { hch> - status = DAT_SUCCESS; hch> - goto bail; hch> - } hch> - hch> - array = kmalloc(array_size * sizeof *array, GFP_ATOMIC); hch> - if (!array) { hch> - status = hch> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, hch> - DAT_RESOURCE_MEMORY); hch> - goto bail; hch> - } hch> - hch> - spin_lock_irqsave(&g_dr_lock, flags); hch> - hch> - status = dat_dictionary_enumerate(&g_dr_dictionary, hch> - (void **) array, array_size); hch> - hch> - spin_unlock_irqrestore(&g_dr_lock, flags); hch> - hch> - if (DAT_SUCCESS == status) hch> - break; hch> - else { hch> - kfree(array); hch> - array = NULL; hch> - continue; hch> - } hch> - } hch> - hch> - for (i = 0; (i < max_to_return) && (i < array_size); i++) { hch> - if (NULL == dat_provider_list[i]) { hch> - status = hch> - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); hch> - goto bail; hch> - } hch> - hch> - *dat_provider_list[i] = array[i]->info; hch> - } hch> - hch> - *entries_returned = i; hch> - hch> -bail: hch> - if (NULL != array) hch> - kfree(array); hch> - hch> - return status; hch> -} hch> Index: linux-kernel/dat/dictionary.c hch> =================================================================== hch> --- linux-kernel/dat/dictionary.c (revision 2647) hch> +++ linux-kernel/dat/dictionary.c (working copy) hch> @@ -1,229 +0,0 @@ hch> -/* hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. hch> - * hch> - * This Software is licensed under one of the following licenses: hch> - * hch> - * 1) under the terms of the "Common Public License 1.0" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/cpl.php. hch> - * hch> - * 2) under the terms of the "The BSD License" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/bsd-license.php. hch> - * hch> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a hch> - * copy of which is available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/gpl-license.php. hch> - * hch> - * Licensee has the right to choose one of the above licenses. hch> - * hch> - * Redistributions of source code must retain the above copyright hch> - * notice and one of the license notices. hch> - * hch> - * Redistributions in binary form must reproduce both the above copyright hch> - * notice, one of the license notices in the documentation hch> - * and/or other materials provided with the distribution. hch> - */ hch> - hch> -/* hch> - * A dictionary data structure implemented with a linked list hch> - * hch> - * $Id$ hch> - */ hch> - hch> -#include "dictionary.h" hch> - hch> -static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, hch> - struct dat_provider_info *new_key); hch> - hch> -static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *a, hch> - const struct dat_provider_info *b); hch> - hch> -void dat_dictionary_destroy(struct dat_dictionary *dictionary) hch> -{ hch> - struct list_head *cur_list, *next_list; hch> - struct dat_dictionary_entry *cur_entry; hch> - hch> - BUG_ON(NULL == dictionary); hch> - hch> - list_for_each_safe(cur_list, next_list, &dictionary->list) { hch> - cur_entry = list_entry(cur_list, struct dat_dictionary_entry, hch> - list); hch> - list_del(&cur_entry->list); hch> - kfree(cur_entry); hch> - } hch> -} hch> - hch> -u32 dat_dictionary_size(const struct dat_dictionary *dictionary, int *size) hch> -{ hch> - BUG_ON(NULL == dictionary); hch> - BUG_ON(NULL == size); hch> - hch> - *size = dictionary->size; hch> - hch> - return DAT_SUCCESS; hch> -} hch> - hch> -u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry) hch> -{ hch> - struct dat_dictionary_entry *entry; hch> - u32 status = DAT_SUCCESS; hch> - hch> - BUG_ON(NULL == p_entry); hch> - hch> - entry = kmalloc(sizeof *entry, GFP_ATOMIC); hch> - if (!entry) { hch> - status = hch> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); hch> - goto bail; hch> - } hch> - hch> - *p_entry = entry; hch> - hch> -bail: hch> - return status; hch> -} hch> - hch> -void dat_dictionary_entry_destroy(const struct dat_dictionary_entry *entry) hch> -{ hch> - kfree(entry); hch> -} hch> - hch> -u32 dat_dictionary_insert(struct dat_dictionary *dictionary, hch> - const struct dat_provider_info *key, hch> - struct dat_dictionary_entry *entry, void *data) hch> -{ hch> - u32 status; hch> - hch> - BUG_ON(NULL == dictionary); hch> - BUG_ON(NULL == entry); hch> - hch> - if (DAT_SUCCESS == dat_dictionary_search(dictionary, key, NULL)) { hch> - status = DAT_ERROR(DAT_PROVIDER_ALREADY_REGISTERED, 0); hch> - goto bail; hch> - } hch> - hch> - status = dat_dictionary_key_dup(key, &entry->key); hch> - if (DAT_SUCCESS != status) hch> - goto bail; hch> - hch> - entry->data = data; hch> - hch> - /* insert node at end of list to preserve registration order */ hch> - list_add_tail(&entry->list, &dictionary->list); hch> - hch> - dictionary->size++; hch> - hch> -bail: hch> - return status; hch> -} hch> - hch> -u32 dat_dictionary_search(struct dat_dictionary *dictionary, hch> - const struct dat_provider_info *key, void **p_data) hch> -{ hch> - struct dat_dictionary_entry *cur_entry; hch> - u32 status; hch> - hch> - BUG_ON(NULL == dictionary); hch> - BUG_ON(NULL == key); hch> - hch> - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); hch> - hch> - list_for_each_entry(cur_entry, &dictionary->list, list) { hch> - if (TRUE == dat_dictionary_key_is_equal(&cur_entry->key, key)) { hch> - if (NULL != p_data) hch> - *p_data = cur_entry->data; hch> - hch> - status = DAT_SUCCESS; hch> - goto bail; hch> - } hch> - } hch> - hch> -bail: hch> - return status; hch> -} hch> - hch> -u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, void *array[], hch> - int array_size) hch> -{ hch> - struct dat_dictionary_entry *cur_entry; hch> - int i; hch> - u32 status; hch> - hch> - BUG_ON(NULL == dictionary); hch> - BUG_ON(NULL == array); hch> - hch> - status = DAT_SUCCESS; hch> - hch> - if (array_size < dictionary->size) { hch> - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); hch> - goto bail; hch> - } hch> - hch> - i = 0; hch> - list_for_each_entry(cur_entry, &dictionary->list, list) hch> - array[i++] = cur_entry->data; hch> - hch> -bail: hch> - return status; hch> -} hch> - hch> -u32 dat_dictionary_remove(struct dat_dictionary *dictionary, hch> - const struct dat_provider_info *key, hch> - struct dat_dictionary_entry **p_entry, void **p_data) hch> -{ hch> - struct list_head *cur_list, *next_list; hch> - struct dat_dictionary_entry *cur_entry; hch> - u32 status; hch> - hch> - BUG_ON(NULL == dictionary); hch> - BUG_ON(NULL == key); hch> - BUG_ON(NULL == p_entry); hch> - hch> - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); hch> - hch> - list_for_each_safe(cur_list, next_list, &dictionary->list) { hch> - cur_entry = list_entry(cur_list, struct dat_dictionary_entry, hch> - list); hch> - if (TRUE == dat_dictionary_key_is_equal(&cur_entry->key, key)) { hch> - if (NULL != p_data) hch> - *p_data = cur_entry->data; hch> - hch> - list_del(&cur_entry->list); hch> - hch> - *p_entry = cur_entry; hch> - hch> - dictionary->size--; hch> - hch> - status = DAT_SUCCESS; hch> - goto bail; hch> - } hch> - } hch> - hch> -bail: hch> - return status; hch> -} hch> - hch> -static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, hch> - struct dat_provider_info *new_key) hch> -{ hch> - BUG_ON(NULL == old_key); hch> - BUG_ON(NULL == new_key); hch> - hch> - strncpy(new_key->ia_name, old_key->ia_name, DAT_NAME_MAX_LENGTH); hch> - return DAT_SUCCESS; hch> -} hch> - hch> -static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *a, hch> - const struct dat_provider_info *b) hch> -{ hch> - BUG_ON(NULL == a); hch> - BUG_ON(NULL == b); hch> - hch> - if ((strlen(a->ia_name) == strlen(b->ia_name)) && hch> - (!strncmp(a->ia_name, b->ia_name, strlen(a->ia_name)))) hch> - return TRUE; hch> - else hch> - return FALSE; hch> -} hch> Index: linux-kernel/dat/core.c hch> =================================================================== hch> --- linux-kernel/dat/core.c (revision 2647) hch> +++ linux-kernel/dat/core.c (working copy) hch> @@ -1,75 +0,0 @@ hch> -/* hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. hch> - * hch> - * This Software is licensed under one of the following licenses: hch> - * hch> - * 1) under the terms of the "Common Public License 1.0" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/cpl.php. hch> - * hch> - * 2) under the terms of the "The BSD License" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/bsd-license.php. hch> - * hch> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a hch> - * copy of which is available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/gpl-license.php. hch> - * hch> - * Licensee has the right to choose one of the above licenses. hch> - * hch> - * Redistributions of source code must retain the above copyright hch> - * notice and one of the license notices. hch> - * hch> - * Redistributions in binary form must reproduce both the above copyright hch> - * notice, one of the license notices in the documentation hch> - * and/or other materials provided with the distribution. hch> - */ hch> - hch> -/* hch> - * Core DAT registry functions and data structures hch> - * hch> - * $Id$ hch> - */ hch> - hch> -#include "core.h" hch> -#include "dr.h" hch> - hch> -MODULE_LICENSE("Dual BSD/GPL"); hch> -MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); hch> -MODULE_AUTHOR("James Lentini"); hch> - hch> -static DAT_DBG_MASK g_dbg_mask = DAT_DBG_TYPE_ERROR; hch> -module_param_named(dbg_mask, g_dbg_mask, int, 0644); hch> -MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug message types."); hch> - hch> -void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...) hch> -{ hch> - static char buf[1024]; hch> - hch> - if (type & g_dbg_mask) { hch> - va_list args; hch> - hch> - va_start(args, fmt); hch> - vsnprintf(buf, sizeof buf, fmt, args); hch> - printk(KERN_ALERT "DAT: %s", buf); hch> - va_end(args); hch> - } hch> -} hch> - hch> -static int __init dat_init(void) hch> -{ hch> - dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry started\n"); hch> - return 0; hch> -} hch> - hch> -module_init(dat_init); hch> - hch> -static void __exit dat_fini(void) hch> -{ hch> - dat_dr_fini(); hch> - hch> - dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry stopped\n"); hch> -} hch> - hch> -module_exit(dat_fini); hch> Index: linux-kernel/dat/api.c hch> =================================================================== hch> --- linux-kernel/dat/api.c (revision 2647) hch> +++ linux-kernel/dat/api.c (working copy) hch> @@ -26,15 +26,53 @@ hch> * and/or other materials provided with the distribution. hch> */ hch> hch> -/* hch> - * DAT functions for consumers hch> - * hch> - * $Id: consumer.c 2251 2005-05-04 05:57:41Z jlentini $ hch> - */ hch> +#include hch> +#include hch> +#include hch> +#include hch> +#include hch> hch> -#include "core.h" hch> -#include "dr.h" hch> +#include hch> hch> + hch> +enum dat_dbg_type { hch> + DAT_DBG_TYPE_ERROR = (1 << 0), hch> + DAT_DBG_TYPE_GENERIC = (1 << 1), hch> + DAT_DBG_TYPE_DR = (1 << 2), hch> + DAT_DBG_TYPE_PROVIDER_API = (1 << 3), hch> + DAT_DBG_TYPE_CONSUMER_API = (1 << 4), hch> + DAT_DBG_TYPE_ALL = 0x1f hch> +}; hch> + hch> +static int dat_dbg_mask = DAT_DBG_TYPE_ERROR; hch> +module_param_named(dbg_mask, dat_dbg_mask, int, 0644); hch> +MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug message types."); hch> + hch> +struct dat_dictionary_entry { hch> + struct list_head list; hch> + struct dat_provider_info info; hch> + DAT_IA_OPEN_FUNC ia_open_func; hch> + int ref_count; hch> +}; hch> + hch> +static LIST_HEAD(dat_provider_list); hch> +static int dat_provider_list_size; hch> +static DEFINE_SPINLOCK(dat_provider_list_lock); hch> + hch> +static void dat_dbg_print(enum dat_dbg_type type, const char *fmt, ...) hch> +{ hch> + static char buf[1024]; hch> + hch> + if (type & dat_dbg_mask) { hch> + va_list args; hch> + hch> + va_start(args, fmt); hch> + vsnprintf(buf, sizeof buf, fmt, args); hch> + printk(KERN_ALERT "DAT: %s", buf); hch> + va_end(args); hch> + } hch> +} hch> + hch> static u32 dat_strerror_major(u32 value, const char **message) hch> { hch> switch (DAT_GET_TYPE(value)) { hch> @@ -350,21 +388,39 @@ hch> } hch> } hch> hch> +static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *a, hch> + const struct dat_provider_info *b) hch> +{ hch> + if (strlen(a->ia_name) == strlen(b->ia_name) && hch> + !strncmp(a->ia_name, b->ia_name, strlen(a->ia_name))) hch> + return 1; hch> + return 0; hch> +} hch> + hch> +static struct dat_dictionary_entry *dat_dictionary_search( hch> + struct list_head *list, const struct dat_provider_info *key) hch> +{ hch> + struct dat_dictionary_entry *entry; hch> + hch> + list_for_each_entry(entry, list, list) hch> + if (dat_dictionary_key_is_equal(&entry->info, key)) hch> + return entry; hch> + hch> + return NULL; hch> +} hch> + hch> u32 dat_ia_open(const char *name, int async_event_qlen, hch> struct dat_evd **async_event_handle, struct dat_ia **ia) hch> { hch> - DAT_IA_OPEN_FUNC ia_open_func; hch> struct dat_provider_info info; hch> - u32 status; hch> + struct dat_dictionary_entry *data; hch> + unsigned long flags; hch> size_t len; hch> hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, hch> "%s(name:%s, async_qlen:%i, ver:%x.%x, thrd_safe:%x)\n", hch> __func__, name, async_event_qlen); hch> hch> - if (DAT_IS_BAD_POINTER(name)) hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); hch> - hch> len = strlen(name); hch> hch> if (DAT_NAME_MAX_LENGTH < len) hch> @@ -373,62 +429,67 @@ hch> strncpy(info.ia_name, name, len); hch> info.ia_name[len] = '\0'; hch> hch> - status = dat_dr_provider_open(&info, &ia_open_func); hch> - if (status != DAT_SUCCESS) { hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); hch> + data = dat_dictionary_search(&dat_provider_list, &info); hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); hch> + hch> + if (!data) { hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, hch> "%s: IA [%s] not found in dynamic registry\n", hch> __func__, name); hch> hch> - return status; hch> + return DAT_ERROR(DAT_PROVIDER_NOT_FOUND, hch> + DAT_NAME_NOT_REGISTERED); hch> } hch> hch> - return ia_open_func(name, async_event_qlen, async_event_handle, ia); hch> + data->ref_count++; hch> + return data->ia_open_func(name, async_event_qlen, async_event_handle, hch> + ia); hch> } hch> - hch> EXPORT_SYMBOL(dat_ia_open); hch> hch> -u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) hch> +u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags close_flags) hch> { hch> struct dat_provider *provider; hch> - struct dat_provider_attr provider_attr; hch> - u32 status; hch> + struct dat_provider_info info; hch> + u32 status = DAT_SUCCESS; hch> const char *ia_name; hch> + struct dat_dictionary_entry *data; hch> + unsigned long flags; hch> + size_t len; hch> hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, hch> - "%s(ia:%x, flags:%x)\n", __func__, ia, flags); hch> + "%s(ia:%x, flags:%x)\n", __func__, ia, close_flags); hch> hch> provider = ia->common.provider; hch> ia_name = provider->device_name; hch> hch> - if (DAT_SUCCESS != (status = dat_ia_query(ia, NULL, NULL, hch> - &provider_attr))) hch> + status = provider->ia_close_func(ia, close_flags); hch> + if (status != DAT_SUCCESS) { hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, hch> - "dat_ia_query for IA %s failed\n", ia_name); hch> - else if (DAT_SUCCESS != (status = provider->ia_close_func(ia, flags))) hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, hch> "close function for IA %s failed\n", ia_name); hch> - else { hch> - struct dat_provider_info info; hch> - size_t len; hch> + return status; hch> + } hch> hch> - len = strlen(ia_name); hch> + len = strlen(ia_name); hch> hch> - BUG_ON(DAT_NAME_MAX_LENGTH < len); hch> + BUG_ON(DAT_NAME_MAX_LENGTH < len); hch> hch> - strncpy(info.ia_name, ia_name, len); hch> - info.ia_name[len] = '\0'; hch> + strncpy(info.ia_name, ia_name, len); hch> + info.ia_name[len] = '\0'; hch> hch> - status = dat_dr_provider_close(&info); hch> - if (DAT_SUCCESS != status) hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, hch> - "dynamic registry unable to close " hch> - "provider for IA %s\n", hch> - ia_name); hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); hch> + data = dat_dictionary_search(&dat_provider_list, &info); hch> + if (data) { hch> + data->ref_count--; hch> + } else { hch> + status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, hch> + DAT_NAME_NOT_REGISTERED); hch> } hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); hch> hch> return status; hch> } hch> - hch> EXPORT_SYMBOL(dat_ia_close); hch> hch> u32 dat_strerror(u32 value, const char **major_message, hch> @@ -454,69 +515,138 @@ hch> hch> return DAT_SUCCESS; hch> } hch> - hch> EXPORT_SYMBOL(dat_strerror); hch> hch> u32 dat_registry_add_provider(const struct dat_provider *provider, hch> - const struct dat_provider_info *provider_info) hch> + const struct dat_provider_info *info) hch> { hch> - dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, hch> - "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", hch> - __func__, provider_info->ia_name); hch> + struct dat_dictionary_entry *entry; hch> + unsigned long flags; hch> + u32 status = DAT_SUCCESS; hch> hch> - if (DAT_IS_BAD_POINTER(provider)) hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); hch> + entry = kmalloc(sizeof(*entry), GFP_ATOMIC); hch> + if (!entry) { hch> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, hch> + DAT_RESOURCE_MEMORY); hch> + goto out; hch> + } hch> hch> - if (DAT_IS_BAD_POINTER(provider_info)) hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); hch> + entry->ref_count = 0; hch> + entry->info = *info; hch> + entry->ia_open_func = provider->ia_open_func; hch> hch> - return dat_dr_insert(provider_info, provider->ia_open_func); hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); hch> + if (dat_dictionary_search(&dat_provider_list, info)) { hch> + status = DAT_ERROR(DAT_PROVIDER_ALREADY_REGISTERED, 0); hch> + goto out_unlock; hch> + } hch> + hch> + strncpy(entry->info.ia_name, info->ia_name, DAT_NAME_MAX_LENGTH); hch> + hch> + /* insert node at end of list to preserve registration order */ hch> + list_add_tail(&entry->list, &dat_provider_list); hch> + dat_provider_list_size++; hch> + hch> + out_unlock: hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); hch> + out: hch> + if (status != DAT_SUCCESS) hch> + kfree(entry); hch> + return status; hch> } hch> +EXPORT_SYMBOL(dat_registry_add_provider); hch> hch> -EXPORT_SYMBOL(dat_registry_remove_provider); hch> - hch> u32 dat_registry_remove_provider(const struct dat_provider *provider, hch> - const struct dat_provider_info *provider_info) hch> + const struct dat_provider_info *info) hch> { hch> - dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, "%s(ia:%s)\n", __func__, hch> - provider_info->ia_name); hch> + struct dat_dictionary_entry *cur, *next; hch> + unsigned long flags; hch> + u32 status = DAT_SUCCESS; hch> hch> - if (DAT_IS_BAD_POINTER(provider)) hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); hch> + list_for_each_entry_safe(cur, next, &dat_provider_list, list) { hch> + if (dat_dictionary_key_is_equal(&cur->info, info)) { hch> + if (cur->ref_count) { hch> + status = DAT_ERROR(DAT_PROVIDER_IN_USE, 0); hch> + goto out; hch> + } hch> hch> - return dat_dr_remove(provider_info); hch> + list_del(&cur->list); hch> + dat_provider_list_size--; hch> + hch> + kfree(cur); hch> + goto out; hch> + } hch> + } hch> + hch> + status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); hch> + out: hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); hch> + return status; hch> } hch> +EXPORT_SYMBOL(dat_registry_remove_provider); hch> hch> -EXPORT_SYMBOL(dat_registry_add_provider); hch> - hch> u32 dat_registry_list_providers(int max_to_return, int *entries_returned, hch> - struct dat_provider_info *(dat_provider_list[])) hch> + struct dat_provider_info *provider_list[]) hch> { hch> + struct dat_dictionary_entry *cur; hch> + unsigned long flags; hch> + int i = 0; hch> u32 status = DAT_SUCCESS; hch> hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, hch> - "%s(max_to_return:%i, entries_rtrn:%p, prvdr_list:%p)\n", hch> - __func__, max_to_return, entries_returned, hch> - dat_provider_list); hch> - hch> - if ((DAT_IS_BAD_POINTER(entries_returned))) hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); hch> - hch> - if (0 != max_to_return && (DAT_IS_BAD_POINTER(dat_provider_list))) hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); hch> - hch> - if (0 == max_to_return) { hch> + if (!max_to_return) { hch> /* the user is allowed to call with max_to_return set to zero. hch> * in which case we simply return (in *entries_returned) the hch> * number of providers currently installed. We must also hch> * (per spec) return an error hch> */ hch> - (void)dat_dr_size(entries_returned); hch> + *entries_returned = dat_provider_list_size; hch> return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); hch> - } else hch> - status = dat_dr_list(max_to_return, entries_returned, hch> - dat_provider_list); hch> + } hch> + hch> + *entries_returned = 0; hch> + hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); hch> + if (!dat_provider_list_size) hch> + goto out_unlock; hch> + hch> + list_for_each_entry(cur, &dat_provider_list, list) { hch> + if (i >= max_to_return) hch> + break; hch> + if (!provider_list[i]) { hch> + status = hch> + DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); hch> + goto out_unlock; hch> + } hch> + provider_list[i++] = &cur->info; hch> + } hch> + *entries_returned = i; hch> + hch> + out_unlock: hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); hch> return status; hch> } hch> +EXPORT_SYMBOL(dat_registry_list_providers); hch> hch> -EXPORT_SYMBOL(dat_registry_list_providers); hch> +static int __init dat_init(void) hch> +{ hch> + dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry started\n"); hch> + return 0; hch> +} hch> + hch> +static void __exit dat_fini(void) hch> +{ hch> + struct dat_dictionary_entry *cur, *next; hch> + hch> + list_for_each_entry_safe(cur, next, &dat_provider_list, list) { hch> + list_del(&cur->list); hch> + kfree(cur); hch> + } hch> +} hch> + hch> +MODULE_LICENSE("Dual BSD/GPL"); hch> +MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); hch> +MODULE_AUTHOR("James Lentini"); hch> + hch> +module_init(dat_init); hch> +module_exit(dat_fini); hch> Index: linux-kernel/dat/dictionary.h hch> =================================================================== hch> --- linux-kernel/dat/dictionary.h (revision 2647) hch> +++ linux-kernel/dat/dictionary.h (working copy) hch> @@ -1,98 +0,0 @@ hch> -/* hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. hch> - * hch> - * This Software is licensed under one of the following licenses: hch> - * hch> - * 1) under the terms of the "Common Public License 1.0" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/cpl.php. hch> - * hch> - * 2) under the terms of the "The BSD License" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/bsd-license.php. hch> - * hch> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a hch> - * copy of which is available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/gpl-license.php. hch> - * hch> - * Licensee has the right to choose one of the above licenses. hch> - * hch> - * Redistributions of source code must retain the above copyright hch> - * notice and one of the license notices. hch> - * hch> - * Redistributions in binary form must reproduce both the above copyright hch> - * notice, one of the license notices in the documentation hch> - * and/or other materials provided with the distribution. hch> - */ hch> - hch> -/* hch> - * dictionary data structure hch> - * hch> - * $Id$ hch> - */ hch> - hch> -#ifndef DICTIONARY_H hch> -#define DICTIONARY_H hch> - hch> -#include "core.h" hch> - hch> -/********************************************************************* hch> - * * hch> - * Structures * hch> - * * hch> - *********************************************************************/ hch> - hch> -struct dat_dictionary_entry { hch> - struct list_head list; hch> - struct dat_provider_info key; hch> - void *data; hch> -}; hch> - hch> -struct dat_dictionary { hch> - struct list_head list; hch> - int size; hch> -}; hch> - hch> -/********************************************************************* hch> - * * hch> - * Macros * hch> - * * hch> - *********************************************************************/ hch> - hch> -#define DAT_DICTIONARY_INIT(name) { LIST_HEAD_INIT((name).list), 0 } hch> - hch> -/********************************************************************* hch> - * * hch> - * Function Prototypes * hch> - * * hch> - *********************************************************************/ hch> - hch> -extern void dat_dictionary_destroy(struct dat_dictionary *dictionary); hch> - hch> -extern u32 dat_dictionary_size(const struct dat_dictionary *dictionary, hch> - int *size); hch> - hch> -extern u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry); hch> - hch> -extern void dat_dictionary_entry_destroy( hch> - const struct dat_dictionary_entry *entry); hch> - hch> -extern u32 dat_dictionary_insert(struct dat_dictionary *dictionary, hch> - const struct dat_provider_info *key, hch> - struct dat_dictionary_entry *entry, hch> - void *data); hch> - hch> -extern u32 dat_dictionary_search(struct dat_dictionary *dictionary, hch> - const struct dat_provider_info *key, hch> - void **p_data); hch> - hch> -extern u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, hch> - void *array[], const int array_size); hch> - hch> -extern u32 dat_dictionary_remove(struct dat_dictionary *dictionary, hch> - const struct dat_provider_info *key, hch> - struct dat_dictionary_entry **p_entry, hch> - void **p_data); hch> - hch> -#endif /* DICTIONARY_H */ hch> Index: linux-kernel/dat/dr.h hch> =================================================================== hch> --- linux-kernel/dat/dr.h (revision 2647) hch> +++ linux-kernel/dat/dr.h (working copy) hch> @@ -1,57 +0,0 @@ hch> -/* hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. hch> - * hch> - * This Software is licensed under one of the following licenses: hch> - * hch> - * 1) under the terms of the "Common Public License 1.0" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/cpl.php. hch> - * hch> - * 2) under the terms of the "The BSD License" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/bsd-license.php. hch> - * hch> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a hch> - * copy of which is available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/gpl-license.php. hch> - * hch> - * Licensee has the right to choose one of the above licenses. hch> - * hch> - * Redistributions of source code must retain the above copyright hch> - * notice and one of the license notices. hch> - * hch> - * Redistributions in binary form must reproduce both the above copyright hch> - * notice, one of the license notices in the documentation hch> - * and/or other materials provided with the distribution. hch> - */ hch> - hch> -/* hch> - * Dynamic Registry (DR) functions and data stucture hch> - * hch> - * $Id$ hch> - */ hch> - hch> -#ifndef DR_H hch> -#define DR_H hch> - hch> -#include hch> - hch> -extern void dat_dr_fini(void); hch> - hch> -extern u32 dat_dr_insert(const struct dat_provider_info *info, hch> - const DAT_IA_OPEN_FUNC ia_open_func); hch> - hch> -extern u32 dat_dr_remove(const struct dat_provider_info *info); hch> - hch> -extern u32 dat_dr_provider_open(const struct dat_provider_info *info, hch> - DAT_IA_OPEN_FUNC *p_ia_open_func); hch> - hch> -extern u32 dat_dr_provider_close(const struct dat_provider_info *info); hch> - hch> -extern u32 dat_dr_size(int *size); hch> - hch> -extern u32 dat_dr_list(int max_to_return, int *entries_returned, hch> - struct dat_provider_info *(dat_provider_list[])); hch> - hch> -#endif /* DR_H */ hch> Index: linux-kernel/dat/Makefile hch> =================================================================== hch> --- linux-kernel/dat/Makefile (revision 2647) hch> +++ linux-kernel/dat/Makefile (working copy) hch> @@ -5,8 +5,4 @@ hch> hch> obj-$(CONFIG_DAT) += dat.o hch> hch> -dat-y := \ hch> - api.o \ hch> - core.o \ hch> - dictionary.o \ hch> - dr.o hch> +dat-y += api.o hch> Index: linux-kernel/dat/core.h hch> =================================================================== hch> --- linux-kernel/dat/core.h (revision 2647) hch> +++ linux-kernel/dat/core.h (working copy) hch> @@ -1,61 +0,0 @@ hch> -/* hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. hch> - * hch> - * This Software is licensed under one of the following licenses: hch> - * hch> - * 1) under the terms of the "Common Public License 1.0" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/cpl.php. hch> - * hch> - * 2) under the terms of the "The BSD License" a copy of which is hch> - * available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/bsd-license.php. hch> - * hch> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a hch> - * copy of which is available from the Open Source Initiative, see hch> - * http://www.opensource.org/licenses/gpl-license.php. hch> - * hch> - * Licensee has the right to choose one of the above licenses. hch> - * hch> - * Redistributions of source code must retain the above copyright hch> - * notice and one of the license notices. hch> - * hch> - * Redistributions in binary form must reproduce both the above copyright hch> - * notice, one of the license notices in the documentation hch> - * and/or other materials provided with the distribution. hch> - */ hch> - hch> -/* hch> - * core DAT registry functions and data structures hch> - * hch> - * $Id$ hch> - */ hch> - hch> -#ifndef CORE_H hch> -#define CORE_H hch> - hch> -#include hch> -#include hch> -#include hch> -#include hch> -#include hch> - hch> -#include hch> - hch> -#define DAT_IS_BAD_POINTER(p) ( NULL == (p) ) hch> - hch> -typedef enum { hch> - DAT_DBG_TYPE_ERROR = (1 << 0), hch> - DAT_DBG_TYPE_GENERIC = (1 << 1), hch> - DAT_DBG_TYPE_DR = (1 << 2), hch> - DAT_DBG_TYPE_PROVIDER_API = (1 << 3), hch> - DAT_DBG_TYPE_CONSUMER_API = (1 << 4), hch> - DAT_DBG_TYPE_ALL = 0x1f hch> -} DAT_DBG_TYPE; hch> - hch> -typedef int DAT_DBG_MASK; hch> - hch> -extern void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...); hch> - hch> -#endif /* CORE_H */ hch> From roland at topspin.com Wed Jun 22 14:42:41 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 14:42:41 -0700 Subject: [openib-general] User Verbs ibv_ud_pingpong Problem In-Reply-To: <42B9CA39.6040205@lanl.gov> (Alfred Torrez's message of "Wed, 22 Jun 2005 14:29:45 -0600") References: <42B9CA39.6040205@lanl.gov> Message-ID: <5264w6qd9q.fsf@topspin.com> Alfred> This is 2.6.11.5 kernel, svn version 2631 running on HP Alfred> X86_64 system, 4X HCA (MT23108) Please update to the latest svn. I believe this was fixed in r2642. - R. From roland at topspin.com Wed Jun 22 14:43:04 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 14:43:04 -0700 Subject: [openib-general] http://www.openib.org/doc.html update In-Reply-To: <1119473069.4480.167.camel@hal.voltaire.com> (Hal Rosenstock's message of "22 Jun 2005 16:44:53 -0400") References: <1118774337.22484.16.camel@duffman> <1118775221.24424.21.camel@localhost> <1118955851.21846.21.camel@duffman> <1119473069.4480.167.camel@hal.voltaire.com> Message-ID: <521x6uqd93.fsf@topspin.com> Hal> This is a great start. I have a few nits, etc.: Go ahead and edit the page... - R. From roland at topspin.com Wed Jun 22 14:43:53 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 14:43:53 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050622201706.GA2488@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 22 Jun 2005 23:17:06 +0300") References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> Message-ID: <52wtomoyna.fsf@topspin.com> Michael> I'd like to add to libibverbs a function returning the Michael> path used to find plugins, and a utility calling that Michael> function and printing it out. Its easy to do - Roland, Michael> OK with you? I guess so. Explain to me again what problem this is solving? - R. From jlentini at netapp.com Wed Jun 22 15:19:15 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 22 Jun 2005 18:19:15 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove dat_os_panic() In-Reply-To: <1119460629.26537.21.camel@duffman> References: <1119460629.26537.21.camel@duffman> Message-ID: Committed in revision 2683. On Wed, 22 Jun 2005, Tom Duffy wrote: tduffy> This patch removes the function dapl_os_panic() in favor of calling tduffy> panic() directly. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.c (revision 2677) tduffy> +++ linux-kernel/dat-provider/dapl_evd.c (working copy) tduffy> @@ -681,7 +681,7 @@ static void dapl_evd_cq_async_error_call tduffy> cause, context); tduffy> tduffy> if (!evd) tduffy> - dapl_os_panic("NULL == context\n"); tduffy> + panic("NULL == context\n"); tduffy> tduffy> async_evd = evd->common.owner_ia->async_error_evd; tduffy> tduffy> @@ -689,9 +689,8 @@ static void dapl_evd_cq_async_error_call tduffy> DAT_ASYNC_ERROR_EVD_OVERFLOW, tduffy> async_evd->common.owner_ia); tduffy> tduffy> - if (status != DAT_SUCCESS) { tduffy> - dapl_os_panic("async EVD overflow\n"); tduffy> - } tduffy> + if (status != DAT_SUCCESS) tduffy> + panic("async EVD overflow\n"); tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, tduffy> "dapl_evd_cq_async_error_callback () returns\n"); tduffy> @@ -707,10 +706,8 @@ void dapl_evd_un_async_error_callback(st tduffy> "dapl_evd_un_async_error_callback (%p, %p)\n", tduffy> cause, context); tduffy> tduffy> - if (NULL == context) { tduffy> - dapl_os_panic("NULL == context\n"); tduffy> - return; tduffy> - } tduffy> + if (NULL == context) tduffy> + panic("NULL == context\n"); tduffy> tduffy> async_evd = (struct dapl_evd *)context; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.h (revision 2677) tduffy> +++ linux-kernel/dat-provider/dapl_util.h (working copy) tduffy> @@ -48,12 +48,6 @@ tduffy> #include tduffy> #endif tduffy> tduffy> -#define dapl_os_panic(fmt, args...) \ tduffy> - do { \ tduffy> - printk("PANIC in %s:%i:%s\n", __FILE__, __LINE__, __func__); \ tduffy> - panic(fmt, ## args); \ tduffy> - } while(0) tduffy> - tduffy> #define dapl_os_assert(expression) \ tduffy> do { \ tduffy> if (!(expression)) { \ tduffy> From jjengla at sandia.gov Wed Jun 22 15:51:33 2005 From: jjengla at sandia.gov (Josh England) Date: Wed, 22 Jun 2005 15:51:33 -0700 Subject: [openib-general] RDMA write with immediate Message-ID: <1119480693.24922.33.camel@localhost> Hi, I'm trying to figure out how to do an RDMA write with immediate data. I've got a regular RDMA write working and tried changing wr.opcode from IBV_WR_RDMA_WRITE to IBV_WR_RDMA_WRITE_WITH_IMM and stuffing a value in wr.imm_data to no avail. The send fails. Does anyone have an example of how to do this, or just a list of things that need to be set up differently from a standard RDMA write operation? Thanks in advance, -JE From roland at topspin.com Wed Jun 22 19:02:51 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 22 Jun 2005 19:02:51 -0700 Subject: [openib-general] RDMA write with immediate In-Reply-To: <1119480693.24922.33.camel@localhost> (Josh England's message of "Wed, 22 Jun 2005 15:51:33 -0700") References: <1119480693.24922.33.camel@localhost> Message-ID: <52k6klq184.fsf@topspin.com> Josh> I'm trying to figure out how to do an RDMA write with Josh> immediate data. I've got a regular RDMA write working and Josh> tried changing wr.opcode from IBV_WR_RDMA_WRITE to Josh> IBV_WR_RDMA_WRITE_WITH_IMM and stuffing a value in Josh> wr.imm_data to no avail. The send fails. This should work, and in fact I tested it a while ago, although it may have broken in the meantime. When you say that your send fails, what do you mean? Does ibv_post_send() return immediately with an error? Or do you get a completion for your send request with a failed status? If so, what is the status? Or does the send just never complete and/or never show up on the remote system? - R. From kjreilly at us.ibm.com Wed Jun 22 20:16:49 2005 From: kjreilly at us.ibm.com (Kevin Reilly) Date: Wed, 22 Jun 2005 23:16:49 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <1119358993.4476.2371.camel@localhost.localdomain> Message-ID: Hal and others, thanks for your responses. This kernel IBAT looks like it provides the function to map an IP address of an network interface to the device name and port number. My reluctance is i don't think is accessible for the user space library. We would need to have a kernel model to help us do the translation. Is there any plans to make it accessible to from user space? I would expect that mapped in both directions from the device name used to open a device to IP address will be very useful for many reason like debug. Kevin J. Reilly STSM, HPC Architecture -Federation/HPS Chief Engineer -HPC interconnect architect (office) 845-433-7976 (tieline) 8-293-7976 Hal Rosenstock To Kevin Reilly/Poughkeepsie/IBM at IBMUS 06/21/2005 09:05 cc AM openib-general at openib.org Subject Re: [openib-general] mapping between IP address and device name Hi Kevin, On Mon, 2005-06-20 at 22:23, Kevin Reilly wrote: > Maybe somebody could help me understand the proper way to map between an IP > address assigned to a port to the > "device name" and "port number" in the gen2 architecture. If I have an IP > address can I map it to a name that i get back > from ibv_get_device_name() or pass to ibv_open_device(). > Here is the problem. Let's say there is a subsystem that used > heartbeating over IP interfaces to ascertain the adapters > and ports state, up or down. The data actively maintained be this > heartbeating subsystem is used by a job scheduler that > desires to only schedule tasks on HCAs that are UP and have active ports. > The job scheduler does not want to launch jobs > on adapters that are broken, not connected to a switch or not configured. > The job scheduler needs a way to map an IP address into a name and > port number sitable to be passed to the job to > use when ibv_open_device() is called or to match the output of > ibv_get_device_name(). With kernel IB Address Translation (IBAT), you can resolve an IP address (remote or local) and obtain the device and port are obtained from the ib_route structure. ib_at_route_by_ip: * ib_at_route_by_ip - asynchronously resolve ip route to ib route * @dst_ip: destination ip * @src_ip: source ip - optional * @tos: ip type of service * @flags: ib_at_route_flags * @ib_route: out structure * @async_comp: asynchronous callback structure - optional * * Resolve the specified dst_ip to a &struct ib_route structure. * src_ip can be provided to force specific output interface. * flags can be used to select resolving method; currently IB-ARP or ATS. and then: * ib_at_paths_by_route - asynchronously resolve ib route to ib path records * @ib_route: ib route to resolve * @mpath_type: ib_at_multipathing_type * @path_arr: SA path record array - out * @npath: maximal number of paths to return * @async_comp: asynchronous callback structure - optional * * Resolve the specified ib_route to a SA path record array. * Number of returned paths will not exceed npath. * Multipathing type may be used to obtain redundant paths for APM, * other failover schemes, bandwidth aggregation or source based routing. * Note that multipathing request is meaningless unless npath is greater than 1. * * Returned ib_route structure includes the recommended pkey and qos_tag for * this route. IBAT is not currently in the OpenIB trunk. See users/jlentini/linux-kernel/patches/ib_at.h or branches/shaharf-ibat/src/linux-kernel/infiniband/include/ib_at.h for more details. -- Hal > Kevin J. Reilly > STSM, HPC Architecture > -Federation/HPS Chief Engineer > -HPC interconnect architect > (office) 845-433-7976 (tieline) 8-293-7976 > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From medivisor1 at optonline.net Wed Jun 22 22:00:00 2005 From: medivisor1 at optonline.net (Medivisor) Date: Thu, 23 Jun 2005 01:00:00 -0400 Subject: [openib-general] Physician's Member Update Request Message-ID: <386-2200564235000@David> David Kary NormalDavid Kary 4132005-02-08T19:42:00Z2005-02-25T16:27:00Z1164938Medivisor72110011.5606 CleanCleanfalsefalsefalseMicrosoftInternetExplorer4 AS we progress into the New Year we have many exciting new plans, programs and services at www.Medivisor.com   and www.StatDose.com . As a member physician you will be among the first to experience unique online clinical tools, exciting newsletters, new products and the launch of our new medical news radio initiative.  We are very excited about our upcoming offerings for this year and hope you will join us as we start a new era in medical communication technology. In order to continue your participation with us we ask that you take a moment to update your current information below and return to us via return email.  Simply hit the reply button, fill in the fields below and hit send.   Thank you for your time, we are sure you will appreciate the upcoming medical communication services. Your email address: Your name: Your specialty: Your state: You have received this email as a member physician. If you wish to be removed from our list please CLICK HERE -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 78743 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 3345 bytes Desc: image003.jpg URL: From mst at mellanox.co.il Wed Jun 22 22:58:35 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 23 Jun 2005 08:58:35 +0300 Subject: [openib-general] Re: Re: [PATCH] rdma_lat-09 and results In-Reply-To: <52wtomoyna.fsf@topspin.com> References: <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> Message-ID: <20050623055835.GA10456@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: Re: [PATCH] rdma_lat-09 and results > > Michael> I'd like to add to libibverbs a function returning the > Michael> path used to find plugins, and a utility calling that > Michael> function and printing it out. Its easy to do - Roland, > Michael> OK with you? > > I guess so. Explain to me again what problem this is solving? > > - R. > Currently I want to be able to script linking mthca.a statically. In the future I can imagine someone might want to script installing additional plugins into an existing system. -- MST From iod00d at hp.com Wed Jun 22 23:20:20 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 22 Jun 2005 23:20:20 -0700 Subject: [openib-general] [PATCH] kDAPL: remove dat_os_panic() In-Reply-To: <1119460629.26537.21.camel@duffman> References: <1119460629.26537.21.camel@duffman> Message-ID: <20050623062020.GB21363@esmail.cup.hp.com> On Wed, Jun 22, 2005 at 10:17:09AM -0700, Tom Duffy wrote: > This patch removes the function dapl_os_panic() in favor of calling > panic() directly. just some nits... > if (!evd) > - dapl_os_panic("NULL == context\n"); > + panic("NULL == context\n"); Isn't this a bit silly? > async_evd = evd->common.owner_ia->async_error_evd; The system is going to panic anyway here if evd is null. I don't see any advantage to testing evd if the only action is to panic. > - if (NULL == context) { > - dapl_os_panic("NULL == context\n"); > - return; > - } > + if (NULL == context) > + panic("NULL == context\n"); Would this be better as "BUG_ON(NULL == context)"? thanks, grant From iod00d at hp.com Wed Jun 22 23:42:08 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 22 Jun 2005 23:42:08 -0700 Subject: [openib-general] Re: page allocation failure with kdapl/mthca In-Reply-To: <52d5qeqksl.fsf@topspin.com> References: <1119366787.4477.26.camel@localhost.localdomain> <52d5qfwtrz.fsf@topspin.com> <52d5qeqksl.fsf@topspin.com> Message-ID: <20050623064208.GE21363@esmail.cup.hp.com> On Wed, Jun 22, 2005 at 12:00:10PM -0700, Roland Dreier wrote: > Roland> I just checked in a change that converts from using > Roland> pci_alloc_consistent(), which is always GFP_ATOMIC, to > Roland> dma_alloc_coherent(..., GFP_KERNEL). This arch specific. On ia64 I expect both functions to land in the same bit of code. For ZX1 platforms, that is sba_alloc_coherent(). See arch/ia64/hp/common/sba_iommu.c since the behavior is different depending on CONFIG_NUMA setting. Sorry, I missed/forgot what the original problem was. grant From halr at voltaire.com Thu Jun 23 03:45:21 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 06:45:21 -0400 Subject: [openib-general] Re: page allocation failure with kdapl/mthca In-Reply-To: <20050623064208.GE21363@esmail.cup.hp.com> References: <1119366787.4477.26.camel@localhost.localdomain> <52d5qfwtrz.fsf@topspin.com> <52d5qeqksl.fsf@topspin.com> <20050623064208.GE21363@esmail.cup.hp.com> Message-ID: <1119523521.4480.232.camel@hal.voltaire.com> On Thu, 2005-06-23 at 02:42, Grant Grundler wrote: > On Wed, Jun 22, 2005 at 12:00:10PM -0700, Roland Dreier wrote: > > Roland> I just checked in a change that converts from using > > Roland> pci_alloc_consistent(), which is always GFP_ATOMIC, to > > Roland> dma_alloc_coherent(..., GFP_KERNEL). > > This arch specific. > > On ia64 I expect both functions to land in the same bit of code. > For ZX1 platforms, that is sba_alloc_coherent(). > See arch/ia64/hp/common/sba_iommu.c since the behavior is > different depending on CONFIG_NUMA setting. > > Sorry, I missed/forgot what the original problem was. The original problem (for me) was that there was a page allocation failure with kdapl/mthca on a create QP (and the driver changed but kdapl didn't in terms of this). I'm not sure which change introduced the problem but it was fixed with r2666. -- Hal From halr at voltaire.com Thu Jun 23 04:12:12 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 07:12:12 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: References: Message-ID: <1119524416.4480.250.camel@hal.voltaire.com> On Wed, 2005-06-22 at 23:16, Kevin Reilly wrote: > > Hal and others, thanks for your responses. > > This kernel IBAT looks like it provides the function to map an IP address > of an network interface to the device name and port > number. Yes. > My reluctance is i don't think is accessible for the user space > library. > We would need to have a kernel model to help > us do the translation. Is there any plans to make it accessible to from > user space? Yes. There is a user space implementation (along with an updated kernel implementation in the shaharf-ibat branch. -- Hal > I would expect that mapped in both directions from the device name used to > open a device to IP address will be very useful for > many reason like debug. > > Kevin J. Reilly > STSM, HPC Architecture > -Federation/HPS Chief Engineer > -HPC interconnect architect > (office) 845-433-7976 (tieline) 8-293-7976 > > > > > Hal Rosenstock > m> To > Kevin Reilly/Poughkeepsie/IBM at IBMUS > 06/21/2005 09:05 cc > AM openib-general at openib.org > Subject > Re: [openib-general] mapping > between IP address and device name > > > > > > > > > > > Hi Kevin, > > On Mon, 2005-06-20 at 22:23, Kevin Reilly wrote: > > Maybe somebody could help me understand the proper way to map between an > IP > > address assigned to a port to the > > "device name" and "port number" in the gen2 architecture. If I have an > IP > > address can I map it to a name that i get back > > from ibv_get_device_name() or pass to ibv_open_device(). > > Here is the problem. Let's say there is a subsystem that > used > > heartbeating over IP interfaces to ascertain the adapters > > and ports state, up or down. The data actively maintained be > this > > heartbeating subsystem is used by a job scheduler that > > desires to only schedule tasks on HCAs that are UP and have active > ports. > > The job scheduler does not want to launch jobs > > on adapters that are broken, not connected to a switch or not configured. > > The job scheduler needs a way to map an IP address into a name > and > > port number sitable to be passed to the job to > > use when ibv_open_device() is called or to match the output > of > > ibv_get_device_name(). > > With kernel IB Address Translation (IBAT), you can resolve an IP address > (remote or local) and obtain the device and port are obtained from the > ib_route structure. > > ib_at_route_by_ip: > * ib_at_route_by_ip - asynchronously resolve ip route to ib route > * @dst_ip: destination ip > * @src_ip: source ip - optional > * @tos: ip type of service > * @flags: ib_at_route_flags > * @ib_route: out structure > * @async_comp: asynchronous callback structure - optional > * > * Resolve the specified dst_ip to a &struct ib_route structure. > * src_ip can be provided to force specific output interface. > * flags can be used to select resolving method; currently IB-ARP or ATS. > and then: > * ib_at_paths_by_route - asynchronously resolve ib route to ib path > records > * @ib_route: ib route to resolve > * @mpath_type: ib_at_multipathing_type > * @path_arr: SA path record array - out > * @npath: maximal number of paths to return > * @async_comp: asynchronous callback structure - optional > * > * Resolve the specified ib_route to a SA path record array. > * Number of returned paths will not exceed npath. > * Multipathing type may be used to obtain redundant paths for APM, > * other failover schemes, bandwidth aggregation or source based routing. > * Note that multipathing request is meaningless unless npath is greater > than 1. > * > * Returned ib_route structure includes the recommended pkey and qos_tag > for > * this route. > > IBAT is not currently in the OpenIB trunk. See > users/jlentini/linux-kernel/patches/ib_at.h or > branches/shaharf-ibat/src/linux-kernel/infiniband/include/ib_at.h for > more details. > > -- Hal > > > Kevin J. Reilly > > STSM, HPC Architecture > > -Federation/HPS Chief Engineer > > -HPC interconnect architect > > (office) 845-433-7976 (tieline) 8-293-7976 > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > From halr at voltaire.com Thu Jun 23 04:15:25 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 07:15:25 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: References: Message-ID: <1119524960.4480.260.camel@hal.voltaire.com> Hi again Kevin, I'd like to go back to your original query to make sure I understand things. On Mon, 2005-06-20 at 22:23, Kevin Reilly wrote: > Maybe somebody could help me understand the proper way to map between an IP > address assigned to a port to the > "device name" and "port number" in the gen2 architecture. If I have an IP > address can I map it to a name that i get back > from ibv_get_device_name() or pass to ibv_open_device(). So the application knows the IP addresses of the local IP interfaces and (potentially) the device names and ports but not their correspondence. It is the correspondence that is needed, Is that correct ? Are the local IP interfaces limited to IPoIB or are there alternate IP network interfaces as well ? Thanks. -- Hal > Here is the problem. Let's say there is a subsystem that used > heartbeating over IP interfaces to ascertain the adapters > and ports state, up or down. The data actively maintained be this > heartbeating subsystem is used by a job scheduler that > desires to only schedule tasks on HCAs that are UP and have active ports. > The job scheduler does not want to launch jobs > on adapters that are broken, not connected to a switch or not configured. > The job scheduler needs a way to map an IP address into a name and > port number sitable to be passed to the job to > use when ibv_open_device() is called or to match the output of > ibv_get_device_name(). > Kevin J. Reilly > STSM, HPC Architecture > -Federation/HPS Chief Engineer > -HPC interconnect architect > (office) 845-433-7976 (tieline) 8-293-7976 > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jjengla at sandia.gov Thu Jun 23 04:36:46 2005 From: jjengla at sandia.gov (Josh England) Date: Thu, 23 Jun 2005 04:36:46 -0700 Subject: [openib-general] RDMA write with immediate In-Reply-To: <52k6klq184.fsf@topspin.com> References: <1119480693.24922.33.camel@localhost> <52k6klq184.fsf@topspin.com> Message-ID: <1119526607.29410.6.camel@localhost> On Wed, 2005-06-22 at 19:02 -0700, Roland Dreier wrote: > Josh> I'm trying to figure out how to do an RDMA write with > Josh> immediate data. I've got a regular RDMA write working and > Josh> tried changing wr.opcode from IBV_WR_RDMA_WRITE to > Josh> IBV_WR_RDMA_WRITE_WITH_IMM and stuffing a value in > Josh> wr.imm_data to no avail. The send fails. > > This should work, and in fact I tested it a while ago, although it may > have broken in the meantime. > > When you say that your send fails, what do you mean? Does > ibv_post_send() return immediately with an error? Or do you get a > completion for your send request with a failed status? If so, what is > the status? I get a completion with failed status 13. Also, it prints out some junk just before the failed completion: [ 0] 007f0406 [ 4] 00cf36ed [ 8] 001da414 [ c] 00003800 [10] 16870000 [14] 00000000 [18] 00000108 [1c] ff100000 Again, this works just fine with IBV_WR_RDMA_WRITE. I'm using 2.6.11.11 with SVN rev 2606. I'll update stuff and try again. -JE > Or does the send just never complete and/or never show up > on the remote system? > > - R. > > From jjengla at sandia.gov Thu Jun 23 05:47:57 2005 From: jjengla at sandia.gov (Josh England) Date: Thu, 23 Jun 2005 05:47:57 -0700 Subject: [openib-general] RDMA write with immediate In-Reply-To: <52k6klq184.fsf@topspin.com> References: <1119480693.24922.33.camel@localhost> <52k6klq184.fsf@topspin.com> Message-ID: <1119530877.29406.29.camel@localhost> Alright, I'm probably being retarded, but just in case: uverbs doesn't seem to work anymore. Simple pingpong uverbs test gives: Couldn't get context for mthca0 This is with rev 2685 on 2.6.12.1 under RHEL3. Fresh rebuild/install of libibverbs, libmthca, and libibcm. Ports are ACTIVE and ipoib ping works. # lsmod ib_ucm 15448 0 ib_cm 33904 1 ib_ucm ib_at 13704 0 ib_uverbs 24088 0 ib_ipoib 38920 0 ib_sa 11660 2 ib_at,ib_ipoib ib_mthca 90768 0 ib_mad 38056 3 ib_cm,ib_sa,ib_mthca ib_core 41088 6 ib_cm,ib_uverbs,ib_ipoib,ib_sa,ib_mthca,ib_mad # ll /dev/infiniband crw-r--r-- 1 root root 231, 255 Jun 23 05:31 ucm crw-rw-rw- 1 root root 231, 128 Jun 23 05:31 uverbs0 crw-rw-rw- 1 root root 231, 129 Jun 23 05:31 uverbs1 Did I miss something? I think this is the same environment that was working before. -JE On Wed, 2005-06-22 at 19:02 -0700, Roland Dreier wrote: > Josh> I'm trying to figure out how to do an RDMA write with > Josh> immediate data. I've got a regular RDMA write working and > Josh> tried changing wr.opcode from IBV_WR_RDMA_WRITE to > Josh> IBV_WR_RDMA_WRITE_WITH_IMM and stuffing a value in > Josh> wr.imm_data to no avail. The send fails. > > This should work, and in fact I tested it a while ago, although it may > have broken in the meantime. > > When you say that your send fails, what do you mean? Does > ibv_post_send() return immediately with an error? Or do you get a > completion for your send request with a failed status? If so, what is > the status? Or does the send just never complete and/or never show up > on the remote system? > > - R. > > From halr at voltaire.com Thu Jun 23 05:56:42 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 08:56:42 -0400 Subject: [openib-general] RDMA write with immediate In-Reply-To: <1119530877.29406.29.camel@localhost> References: <1119480693.24922.33.camel@localhost> <52k6klq184.fsf@topspin.com> <1119530877.29406.29.camel@localhost> Message-ID: <1119531169.4481.15.camel@localhost.localdomain> On Thu, 2005-06-23 at 08:47, Josh England wrote: > Alright, I'm probably being retarded, but just in case: > uverbs doesn't seem to work anymore. Simple pingpong uverbs test gives: > > Couldn't get context for mthca0 > > This is with rev 2685 on 2.6.12.1 under RHEL3. Fresh rebuild/install of > libibverbs, libmthca, and libibcm. Ports are ACTIVE and ipoib ping > works. > > # lsmod > ib_ucm 15448 0 > ib_cm 33904 1 ib_ucm > ib_at 13704 0 > ib_uverbs 24088 0 > ib_ipoib 38920 0 > ib_sa 11660 2 ib_at,ib_ipoib > ib_mthca 90768 0 > ib_mad 38056 3 ib_cm,ib_sa,ib_mthca > ib_core 41088 6 > ib_cm,ib_uverbs,ib_ipoib,ib_sa,ib_mthca,ib_mad > > # ll /dev/infiniband > crw-r--r-- 1 root root 231, 255 Jun 23 05:31 ucm > crw-rw-rw- 1 root root 231, 128 Jun 23 05:31 uverbs0 > crw-rw-rw- 1 root root 231, 129 Jun 23 05:31 uverbs1 > > > Did I miss something? I think this is the same environment that was > working before. Are you sure this is 2685 ? As of 2658, I think uverbs now starts at minor 192 rather than 128. 2658 Modified: gen2/trunk/src/linux-kernel/infiniband/core/uverbs_main.c Log: Fix order of cleanup in module unload. Move first minor from 128 to 192. -- Hal > > -JE > > On Wed, 2005-06-22 at 19:02 -0700, Roland Dreier wrote: > > Josh> I'm trying to figure out how to do an RDMA write with > > Josh> immediate data. I've got a regular RDMA write working and > > Josh> tried changing wr.opcode from IBV_WR_RDMA_WRITE to > > Josh> IBV_WR_RDMA_WRITE_WITH_IMM and stuffing a value in > > Josh> wr.imm_data to no avail. The send fails. > > > > This should work, and in fact I tested it a while ago, although it may > > have broken in the meantime. > > > > When you say that your send fails, what do you mean? Does > > ibv_post_send() return immediately with an error? Or do you get a > > completion for your send request with a failed status? If so, what is > > the status? Or does the send just never complete and/or never show up > > on the remote system? > > > > - R. > > > > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From itamar at mellanox.co.il Thu Jun 23 06:52:55 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Thu, 23 Jun 2005 16:52:55 +0300 Subject: [openib-general] Re: [PATCH] cleanup dat provider registratio n Message-ID: <91DB792C7985D411BEC300B40080D29CC35D07@mtvex01.mtv.mtl.com> i have a question: after we have cleaned up the dat_provider_info struct and now it is define as : struct dat_provider_info { char ia_name[DAT_NAME_MAX_LENGTH]; }; why cant we replace it with a simple char * it is only a string that the only thing we do with it is to store it and compare with others strings. Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Thursday, June 23, 2005 12:13 AM > To: Christoph Hellwig > Cc: openib-general at openib.org > Subject: [openib-general] Re: [PATCH] cleanup dat provider > registration > > > > This is an excellent simplification. Committed in revision > 2682 with a > few minor modifications: > > - kept printouts in dat_registry_add_provider, > dat_registry_remove_provider, and dat_registry_list_providers > > - updated printout in dat_ia_close (this wasn't something you > changed) > > - removed parens around sizeof > > - removed space in front of labels > > The last two are for consistency with the coding style we've been > using. If we've deviated from what is acceptable, let us know. > > Given this simplification, I can think of a few more changes: > > - rename api.c to registry.c > - remove all "dictionary" references: rename > dat_dictionary_search to > dat_provider_list_search, rename struct dat_dictionary_entry to > struct dat_provider_list_entry, rename > dat_dictionary_key_is_equal() to dat_provider_info_is_equal() > > Anyone have opinions for or against? > > On Fri, 17 Jun 2005, Christoph Hellwig wrote: > > hch> This huge patch revamps almost everything under dat, and > merges the three > hch> layers of external API, dynamic registry and dictinary > into a single > hch> one, greatly streamlining all of the code. > hch> > hch> Note that I don't have any RDMA hardware to actually > takes this, so it > hch> probably needs some testing first. > hch> > hch> > hch> Signed-off-by: Christoph Hellwig > hch> > hch> Index: linux-kernel/dat/dr.c > hch> > =================================================================== > hch> --- linux-kernel/dat/dr.c (revision 2647) > hch> +++ linux-kernel/dat/dr.c (working copy) > hch> @@ -1,242 +0,0 @@ > hch> -/* > hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All > rights reserved. > hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights > reserved. > hch> - * > hch> - * This Software is licensed under one of the following > licenses: > hch> - * > hch> - * 1) under the terms of the "Common Public License > 1.0" a copy of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/cpl.php. > hch> - * > hch> - * 2) under the terms of the "The BSD License" a copy > of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/bsd-license.php. > hch> - * > hch> - * 3) under the terms of the "GNU General Public > License (GPL) Version 2" a > hch> - * copy of which is available from the Open Source > Initiative, see > hch> - * http://www.opensource.org/licenses/gpl-license.php. > hch> - * > hch> - * Licensee has the right to choose one of the above licenses. > hch> - * > hch> - * Redistributions of source code must retain the above > copyright > hch> - * notice and one of the license notices. > hch> - * > hch> - * Redistributions in binary form must reproduce both > the above copyright > hch> - * notice, one of the license notices in the documentation > hch> - * and/or other materials provided with the distribution. > hch> - */ > hch> - > hch> -/* > hch> - * Dynamic Registry (DR) functions and data structures > hch> - * > hch> - * $Id$ > hch> - */ > hch> - > hch> -#include "core.h" > hch> -#include "dictionary.h" > hch> -#include "dr.h" > hch> - > hch> -struct dat_dr_entry { > hch> - int ref_count; > hch> - struct dat_provider_info info; > hch> - DAT_IA_OPEN_FUNC ia_open_func; > hch> -}; > hch> - > hch> -static spinlock_t g_dr_lock = SPIN_LOCK_UNLOCKED; > hch> -static struct dat_dictionary g_dr_dictionary = > hch> - DAT_DICTIONARY_INIT(g_dr_dictionary); > hch> - > hch> -void dat_dr_fini(void) > hch> -{ > hch> - dat_dictionary_destroy(&g_dr_dictionary); > hch> -} > hch> - > hch> -u32 dat_dr_insert(const struct dat_provider_info *info, > hch> - const DAT_IA_OPEN_FUNC ia_open_func) > hch> -{ > hch> - struct dat_dr_entry *data; > hch> - struct dat_dictionary_entry *dict_entry; > hch> - unsigned long flags; > hch> - u32 status; > hch> - > hch> - data = kmalloc(sizeof *data, GFP_ATOMIC); > hch> - if (!data) { > hch> - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, > hch> - DAT_RESOURCE_MEMORY); > hch> - goto bail; > hch> - } > hch> - > hch> - data->ref_count = 0; > hch> - data->info = *info; > hch> - data->ia_open_func = ia_open_func; > hch> - > hch> - dict_entry = NULL; > hch> - status = dat_dictionary_entry_create(&dict_entry); > hch> - if (DAT_SUCCESS != status) > hch> - goto bail; > hch> - > hch> - spin_lock_irqsave(&g_dr_lock, flags); > hch> - > hch> - status = > dat_dictionary_insert(&g_dr_dictionary, info, dict_entry, > hch> - data); > hch> - > hch> - spin_unlock_irqrestore(&g_dr_lock, flags); > hch> - > hch> -bail: > hch> - if (DAT_SUCCESS != status) { > hch> - if (NULL != data) > hch> - kfree(data); > hch> - > hch> - if (NULL != dict_entry) > hch> - > dat_dictionary_entry_destroy(dict_entry); > hch> - } > hch> - > hch> - return status; > hch> -} > hch> - > hch> -u32 dat_dr_remove(const struct dat_provider_info *info) > hch> -{ > hch> - struct dat_dr_entry *data; > hch> - struct dat_dictionary_entry *dict_entry; > hch> - unsigned long flags; > hch> - u32 status; > hch> - > hch> - dict_entry = NULL; > hch> - spin_lock_irqsave(&g_dr_lock, flags); > hch> - > hch> - status = > dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); > hch> - > hch> - if (DAT_SUCCESS != status) > hch> - goto bail; /* return status from > dat_dictionary_search() */ > hch> - > hch> - if (0 != data->ref_count) { > hch> - status = DAT_ERROR(DAT_PROVIDER_IN_USE, 0); > hch> - goto bail; > hch> - } > hch> - > hch> - status = > dat_dictionary_remove(&g_dr_dictionary, info, &dict_entry, > hch> - (void **) &data); > hch> - if (DAT_SUCCESS != status) > hch> - goto bail; /* return status from > dat_dictionary_remove() */ > hch> - > hch> - kfree(data); > hch> - > hch> -bail: > hch> - spin_unlock_irqrestore(&g_dr_lock, flags); > hch> - > hch> - if (NULL != dict_entry) > hch> - dat_dictionary_entry_destroy(dict_entry); > hch> - > hch> - return status; > hch> -} > hch> - > hch> -u32 dat_dr_provider_open(const struct dat_provider_info *info, > hch> - DAT_IA_OPEN_FUNC > *p_ia_open_func) > hch> -{ > hch> - struct dat_dr_entry *data; > hch> - unsigned long flags; > hch> - u32 status; > hch> - > hch> - spin_lock_irqsave(&g_dr_lock, flags); > hch> - > hch> - status = > dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); > hch> - > hch> - spin_unlock_irqrestore(&g_dr_lock, flags); > hch> - > hch> - if (DAT_SUCCESS == status) { > hch> - data->ref_count++; > hch> - *p_ia_open_func = data->ia_open_func; > hch> - } > hch> - > hch> - return status; > hch> -} > hch> - > hch> -u32 dat_dr_provider_close(const struct dat_provider_info *info) > hch> -{ > hch> - struct dat_dr_entry *data; > hch> - unsigned long flags; > hch> - u32 status; > hch> - > hch> - spin_lock_irqsave(&g_dr_lock, flags); > hch> - > hch> - status = > dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); > hch> - > hch> - spin_unlock_irqrestore(&g_dr_lock, flags); > hch> - > hch> - if (DAT_SUCCESS == status) > hch> - data->ref_count--; > hch> - > hch> - return status; > hch> -} > hch> - > hch> -u32 dat_dr_size(int *size) > hch> -{ > hch> - return dat_dictionary_size(&g_dr_dictionary, size); > hch> -} > hch> - > hch> -u32 dat_dr_list(int max_to_return, int *entries_returned, > hch> - struct dat_provider_info *(dat_provider_list[])) > hch> -{ > hch> - struct dat_dr_entry **array = NULL; > hch> - int i, array_size; > hch> - unsigned long flags; > hch> - u32 status = DAT_SUCCESS; > hch> - > hch> - /* > hch> - * The dictionary size may increase between the call to > hch> - * dat_dictionary_size() and dat_dictionary_enumerate(). > hch> - * Therefore we loop until a successful > enumeration is made. > hch> - */ > hch> - *entries_returned = 0; > hch> - for (;;) { > hch> - status = > dat_dictionary_size(&g_dr_dictionary, &array_size); > hch> - if (DAT_SUCCESS != status) > hch> - goto bail; > hch> - > hch> - if (0 == array_size) { > hch> - status = DAT_SUCCESS; > hch> - goto bail; > hch> - } > hch> - > hch> - array = kmalloc(array_size * sizeof > *array, GFP_ATOMIC); > hch> - if (!array) { > hch> - status = > hch> - > DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, > hch> - DAT_RESOURCE_MEMORY); > hch> - goto bail; > hch> - } > hch> - > hch> - spin_lock_irqsave(&g_dr_lock, flags); > hch> - > hch> - status = > dat_dictionary_enumerate(&g_dr_dictionary, > hch> - (void > **) array, array_size); > hch> - > hch> - spin_unlock_irqrestore(&g_dr_lock, flags); > hch> - > hch> - if (DAT_SUCCESS == status) > hch> - break; > hch> - else { > hch> - kfree(array); > hch> - array = NULL; > hch> - continue; > hch> - } > hch> - } > hch> - > hch> - for (i = 0; (i < max_to_return) && (i < > array_size); i++) { > hch> - if (NULL == dat_provider_list[i]) { > hch> - status = > hch> - > DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); > hch> - goto bail; > hch> - } > hch> - > hch> - *dat_provider_list[i] = array[i]->info; > hch> - } > hch> - > hch> - *entries_returned = i; > hch> - > hch> -bail: > hch> - if (NULL != array) > hch> - kfree(array); > hch> - > hch> - return status; > hch> -} > hch> Index: linux-kernel/dat/dictionary.c > hch> > =================================================================== > hch> --- linux-kernel/dat/dictionary.c (revision 2647) > hch> +++ linux-kernel/dat/dictionary.c (working copy) > hch> @@ -1,229 +0,0 @@ > hch> -/* > hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All > rights reserved. > hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights > reserved. > hch> - * > hch> - * This Software is licensed under one of the following > licenses: > hch> - * > hch> - * 1) under the terms of the "Common Public License > 1.0" a copy of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/cpl.php. > hch> - * > hch> - * 2) under the terms of the "The BSD License" a copy > of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/bsd-license.php. > hch> - * > hch> - * 3) under the terms of the "GNU General Public > License (GPL) Version 2" a > hch> - * copy of which is available from the Open Source > Initiative, see > hch> - * http://www.opensource.org/licenses/gpl-license.php. > hch> - * > hch> - * Licensee has the right to choose one of the above licenses. > hch> - * > hch> - * Redistributions of source code must retain the above > copyright > hch> - * notice and one of the license notices. > hch> - * > hch> - * Redistributions in binary form must reproduce both > the above copyright > hch> - * notice, one of the license notices in the documentation > hch> - * and/or other materials provided with the distribution. > hch> - */ > hch> - > hch> -/* > hch> - * A dictionary data structure implemented with a linked list > hch> - * > hch> - * $Id$ > hch> - */ > hch> - > hch> -#include "dictionary.h" > hch> - > hch> -static u32 dat_dictionary_key_dup(const struct > dat_provider_info *old_key, > hch> - struct > dat_provider_info *new_key); > hch> - > hch> -static boolean_t dat_dictionary_key_is_equal(const > struct dat_provider_info *a, > hch> - const > struct dat_provider_info *b); > hch> - > hch> -void dat_dictionary_destroy(struct dat_dictionary *dictionary) > hch> -{ > hch> - struct list_head *cur_list, *next_list; > hch> - struct dat_dictionary_entry *cur_entry; > hch> - > hch> - BUG_ON(NULL == dictionary); > hch> - > hch> - list_for_each_safe(cur_list, next_list, > &dictionary->list) { > hch> - cur_entry = list_entry(cur_list, struct > dat_dictionary_entry, > hch> - list); > hch> - list_del(&cur_entry->list); > hch> - kfree(cur_entry); > hch> - } > hch> -} > hch> - > hch> -u32 dat_dictionary_size(const struct dat_dictionary > *dictionary, int *size) > hch> -{ > hch> - BUG_ON(NULL == dictionary); > hch> - BUG_ON(NULL == size); > hch> - > hch> - *size = dictionary->size; > hch> - > hch> - return DAT_SUCCESS; > hch> -} > hch> - > hch> -u32 dat_dictionary_entry_create(struct > dat_dictionary_entry **p_entry) > hch> -{ > hch> - struct dat_dictionary_entry *entry; > hch> - u32 status = DAT_SUCCESS; > hch> - > hch> - BUG_ON(NULL == p_entry); > hch> - > hch> - entry = kmalloc(sizeof *entry, GFP_ATOMIC); > hch> - if (!entry) { > hch> - status = > hch> - > DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); > hch> - goto bail; > hch> - } > hch> - > hch> - *p_entry = entry; > hch> - > hch> -bail: > hch> - return status; > hch> -} > hch> - > hch> -void dat_dictionary_entry_destroy(const struct > dat_dictionary_entry *entry) > hch> -{ > hch> - kfree(entry); > hch> -} > hch> - > hch> -u32 dat_dictionary_insert(struct dat_dictionary *dictionary, > hch> - const struct dat_provider_info *key, > hch> - struct dat_dictionary_entry > *entry, void *data) > hch> -{ > hch> - u32 status; > hch> - > hch> - BUG_ON(NULL == dictionary); > hch> - BUG_ON(NULL == entry); > hch> - > hch> - if (DAT_SUCCESS == > dat_dictionary_search(dictionary, key, NULL)) { > hch> - status = > DAT_ERROR(DAT_PROVIDER_ALREADY_REGISTERED, 0); > hch> - goto bail; > hch> - } > hch> - > hch> - status = dat_dictionary_key_dup(key, &entry->key); > hch> - if (DAT_SUCCESS != status) > hch> - goto bail; > hch> - > hch> - entry->data = data; > hch> - > hch> - /* insert node at end of list to preserve > registration order */ > hch> - list_add_tail(&entry->list, &dictionary->list); > hch> - > hch> - dictionary->size++; > hch> - > hch> -bail: > hch> - return status; > hch> -} > hch> - > hch> -u32 dat_dictionary_search(struct dat_dictionary *dictionary, > hch> - const struct > dat_provider_info *key, void **p_data) > hch> -{ > hch> - struct dat_dictionary_entry *cur_entry; > hch> - u32 status; > hch> - > hch> - BUG_ON(NULL == dictionary); > hch> - BUG_ON(NULL == key); > hch> - > hch> - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, > DAT_NAME_NOT_REGISTERED); > hch> - > hch> - list_for_each_entry(cur_entry, > &dictionary->list, list) { > hch> - if (TRUE == > dat_dictionary_key_is_equal(&cur_entry->key, key)) { > hch> - if (NULL != p_data) > hch> - *p_data = cur_entry->data; > hch> - > hch> - status = DAT_SUCCESS; > hch> - goto bail; > hch> - } > hch> - } > hch> - > hch> -bail: > hch> - return status; > hch> -} > hch> - > hch> -u32 dat_dictionary_enumerate(struct dat_dictionary > *dictionary, void *array[], > hch> - int array_size) > hch> -{ > hch> - struct dat_dictionary_entry *cur_entry; > hch> - int i; > hch> - u32 status; > hch> - > hch> - BUG_ON(NULL == dictionary); > hch> - BUG_ON(NULL == array); > hch> - > hch> - status = DAT_SUCCESS; > hch> - > hch> - if (array_size < dictionary->size) { > hch> - status = > DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); > hch> - goto bail; > hch> - } > hch> - > hch> - i = 0; > hch> - list_for_each_entry(cur_entry, &dictionary->list, list) > hch> - array[i++] = cur_entry->data; > hch> - > hch> -bail: > hch> - return status; > hch> -} > hch> - > hch> -u32 dat_dictionary_remove(struct dat_dictionary *dictionary, > hch> - const struct dat_provider_info *key, > hch> - struct dat_dictionary_entry > **p_entry, void **p_data) > hch> -{ > hch> - struct list_head *cur_list, *next_list; > hch> - struct dat_dictionary_entry *cur_entry; > hch> - u32 status; > hch> - > hch> - BUG_ON(NULL == dictionary); > hch> - BUG_ON(NULL == key); > hch> - BUG_ON(NULL == p_entry); > hch> - > hch> - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, > DAT_NAME_NOT_REGISTERED); > hch> - > hch> - list_for_each_safe(cur_list, next_list, > &dictionary->list) { > hch> - cur_entry = list_entry(cur_list, struct > dat_dictionary_entry, > hch> - list); > hch> - if (TRUE == > dat_dictionary_key_is_equal(&cur_entry->key, key)) { > hch> - if (NULL != p_data) > hch> - *p_data = cur_entry->data; > hch> - > hch> - list_del(&cur_entry->list); > hch> - > hch> - *p_entry = cur_entry; > hch> - > hch> - dictionary->size--; > hch> - > hch> - status = DAT_SUCCESS; > hch> - goto bail; > hch> - } > hch> - } > hch> - > hch> -bail: > hch> - return status; > hch> -} > hch> - > hch> -static u32 dat_dictionary_key_dup(const struct > dat_provider_info *old_key, > hch> - struct > dat_provider_info *new_key) > hch> -{ > hch> - BUG_ON(NULL == old_key); > hch> - BUG_ON(NULL == new_key); > hch> - > hch> - strncpy(new_key->ia_name, old_key->ia_name, > DAT_NAME_MAX_LENGTH); > hch> - return DAT_SUCCESS; > hch> -} > hch> - > hch> -static boolean_t dat_dictionary_key_is_equal(const > struct dat_provider_info *a, > hch> - const > struct dat_provider_info *b) > hch> -{ > hch> - BUG_ON(NULL == a); > hch> - BUG_ON(NULL == b); > hch> - > hch> - if ((strlen(a->ia_name) == strlen(b->ia_name)) && > hch> - (!strncmp(a->ia_name, b->ia_name, > strlen(a->ia_name)))) > hch> - return TRUE; > hch> - else > hch> - return FALSE; > hch> -} > hch> Index: linux-kernel/dat/core.c > hch> > =================================================================== > hch> --- linux-kernel/dat/core.c (revision 2647) > hch> +++ linux-kernel/dat/core.c (working copy) > hch> @@ -1,75 +0,0 @@ > hch> -/* > hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All > rights reserved. > hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights > reserved. > hch> - * > hch> - * This Software is licensed under one of the following > licenses: > hch> - * > hch> - * 1) under the terms of the "Common Public License > 1.0" a copy of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/cpl.php. > hch> - * > hch> - * 2) under the terms of the "The BSD License" a copy > of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/bsd-license.php. > hch> - * > hch> - * 3) under the terms of the "GNU General Public > License (GPL) Version 2" a > hch> - * copy of which is available from the Open Source > Initiative, see > hch> - * http://www.opensource.org/licenses/gpl-license.php. > hch> - * > hch> - * Licensee has the right to choose one of the above licenses. > hch> - * > hch> - * Redistributions of source code must retain the above > copyright > hch> - * notice and one of the license notices. > hch> - * > hch> - * Redistributions in binary form must reproduce both > the above copyright > hch> - * notice, one of the license notices in the documentation > hch> - * and/or other materials provided with the distribution. > hch> - */ > hch> - > hch> -/* > hch> - * Core DAT registry functions and data structures > hch> - * > hch> - * $Id$ > hch> - */ > hch> - > hch> -#include "core.h" > hch> -#include "dr.h" > hch> - > hch> -MODULE_LICENSE("Dual BSD/GPL"); > hch> -MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); > hch> -MODULE_AUTHOR("James Lentini"); > hch> - > hch> -static DAT_DBG_MASK g_dbg_mask = DAT_DBG_TYPE_ERROR; > hch> -module_param_named(dbg_mask, g_dbg_mask, int, 0644); > hch> -MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug > message types."); > hch> - > hch> -void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...) > hch> -{ > hch> - static char buf[1024]; > hch> - > hch> - if (type & g_dbg_mask) { > hch> - va_list args; > hch> - > hch> - va_start(args, fmt); > hch> - vsnprintf(buf, sizeof buf, fmt, args); > hch> - printk(KERN_ALERT "DAT: %s", buf); > hch> - va_end(args); > hch> - } > hch> -} > hch> - > hch> -static int __init dat_init(void) > hch> -{ > hch> - dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry > started\n"); > hch> - return 0; > hch> -} > hch> - > hch> -module_init(dat_init); > hch> - > hch> -static void __exit dat_fini(void) > hch> -{ > hch> - dat_dr_fini(); > hch> - > hch> - dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry > stopped\n"); > hch> -} > hch> - > hch> -module_exit(dat_fini); > hch> Index: linux-kernel/dat/api.c > hch> > =================================================================== > hch> --- linux-kernel/dat/api.c (revision 2647) > hch> +++ linux-kernel/dat/api.c (working copy) > hch> @@ -26,15 +26,53 @@ > hch> * and/or other materials provided with the distribution. > hch> */ > hch> > hch> -/* > hch> - * DAT functions for consumers > hch> - * > hch> - * $Id: consumer.c 2251 2005-05-04 05:57:41Z jlentini $ > hch> - */ > hch> +#include > hch> +#include > hch> +#include > hch> +#include > hch> +#include > hch> > hch> -#include "core.h" > hch> -#include "dr.h" > hch> +#include > hch> > hch> + > hch> +enum dat_dbg_type { > hch> + DAT_DBG_TYPE_ERROR = (1 << 0), > hch> + DAT_DBG_TYPE_GENERIC = (1 << 1), > hch> + DAT_DBG_TYPE_DR = (1 << 2), > hch> + DAT_DBG_TYPE_PROVIDER_API = (1 << 3), > hch> + DAT_DBG_TYPE_CONSUMER_API = (1 << 4), > hch> + DAT_DBG_TYPE_ALL = 0x1f > hch> +}; > hch> + > hch> +static int dat_dbg_mask = DAT_DBG_TYPE_ERROR; > hch> +module_param_named(dbg_mask, dat_dbg_mask, int, 0644); > hch> +MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug > message types."); > hch> + > hch> +struct dat_dictionary_entry { > hch> + struct list_head list; > hch> + struct dat_provider_info info; > hch> + DAT_IA_OPEN_FUNC ia_open_func; > hch> + int ref_count; > hch> +}; > hch> + > hch> +static LIST_HEAD(dat_provider_list); > hch> +static int dat_provider_list_size; > hch> +static DEFINE_SPINLOCK(dat_provider_list_lock); > hch> + > hch> +static void dat_dbg_print(enum dat_dbg_type type, const > char *fmt, ...) > hch> +{ > hch> + static char buf[1024]; > hch> + > hch> + if (type & dat_dbg_mask) { > hch> + va_list args; > hch> + > hch> + va_start(args, fmt); > hch> + vsnprintf(buf, sizeof buf, fmt, args); > hch> + printk(KERN_ALERT "DAT: %s", buf); > hch> + va_end(args); > hch> + } > hch> +} > hch> + > hch> static u32 dat_strerror_major(u32 value, const char **message) > hch> { > hch> switch (DAT_GET_TYPE(value)) { > hch> @@ -350,21 +388,39 @@ > hch> } > hch> } > hch> > hch> +static boolean_t dat_dictionary_key_is_equal(const > struct dat_provider_info *a, > hch> + const > struct dat_provider_info *b) > hch> +{ > hch> + if (strlen(a->ia_name) == strlen(b->ia_name) && > hch> + !strncmp(a->ia_name, b->ia_name, > strlen(a->ia_name))) > hch> + return 1; > hch> + return 0; > hch> +} > hch> + > hch> +static struct dat_dictionary_entry *dat_dictionary_search( > hch> + struct list_head *list, const struct > dat_provider_info *key) > hch> +{ > hch> + struct dat_dictionary_entry *entry; > hch> + > hch> + list_for_each_entry(entry, list, list) > hch> + if > (dat_dictionary_key_is_equal(&entry->info, key)) > hch> + return entry; > hch> + > hch> + return NULL; > hch> +} > hch> + > hch> u32 dat_ia_open(const char *name, int async_event_qlen, > hch> struct dat_evd **async_event_handle, > struct dat_ia **ia) > hch> { > hch> - DAT_IA_OPEN_FUNC ia_open_func; > hch> struct dat_provider_info info; > hch> - u32 status; > hch> + struct dat_dictionary_entry *data; > hch> + unsigned long flags; > hch> size_t len; > hch> > hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > hch> "%s(name:%s, async_qlen:%i, > ver:%x.%x, thrd_safe:%x)\n", > hch> __func__, name, async_event_qlen); > hch> > hch> - if (DAT_IS_BAD_POINTER(name)) > hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, > DAT_INVALID_ARG1); > hch> - > hch> len = strlen(name); > hch> > hch> if (DAT_NAME_MAX_LENGTH < len) > hch> @@ -373,62 +429,67 @@ > hch> strncpy(info.ia_name, name, len); > hch> info.ia_name[len] = '\0'; > hch> > hch> - status = dat_dr_provider_open(&info, &ia_open_func); > hch> - if (status != DAT_SUCCESS) { > hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); > hch> + data = dat_dictionary_search(&dat_provider_list, &info); > hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); > hch> + > hch> + if (!data) { > hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > hch> "%s: IA [%s] not found in > dynamic registry\n", > hch> __func__, name); > hch> > hch> - return status; > hch> + return DAT_ERROR(DAT_PROVIDER_NOT_FOUND, > hch> + DAT_NAME_NOT_REGISTERED); > hch> } > hch> > hch> - return ia_open_func(name, async_event_qlen, > async_event_handle, ia); > hch> + data->ref_count++; > hch> + return data->ia_open_func(name, > async_event_qlen, async_event_handle, > hch> + ia); > hch> } > hch> - > hch> EXPORT_SYMBOL(dat_ia_open); > hch> > hch> -u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) > hch> +u32 dat_ia_close(struct dat_ia *ia, enum > dat_close_flags close_flags) > hch> { > hch> struct dat_provider *provider; > hch> - struct dat_provider_attr provider_attr; > hch> - u32 status; > hch> + struct dat_provider_info info; > hch> + u32 status = DAT_SUCCESS; > hch> const char *ia_name; > hch> + struct dat_dictionary_entry *data; > hch> + unsigned long flags; > hch> + size_t len; > hch> > hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > hch> - "%s(ia:%x, flags:%x)\n", > __func__, ia, flags); > hch> + "%s(ia:%x, flags:%x)\n", > __func__, ia, close_flags); > hch> > hch> provider = ia->common.provider; > hch> ia_name = provider->device_name; > hch> > hch> - if (DAT_SUCCESS != (status = dat_ia_query(ia, > NULL, NULL, > hch> - > &provider_attr))) > hch> + status = provider->ia_close_func(ia, close_flags); > hch> + if (status != DAT_SUCCESS) { > hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > hch> - "dat_ia_query for IA %s > failed\n", ia_name); > hch> - else if (DAT_SUCCESS != (status = > provider->ia_close_func(ia, flags))) > hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > hch> "close function for IA %s > failed\n", ia_name); > hch> - else { > hch> - struct dat_provider_info info; > hch> - size_t len; > hch> + return status; > hch> + } > hch> > hch> - len = strlen(ia_name); > hch> + len = strlen(ia_name); > hch> > hch> - BUG_ON(DAT_NAME_MAX_LENGTH < len); > hch> + BUG_ON(DAT_NAME_MAX_LENGTH < len); > hch> > hch> - strncpy(info.ia_name, ia_name, len); > hch> - info.ia_name[len] = '\0'; > hch> + strncpy(info.ia_name, ia_name, len); > hch> + info.ia_name[len] = '\0'; > hch> > hch> - status = dat_dr_provider_close(&info); > hch> - if (DAT_SUCCESS != status) > hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > hch> - "dynamic registry > unable to close " > hch> - "provider for IA %s\n", > hch> - ia_name); > hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); > hch> + data = dat_dictionary_search(&dat_provider_list, &info); > hch> + if (data) { > hch> + data->ref_count--; > hch> + } else { > hch> + status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, > hch> + DAT_NAME_NOT_REGISTERED); > hch> } > hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); > hch> > hch> return status; > hch> } > hch> - > hch> EXPORT_SYMBOL(dat_ia_close); > hch> > hch> u32 dat_strerror(u32 value, const char **major_message, > hch> @@ -454,69 +515,138 @@ > hch> > hch> return DAT_SUCCESS; > hch> } > hch> - > hch> EXPORT_SYMBOL(dat_strerror); > hch> > hch> u32 dat_registry_add_provider(const struct dat_provider > *provider, > hch> - const struct > dat_provider_info *provider_info) > hch> + const struct > dat_provider_info *info) > hch> { > hch> - dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, > hch> - "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", > hch> - __func__, provider_info->ia_name); > hch> + struct dat_dictionary_entry *entry; > hch> + unsigned long flags; > hch> + u32 status = DAT_SUCCESS; > hch> > hch> - if (DAT_IS_BAD_POINTER(provider)) > hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, > DAT_INVALID_ARG1); > hch> + entry = kmalloc(sizeof(*entry), GFP_ATOMIC); > hch> + if (!entry) { > hch> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, > hch> + DAT_RESOURCE_MEMORY); > hch> + goto out; > hch> + } > hch> > hch> - if (DAT_IS_BAD_POINTER(provider_info)) > hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, > DAT_INVALID_ARG2); > hch> + entry->ref_count = 0; > hch> + entry->info = *info; > hch> + entry->ia_open_func = provider->ia_open_func; > hch> > hch> - return dat_dr_insert(provider_info, > provider->ia_open_func); > hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); > hch> + if (dat_dictionary_search(&dat_provider_list, info)) { > hch> + status = > DAT_ERROR(DAT_PROVIDER_ALREADY_REGISTERED, 0); > hch> + goto out_unlock; > hch> + } > hch> + > hch> + strncpy(entry->info.ia_name, info->ia_name, > DAT_NAME_MAX_LENGTH); > hch> + > hch> + /* insert node at end of list to preserve > registration order */ > hch> + list_add_tail(&entry->list, &dat_provider_list); > hch> + dat_provider_list_size++; > hch> + > hch> + out_unlock: > hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); > hch> + out: > hch> + if (status != DAT_SUCCESS) > hch> + kfree(entry); > hch> + return status; > hch> } > hch> +EXPORT_SYMBOL(dat_registry_add_provider); > hch> > hch> -EXPORT_SYMBOL(dat_registry_remove_provider); > hch> - > hch> u32 dat_registry_remove_provider(const struct > dat_provider *provider, > hch> - const struct > dat_provider_info *provider_info) > hch> + const struct > dat_provider_info *info) > hch> { > hch> - dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, > "%s(ia:%s)\n", __func__, > hch> - provider_info->ia_name); > hch> + struct dat_dictionary_entry *cur, *next; > hch> + unsigned long flags; > hch> + u32 status = DAT_SUCCESS; > hch> > hch> - if (DAT_IS_BAD_POINTER(provider)) > hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, > DAT_INVALID_ARG1); > hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); > hch> + list_for_each_entry_safe(cur, next, > &dat_provider_list, list) { > hch> + if > (dat_dictionary_key_is_equal(&cur->info, info)) { > hch> + if (cur->ref_count) { > hch> + status = > DAT_ERROR(DAT_PROVIDER_IN_USE, 0); > hch> + goto out; > hch> + } > hch> > hch> - return dat_dr_remove(provider_info); > hch> + list_del(&cur->list); > hch> + dat_provider_list_size--; > hch> + > hch> + kfree(cur); > hch> + goto out; > hch> + } > hch> + } > hch> + > hch> + status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, > DAT_NAME_NOT_REGISTERED); > hch> + out: > hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); > hch> + return status; > hch> } > hch> +EXPORT_SYMBOL(dat_registry_remove_provider); > hch> > hch> -EXPORT_SYMBOL(dat_registry_add_provider); > hch> - > hch> u32 dat_registry_list_providers(int max_to_return, int > *entries_returned, > hch> - struct > dat_provider_info *(dat_provider_list[])) > hch> + struct > dat_provider_info *provider_list[]) > hch> { > hch> + struct dat_dictionary_entry *cur; > hch> + unsigned long flags; > hch> + int i = 0; > hch> u32 status = DAT_SUCCESS; > hch> > hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, > hch> - "%s(max_to_return:%i, > entries_rtrn:%p, prvdr_list:%p)\n", > hch> - __func__, max_to_return, > entries_returned, > hch> - dat_provider_list); > hch> - > hch> - if ((DAT_IS_BAD_POINTER(entries_returned))) > hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, > DAT_INVALID_ARG2); > hch> - > hch> - if (0 != max_to_return && > (DAT_IS_BAD_POINTER(dat_provider_list))) > hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, > DAT_INVALID_ARG3); > hch> - > hch> - if (0 == max_to_return) { > hch> + if (!max_to_return) { > hch> /* the user is allowed to call with > max_to_return set to zero. > hch> * in which case we simply return (in > *entries_returned) the > hch> * number of providers currently > installed. We must also > hch> * (per spec) return an error > hch> */ > hch> - (void)dat_dr_size(entries_returned); > hch> + *entries_returned = dat_provider_list_size; > hch> return DAT_ERROR(DAT_INVALID_PARAMETER, > DAT_INVALID_ARG1); > hch> - } else > hch> - status = dat_dr_list(max_to_return, > entries_returned, > hch> - dat_provider_list); > hch> + } > hch> + > hch> + *entries_returned = 0; > hch> + > hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); > hch> + if (!dat_provider_list_size) > hch> + goto out_unlock; > hch> + > hch> + list_for_each_entry(cur, &dat_provider_list, list) { > hch> + if (i >= max_to_return) > hch> + break; > hch> + if (!provider_list[i]) { > hch> + status = > hch> + > DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); > hch> + goto out_unlock; > hch> + } > hch> + provider_list[i++] = &cur->info; > hch> + } > hch> + *entries_returned = i; > hch> + > hch> + out_unlock: > hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); > hch> return status; > hch> } > hch> +EXPORT_SYMBOL(dat_registry_list_providers); > hch> > hch> -EXPORT_SYMBOL(dat_registry_list_providers); > hch> +static int __init dat_init(void) > hch> +{ > hch> + dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry > started\n"); > hch> + return 0; > hch> +} > hch> + > hch> +static void __exit dat_fini(void) > hch> +{ > hch> + struct dat_dictionary_entry *cur, *next; > hch> + > hch> + list_for_each_entry_safe(cur, next, > &dat_provider_list, list) { > hch> + list_del(&cur->list); > hch> + kfree(cur); > hch> + } > hch> +} > hch> + > hch> +MODULE_LICENSE("Dual BSD/GPL"); > hch> +MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); > hch> +MODULE_AUTHOR("James Lentini"); > hch> + > hch> +module_init(dat_init); > hch> +module_exit(dat_fini); > hch> Index: linux-kernel/dat/dictionary.h > hch> > =================================================================== > hch> --- linux-kernel/dat/dictionary.h (revision 2647) > hch> +++ linux-kernel/dat/dictionary.h (working copy) > hch> @@ -1,98 +0,0 @@ > hch> -/* > hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All > rights reserved. > hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights > reserved. > hch> - * > hch> - * This Software is licensed under one of the following > licenses: > hch> - * > hch> - * 1) under the terms of the "Common Public License > 1.0" a copy of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/cpl.php. > hch> - * > hch> - * 2) under the terms of the "The BSD License" a copy > of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/bsd-license.php. > hch> - * > hch> - * 3) under the terms of the "GNU General Public > License (GPL) Version 2" a > hch> - * copy of which is available from the Open Source > Initiative, see > hch> - * http://www.opensource.org/licenses/gpl-license.php. > hch> - * > hch> - * Licensee has the right to choose one of the above licenses. > hch> - * > hch> - * Redistributions of source code must retain the above > copyright > hch> - * notice and one of the license notices. > hch> - * > hch> - * Redistributions in binary form must reproduce both > the above copyright > hch> - * notice, one of the license notices in the documentation > hch> - * and/or other materials provided with the distribution. > hch> - */ > hch> - > hch> -/* > hch> - * dictionary data structure > hch> - * > hch> - * $Id$ > hch> - */ > hch> - > hch> -#ifndef DICTIONARY_H > hch> -#define DICTIONARY_H > hch> - > hch> -#include "core.h" > hch> - > hch> > -/************************************************************ > ********* > hch> - * > * > hch> - * Structures > * > hch> - * > * > hch> - > *********************************************************************/ > hch> - > hch> -struct dat_dictionary_entry { > hch> - struct list_head list; > hch> - struct dat_provider_info key; > hch> - void *data; > hch> -}; > hch> - > hch> -struct dat_dictionary { > hch> - struct list_head list; > hch> - int size; > hch> -}; > hch> - > hch> > -/************************************************************ > ********* > hch> - * > * > hch> - * Macros > * > hch> - * > * > hch> - > *********************************************************************/ > hch> - > hch> -#define DAT_DICTIONARY_INIT(name) { > LIST_HEAD_INIT((name).list), 0 } > hch> - > hch> > -/************************************************************ > ********* > hch> - * > * > hch> - * Function Prototypes > * > hch> - * > * > hch> - > *********************************************************************/ > hch> - > hch> -extern void dat_dictionary_destroy(struct > dat_dictionary *dictionary); > hch> - > hch> -extern u32 dat_dictionary_size(const struct > dat_dictionary *dictionary, > hch> - int *size); > hch> - > hch> -extern u32 dat_dictionary_entry_create(struct > dat_dictionary_entry **p_entry); > hch> - > hch> -extern void dat_dictionary_entry_destroy( > hch> - const struct dat_dictionary_entry *entry); > hch> - > hch> -extern u32 dat_dictionary_insert(struct dat_dictionary > *dictionary, > hch> - const struct > dat_provider_info *key, > hch> - struct > dat_dictionary_entry *entry, > hch> - void *data); > hch> - > hch> -extern u32 dat_dictionary_search(struct dat_dictionary > *dictionary, > hch> - const struct > dat_provider_info *key, > hch> - void **p_data); > hch> - > hch> -extern u32 dat_dictionary_enumerate(struct > dat_dictionary *dictionary, > hch> - void *array[], > const int array_size); > hch> - > hch> -extern u32 dat_dictionary_remove(struct dat_dictionary > *dictionary, > hch> - const struct > dat_provider_info *key, > hch> - struct > dat_dictionary_entry **p_entry, > hch> - void **p_data); > hch> - > hch> -#endif /* DICTIONARY_H */ > hch> Index: linux-kernel/dat/dr.h > hch> > =================================================================== > hch> --- linux-kernel/dat/dr.h (revision 2647) > hch> +++ linux-kernel/dat/dr.h (working copy) > hch> @@ -1,57 +0,0 @@ > hch> -/* > hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All > rights reserved. > hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights > reserved. > hch> - * > hch> - * This Software is licensed under one of the following > licenses: > hch> - * > hch> - * 1) under the terms of the "Common Public License > 1.0" a copy of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/cpl.php. > hch> - * > hch> - * 2) under the terms of the "The BSD License" a copy > of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/bsd-license.php. > hch> - * > hch> - * 3) under the terms of the "GNU General Public > License (GPL) Version 2" a > hch> - * copy of which is available from the Open Source > Initiative, see > hch> - * http://www.opensource.org/licenses/gpl-license.php. > hch> - * > hch> - * Licensee has the right to choose one of the above licenses. > hch> - * > hch> - * Redistributions of source code must retain the above > copyright > hch> - * notice and one of the license notices. > hch> - * > hch> - * Redistributions in binary form must reproduce both > the above copyright > hch> - * notice, one of the license notices in the documentation > hch> - * and/or other materials provided with the distribution. > hch> - */ > hch> - > hch> -/* > hch> - * Dynamic Registry (DR) functions and data stucture > hch> - * > hch> - * $Id$ > hch> - */ > hch> - > hch> -#ifndef DR_H > hch> -#define DR_H > hch> - > hch> -#include > hch> - > hch> -extern void dat_dr_fini(void); > hch> - > hch> -extern u32 dat_dr_insert(const struct dat_provider_info *info, > hch> - const DAT_IA_OPEN_FUNC ia_open_func); > hch> - > hch> -extern u32 dat_dr_remove(const struct dat_provider_info *info); > hch> - > hch> -extern u32 dat_dr_provider_open(const struct > dat_provider_info *info, > hch> - DAT_IA_OPEN_FUNC > *p_ia_open_func); > hch> - > hch> -extern u32 dat_dr_provider_close(const struct > dat_provider_info *info); > hch> - > hch> -extern u32 dat_dr_size(int *size); > hch> - > hch> -extern u32 dat_dr_list(int max_to_return, int *entries_returned, > hch> - struct dat_provider_info > *(dat_provider_list[])); > hch> - > hch> -#endif /* DR_H */ > hch> Index: linux-kernel/dat/Makefile > hch> > =================================================================== > hch> --- linux-kernel/dat/Makefile (revision 2647) > hch> +++ linux-kernel/dat/Makefile (working copy) > hch> @@ -5,8 +5,4 @@ > hch> > hch> obj-$(CONFIG_DAT) += dat.o > hch> > hch> -dat-y := \ > hch> - api.o \ > hch> - core.o \ > hch> - dictionary.o \ > hch> - dr.o > hch> +dat-y += api.o > hch> Index: linux-kernel/dat/core.h > hch> > =================================================================== > hch> --- linux-kernel/dat/core.h (revision 2647) > hch> +++ linux-kernel/dat/core.h (working copy) > hch> @@ -1,61 +0,0 @@ > hch> -/* > hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All > rights reserved. > hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights > reserved. > hch> - * > hch> - * This Software is licensed under one of the following > licenses: > hch> - * > hch> - * 1) under the terms of the "Common Public License > 1.0" a copy of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/cpl.php. > hch> - * > hch> - * 2) under the terms of the "The BSD License" a copy > of which is > hch> - * available from the Open Source Initiative, see > hch> - * http://www.opensource.org/licenses/bsd-license.php. > hch> - * > hch> - * 3) under the terms of the "GNU General Public > License (GPL) Version 2" a > hch> - * copy of which is available from the Open Source > Initiative, see > hch> - * http://www.opensource.org/licenses/gpl-license.php. > hch> - * > hch> - * Licensee has the right to choose one of the above licenses. > hch> - * > hch> - * Redistributions of source code must retain the above > copyright > hch> - * notice and one of the license notices. > hch> - * > hch> - * Redistributions in binary form must reproduce both > the above copyright > hch> - * notice, one of the license notices in the documentation > hch> - * and/or other materials provided with the distribution. > hch> - */ > hch> - > hch> -/* > hch> - * core DAT registry functions and data structures > hch> - * > hch> - * $Id$ > hch> - */ > hch> - > hch> -#ifndef CORE_H > hch> -#define CORE_H > hch> - > hch> -#include > hch> -#include > hch> -#include > hch> -#include > hch> -#include > hch> - > hch> -#include > hch> - > hch> -#define DAT_IS_BAD_POINTER(p) ( NULL == (p) ) > hch> - > hch> -typedef enum { > hch> - DAT_DBG_TYPE_ERROR = (1 << 0), > hch> - DAT_DBG_TYPE_GENERIC = (1 << 1), > hch> - DAT_DBG_TYPE_DR = (1 << 2), > hch> - DAT_DBG_TYPE_PROVIDER_API = (1 << 3), > hch> - DAT_DBG_TYPE_CONSUMER_API = (1 << 4), > hch> - DAT_DBG_TYPE_ALL = 0x1f > hch> -} DAT_DBG_TYPE; > hch> - > hch> -typedef int DAT_DBG_MASK; > hch> - > hch> -extern void dat_dbg_print(DAT_DBG_TYPE type, const char > *fmt, ...); > hch> - > hch> -#endif /* CORE_H */ > hch> > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jlentini at netapp.com Thu Jun 23 07:15:29 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 10:15:29 -0400 (EDT) Subject: [openib-general] kdapl locking problem In-Reply-To: <20050621115134.GA17046@mellanox.co.il> References: <20050621115134.GA17046@mellanox.co.il> Message-ID: In this instance, the code path getting us in trouble is: ib_modify_qp() calls mthca_modify_qp() calls mthca_MODIFY_QP() calls mthca_cmd_box() calls mthca_cmd_wait() calls wait_for_completion() calls schedule() but there are numerous code paths that reach mthca_cmd_wait(). We may have made this mistake in other places. Roland, is an OpenIB user allowed to hold a spin lock over a verbs call? On Tue, 21 Jun 2005, Itamar Rabenstein wrote: itamar> >Jun 20 09:13:50 localhost kernel: dapl_cm_active_cb_handler 138 event = 9 itamar> >Jun 20 09:13:50 localhost kernel: dapl_evd_connection_callback 760 event=16389 itamar> >Jun 20 09:13:50 localhost kernel: dapl_evd_connection_callback 798 ep = c0a17bf8 itamar> >Jun 20 09:13:50 localhost kernel: dapl_ib_disconnect_clean 579 ep=c0a17bf8 itamar> >Jun 20 09:13:50 localhost kernel: dapl_ib_disconnect 538 ep=c0a17bf8 flags=0 itamar> >Jun 20 09:13:50 localhost kernel: dapl_destroy_cm_id 63 itamar> >Jun 20 09:13:50 localhost kernel: dapl_modify_qp_state_to_error 300 itamar> >Jun 20 09:13:50 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_ep.c:1111: spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c0a17c08) already locked by drivers/infiniband/ulp/dat-provider/dapl_evd.c/759 itamar> >Jun 20 09:13:51 localhost kernel: dapl_modify_qp_state_to_error 305 itamar> >Jun 20 09:13:51 localhost kernel: dapl_evd_connection_callback 800 ep = c0a17bf8 itamar> >Jun 20 09:13:51 localhost kernel: drivers/infiniband/ulp/dat-provider/dapl_evd.c:802: spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep.c:c0a17c08) not locked itamar> >Jun 20 09:13:51 localhost kernel: dapl_cr_callback 434 event=16389 itamar> >Jun 20 09:13:51 localhost kernel: dapl_cr_callback 512 ep = c9821bf8 itamar> >Jun 20 09:13:51 localhost kernel: dapl_ib_disconnect_clean 579 ep=c982m: DREQ rcvd itamar> itamar> Hi Hal, itamar> I think i have found the problem. It is not legal to call ib_modify_qp() itamar> inside spin_lock. This function can sleep. itamar> itamar> Please try the patch below and if it works please tell james to ci it. itamar> itamar> fix locking problem in cm callback functions itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: dapl_openib_util.h itamar> =================================================================== itamar> --- dapl_openib_util.h (revision 2665) itamar> +++ dapl_openib_util.h (working copy) itamar> @@ -125,7 +125,7 @@ itamar> itamar> void dapl_ib_reinit_ep(struct dapl_ep *ep); itamar> itamar> -void dapl_ib_disconnect_clean(struct dapl_ep *ep, boolean_t passive); itamar> +void dapl_ib_disconnect_clean(struct dapl_ep *ep); itamar> itamar> u32 dapl_ib_get_async_event(struct ib_event *cause, itamar> enum dat_event_number *async_event); itamar> Index: dapl_cr.c itamar> =================================================================== itamar> --- dapl_cr.c (revision 2665) itamar> +++ dapl_cr.c (working copy) itamar> @@ -479,8 +479,7 @@ itamar> /* If someone pulled the plug on the EP or connection, itamar> * just exit itamar> */ itamar> - spin_unlock_irqrestore(&ep->common.lock, itamar> - ep->common.flags); itamar> + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> status = DAT_SUCCESS; itamar> /* Set evd = NULL so we don't generate an event below */ itamar> evd = NULL; itamar> @@ -504,36 +503,23 @@ itamar> /* The disconnect has already occurred, we are now itamar> * cleaned up and ready to exit itamar> */ itamar> - spin_unlock_irqrestore(&ep->common.lock, itamar> - ep->common.flags); itamar> + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> return; itamar> } itamar> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; itamar> - dapl_ib_disconnect_clean(ep, FALSE); itamar> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> + dapl_ib_disconnect_clean(ep); itamar> itamar> break; itamar> case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: itamar> case DAT_CONNECTION_EVENT_PEER_REJECTED: itamar> case DAT_CONNECTION_EVENT_UNREACHABLE: itamar> - /* itamar> - * After posting an accept the requesting node has itamar> - * stopped talking. itamar> - */ itamar> + case DAT_CONNECTION_EVENT_BROKEN: itamar> spin_lock_irqsave(&ep->common.lock, ep->common.flags); itamar> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; itamar> - dapl_ib_disconnect_clean(ep, FALSE); itamar> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> - itamar> + dapl_ib_disconnect_clean(ep); itamar> break; itamar> - case DAT_CONNECTION_EVENT_BROKEN: itamar> - spin_lock_irqsave(&ep->common.lock, ep->common.flags); itamar> - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; itamar> - dapl_ib_disconnect_clean(ep, FALSE); itamar> - spin_unlock_irqrestore(&ep->common.lock, itamar> - ep->common.flags); itamar> - itamar> - break; itamar> default: itamar> evd = NULL; itamar> dapl_os_assert(0); /* shouldn't happen */ itamar> Index: dapl_evd.c itamar> =================================================================== itamar> --- dapl_evd.c (revision 2665) itamar> +++ dapl_evd.c (working copy) itamar> @@ -760,7 +760,6 @@ itamar> itamar> switch (event) { itamar> case DAT_CONNECTION_EVENT_ESTABLISHED: itamar> - { itamar> /* itamar> * If we don't have an EP at this point we are very screwed up itamar> */ itamar> @@ -784,65 +783,28 @@ itamar> private_data_size); itamar> } itamar> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> - itamar> break; itamar> - } itamar> case DAT_CONNECTION_EVENT_DISCONNECTED: itamar> - { itamar> - /* itamar> - * EP is now fully disconnected; initiate any post processing itamar> - * to reset the underlying QP and get the EP ready for itamar> - * another connection itamar> - */ itamar> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; itamar> - dapl_ib_disconnect_clean(ep, TRUE); itamar> - spin_unlock_irqrestore(&ep->common.lock, itamar> - ep->common.flags); itamar> - itamar> + spin_unlock_irqrestore(&ep->common.lock,ep->common.flags); itamar> + dapl_ib_disconnect_clean(ep); itamar> break; itamar> - } itamar> case DAT_CONNECTION_EVENT_PEER_REJECTED: itamar> - { itamar> - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; itamar> - dapl_ib_disconnect_clean(ep, TRUE); itamar> - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> - itamar> - break; itamar> - } itamar> case DAT_CONNECTION_EVENT_UNREACHABLE: itamar> - { itamar> - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; itamar> - dapl_ib_disconnect_clean(ep, TRUE); itamar> - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> - itamar> - break; itamar> - } itamar> case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: itamar> - { itamar> - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; itamar> - dapl_ib_disconnect_clean(ep, TRUE); itamar> - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> - itamar> - break; itamar> - } itamar> case DAT_CONNECTION_EVENT_BROKEN: itamar> - { itamar> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; itamar> - dapl_ib_disconnect_clean(ep, FALSE); itamar> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> - itamar> + dapl_ib_disconnect_clean(ep); itamar> break; itamar> - } itamar> case DAT_CONNECTION_REQUEST_EVENT: itamar> default: itamar> - { itamar> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); itamar> evd = NULL; itamar> itamar> dapl_os_assert(0); /* shouldn't happen */ itamar> break; itamar> } itamar> - } itamar> itamar> /* itamar> * Post the event itamar> Index: dapl_openib_cm.c itamar> =================================================================== itamar> --- dapl_openib_cm.c (revision 2665) itamar> +++ dapl_openib_cm.c (working copy) itamar> @@ -562,13 +562,12 @@ itamar> * void itamar> * itamar> */ itamar> -void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t active) itamar> +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr) itamar> { itamar> int status; itamar> itamar> dapl_dbg_log(DAPL_DBG_TYPE_CM, itamar> - " >>> dapl_ib_disconnect_clean: EP: %p active %d\n", itamar> - ep_ptr, active); itamar> + " >>> dapl_ib_disconnect_clean: EP: %p \n", ep_ptr); itamar> itamar> /* itamar> * Clean up outstanding connection state itamar> -- itamar> Itamar itamar> From jlentini at netapp.com Thu Jun 23 07:38:31 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 10:38:31 -0400 (EDT) Subject: [openib-general] mapping between IP address and device name In-Reply-To: <1119359005.4476.2373.camel@localhost.localdomain> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> Message-ID: On Tue, 21 Jun 2005, Hal Rosenstock wrote: > On Mon, 2005-06-20 at 23:40, Roland Dreier wrote: >> There was some discussion a while ago about an "IB address >> translation" service proposed by the Voltaire crew, which would >> encapsulate some of this. However, we didn't make much progress >> towards a good API design, and the discussion fizzled out. > > IBAT is currently being used in conjunction with kdapl. There is further > work to incorporate ATS (service records) going on currently as well. > > Perhaps it is time to restart the IBAT API discussion as to what needs > to change to make it acceptable for the mainstream. The IBAT API provides the two key services needed by kDAPL: the ability to obtain a route based on an IP address and the ability to map a GID to an IP address. Is there agreement that an IB address translation service must provide these two services? If we have the requirements down, we can concentrate on the implementation. james From jlentini at netapp.com Thu Jun 23 07:46:41 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 10:46:41 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kDAPL} Fix locking problem in some CM callback functions In-Reply-To: <1119471298.4480.145.camel@hal.voltaire.com> References: <1119471298.4480.145.camel@hal.voltaire.com> Message-ID: Committed in revision 2686. On Wed, 22 Jun 2005, Hal Rosenstock wrote: halr> Fix locking problem in some CM callback functions halr> Since modify QP can sleep, dapl_ib_disconnect must be called without halr> holding spinlock. halr> halr> Signed-off-by: Itamar Rabenstein halr> Signed-off-by: Hal Rosenstock halr> halr> Index: dapl_openib_util.h halr> =================================================================== halr> --- dapl_openib_util.h (revision 2681) halr> +++ dapl_openib_util.h (working copy) halr> @@ -125,7 +125,7 @@ halr> halr> void dapl_ib_reinit_ep(struct dapl_ep *ep); halr> halr> -void dapl_ib_disconnect_clean(struct dapl_ep *ep, boolean_t passive); halr> +void dapl_ib_disconnect_clean(struct dapl_ep *ep); halr> halr> u32 dapl_ib_get_async_event(struct ib_event *cause, halr> enum dat_event_number *async_event); halr> Index: dapl_cr.c halr> =================================================================== halr> --- dapl_cr.c (revision 2681) halr> +++ dapl_cr.c (working copy) halr> @@ -397,7 +397,7 @@ halr> struct dapl_ep *ep; halr> struct dapl_evd *evd; halr> u32 status; halr> - halr> + halr> dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, halr> "--> dapl_cr_callback! cm_handle: %p event: %x sp: %p\n", halr> cm_ctx, event, sp); halr> @@ -431,7 +431,6 @@ halr> } halr> halr> status = DAT_INTERNAL_ERROR; /* init to ERR */ halr> - halr> switch (event) { halr> case DAT_CONNECTION_REQUEST_EVENT: halr> /* halr> @@ -479,8 +478,7 @@ halr> /* If someone pulled the plug on the EP or connection, halr> * just exit halr> */ halr> - spin_unlock_irqrestore(&ep->common.lock, halr> - ep->common.flags); halr> + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> status = DAT_SUCCESS; halr> /* Set evd = NULL so we don't generate an event below */ halr> evd = NULL; halr> @@ -504,36 +502,22 @@ halr> /* The disconnect has already occurred, we are now halr> * cleaned up and ready to exit halr> */ halr> - spin_unlock_irqrestore(&ep->common.lock, halr> - ep->common.flags); halr> + spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> return; halr> } halr> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> - dapl_ib_disconnect_clean(ep, FALSE); halr> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> - halr> + dapl_ib_disconnect_clean(ep); halr> break; halr> case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: halr> case DAT_CONNECTION_EVENT_PEER_REJECTED: halr> case DAT_CONNECTION_EVENT_UNREACHABLE: halr> - /* halr> - * After posting an accept the requesting node has halr> - * stopped talking. halr> - */ halr> + case DAT_CONNECTION_EVENT_BROKEN: halr> spin_lock_irqsave(&ep->common.lock, ep->common.flags); halr> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> - dapl_ib_disconnect_clean(ep, FALSE); halr> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> - halr> + dapl_ib_disconnect_clean(ep); halr> break; halr> - case DAT_CONNECTION_EVENT_BROKEN: halr> - spin_lock_irqsave(&ep->common.lock, ep->common.flags); halr> - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> - dapl_ib_disconnect_clean(ep, FALSE); halr> - spin_unlock_irqrestore(&ep->common.lock, halr> - ep->common.flags); halr> - halr> - break; halr> default: halr> evd = NULL; halr> dapl_os_assert(0); /* shouldn't happen */ halr> Index: dapl_evd.c halr> =================================================================== halr> --- dapl_evd.c (revision 2681) halr> +++ dapl_evd.c (working copy) halr> @@ -757,10 +757,8 @@ halr> * when necessary halr> */ halr> spin_lock_irqsave(&ep->common.lock, ep->common.flags); halr> - halr> switch (event) { halr> case DAT_CONNECTION_EVENT_ESTABLISHED: halr> - { halr> /* halr> * If we don't have an EP at this point we are very screwed up halr> */ halr> @@ -784,65 +782,28 @@ halr> private_data_size); halr> } halr> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> - halr> break; halr> - } halr> case DAT_CONNECTION_EVENT_DISCONNECTED: halr> - { halr> - /* halr> - * EP is now fully disconnected; initiate any post processing halr> - * to reset the underlying QP and get the EP ready for halr> - * another connection halr> - */ halr> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> - dapl_ib_disconnect_clean(ep, TRUE); halr> - spin_unlock_irqrestore(&ep->common.lock, halr> - ep->common.flags); halr> - halr> - break; halr> - } halr> - case DAT_CONNECTION_EVENT_PEER_REJECTED: halr> - { halr> - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> - dapl_ib_disconnect_clean(ep, TRUE); halr> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> - halr> + dapl_ib_disconnect_clean(ep); halr> break; halr> - } halr> + case DAT_CONNECTION_EVENT_PEER_REJECTED: halr> case DAT_CONNECTION_EVENT_UNREACHABLE: halr> - { halr> - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> - dapl_ib_disconnect_clean(ep, TRUE); halr> - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> - halr> - break; halr> - } halr> case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: halr> - { halr> - ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> - dapl_ib_disconnect_clean(ep, TRUE); halr> - spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> - halr> - break; halr> - } halr> case DAT_CONNECTION_EVENT_BROKEN: halr> - { halr> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> - dapl_ib_disconnect_clean(ep, FALSE); halr> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> - halr> + dapl_ib_disconnect_clean(ep); halr> break; halr> - } halr> case DAT_CONNECTION_REQUEST_EVENT: halr> default: halr> - { halr> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> evd = NULL; halr> halr> dapl_os_assert(0); /* shouldn't happen */ halr> break; halr> } halr> - } halr> halr> /* halr> * Post the event halr> Index: dapl_openib_cm.c halr> =================================================================== halr> --- dapl_openib_cm.c (revision 2681) halr> +++ dapl_openib_cm.c (working copy) halr> @@ -553,7 +553,6 @@ halr> * halr> * Input: halr> * ep_ptr pointer to dapl_ep struct halr> - * active Indicates active side of connection halr> * halr> * Output: halr> * none halr> @@ -562,13 +561,12 @@ halr> * void halr> * halr> */ halr> -void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t active) halr> +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr) halr> { halr> int status; halr> halr> dapl_dbg_log(DAPL_DBG_TYPE_CM, halr> - " >>> dapl_ib_disconnect_clean: EP: %p active %d\n", halr> - ep_ptr, active); halr> + " >>> dapl_ib_disconnect_clean: EP: %p\n", ep_ptr); halr> halr> /* halr> * Clean up outstanding connection state halr> halr> halr> From halr at voltaire.com Thu Jun 23 07:57:01 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 10:57:01 -0400 Subject: [openib-general] Setting kdapl ep_state relative to calling disconnect Message-ID: <1119538391.4476.24.camel@localhost.localdomain> Hi, In dapl_evd.c, there are cases where the ep_state is updated before calling disconnect and others where this is done after. Should these be made consistent ? void dapl_evd_connection_callback(...) { ... case DAT_CONNECTION_EVENT_DISCONNECTED: ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); dapl_ib_disconnect_clean(ep); break; case DAT_CONNECTION_EVENT_PEER_REJECTED: case DAT_CONNECTION_EVENT_UNREACHABLE: case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: case DAT_CONNECTION_EVENT_BROKEN: ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); dapl_ib_disconnect_clean(ep); break; void dapl_evd_connection_callback(...) { dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); spin_lock_irqsave(&ep->common.lock, ep->common.flags); ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); -- Hal From halr at voltaire.com Thu Jun 23 08:15:39 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 11:15:39 -0400 Subject: [openib-general] [PATCH] Fix ib_pack/unpack for 64 bits Message-ID: <1119539738.4617.2.camel@localhost.localdomain> Fix ib_pack/unpack for 64 bits Signed-off-by: Hal Rosenstock Index: packer.c =================================================================== -- packer.c (revision 2681) +++ packer.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -96,7 +97,10 @@ else val = 0; - mask = cpu_to_be64(((1ull << desc[i].size_bits) - 1) << shift); + if (desc[i].size_bits < 64) + mask = cpu_to_be64(((1ull << desc[i].size_bits) - 1) << shift); + else + mask = cpu_to_be64(0xffffffffffffffff << shift); addr = (__be64 *) ((__be32 *) buf + desc[i].offset_words); *addr = (*addr & ~mask) | (cpu_to_be64(val) & mask); } else { @@ -176,7 +180,10 @@ __be64 *addr; shift = 64 - desc[i].offset_bits - desc[i].size_bits; - mask = ((1ull << desc[i].size_bits) - 1) << shift; + if (desc[i].size_bits < 64) + mask = ((1ull << desc[i].size_bits) - 1) << shift; + else + mask = cpu_to_be64(0xffffffffffffffff << shift); addr = (__be64 *) buf + desc[i].offset_words; val = (be64_to_cpup(addr) & mask) >> shift; value_write(desc[i].struct_offset_bytes, From roland at topspin.com Thu Jun 23 08:47:06 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 08:47:06 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: (James Lentini's message of "Thu, 23 Jun 2005 10:38:31 -0400 (EDT)") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> Message-ID: <527jgloz2d.fsf@topspin.com> James> The IBAT API provides the two key services needed by kDAPL: James> the ability to obtain a route based on an IP address and James> the ability to map a GID to an IP address. James> Is there agreement that an IB address translation service James> must provide these two services? I think the first service is reasonable, although one has to be careful to design it so it works with things like IPv6 link-local addresses. The second service, mapping a GID to an IP address, doesn't make sense to me: for a given GID, there may be no associated IPoIB address, or there may be many IPoIB addresses -- in fact, it is difficult to configure an IPoIB interface so that it doesn't have both IPv4 and IPv6 addresses. - R. From roland at topspin.com Thu Jun 23 08:51:46 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 08:51:46 -0700 Subject: [openib-general] Re: page allocation failure with kdapl/mthca In-Reply-To: <20050623064208.GE21363@esmail.cup.hp.com> (Grant Grundler's message of "Wed, 22 Jun 2005 23:42:08 -0700") References: <1119366787.4477.26.camel@localhost.localdomain> <52d5qfwtrz.fsf@topspin.com> <52d5qeqksl.fsf@topspin.com> <20050623064208.GE21363@esmail.cup.hp.com> Message-ID: <523br9oyul.fsf@topspin.com> Roland> I just checked in a change that converts from using Roland> pci_alloc_consistent(), which is always GFP_ATOMIC, to Roland> dma_alloc_coherent(..., GFP_KERNEL). Grant> This arch specific. Grant> On ia64 I expect both functions to land in the same bit of Grant> code. For ZX1 platforms, that is sba_alloc_coherent(). Grant> See arch/ia64/hp/common/sba_iommu.c since the behavior is Grant> different depending on CONFIG_NUMA setting. Sure, but follow the code all the way through. ia64 uses , which defines pci_alloc_consistent() to be dma_alloc_coherent(..., GFP_ATOMIC). You're right that both will end up in sba_alloc_coherent(), but the difference is the flags parameter. GFP_ATOMIC allocations can easily fail even when the system has plenty of pages available for GFP_KERNEL allocations. - R. From roland at topspin.com Thu Jun 23 08:53:30 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 08:53:30 -0700 Subject: [openib-general] RDMA write with immediate In-Reply-To: <1119530877.29406.29.camel@localhost> (Josh England's message of "Thu, 23 Jun 2005 05:47:57 -0700") References: <1119480693.24922.33.camel@localhost> <52k6klq184.fsf@topspin.com> <1119530877.29406.29.camel@localhost> Message-ID: <52y891nk79.fsf@topspin.com> > # ll /dev/infiniband > crw-r--r-- 1 root root 231, 255 Jun 23 05:31 ucm > crw-rw-rw- 1 root root 231, 128 Jun 23 05:31 uverbs0 > crw-rw-rw- 1 root root 231, 129 Jun 23 05:31 uverbs1 If you're using a static /dev directory, you'll need to remake the /dev/infiniband/uverbsN character nodes starting at minor 192. - R. From iod00d at hp.com Thu Jun 23 08:58:42 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 23 Jun 2005 08:58:42 -0700 Subject: [openib-general] Re: page allocation failure with kdapl/mthca In-Reply-To: <1119523521.4480.232.camel@hal.voltaire.com> References: <1119366787.4477.26.camel@localhost.localdomain> <52d5qfwtrz.fsf@topspin.com> <52d5qeqksl.fsf@topspin.com> <20050623064208.GE21363@esmail.cup.hp.com> <1119523521.4480.232.camel@hal.voltaire.com> Message-ID: <20050623155842.GD22934@esmail.cup.hp.com> On Thu, Jun 23, 2005 at 06:45:21AM -0400, Hal Rosenstock wrote: > > On ia64 I expect both functions to land in the same bit of code. > > For ZX1 platforms, that is sba_alloc_coherent(). > > See arch/ia64/hp/common/sba_iommu.c since the behavior is > > different depending on CONFIG_NUMA setting. > > > > Sorry, I missed/forgot what the original problem was. > > The original problem (for me) was that there was a page allocation > failure with kdapl/mthca on a create QP (and the driver changed but > kdapl didn't in terms of this). I'm not sure which change introduced the > problem but it was fixed with r2666. Ah ok. We (or at least I) haven't "leaned on" openib.org very hard yet in terms of memory pressure and "normal" operation. I've asked for some help with testing inside HP and expect I can look more into this after OLS/LWE-SF/HPWorld in July/August. thanks, grant From roland at topspin.com Thu Jun 23 08:58:29 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 08:58:29 -0700 Subject: [openib-general] RDMA write with immediate In-Reply-To: <1119526607.29410.6.camel@localhost> (Josh England's message of "Thu, 23 Jun 2005 04:36:46 -0700") References: <1119480693.24922.33.camel@localhost> <52k6klq184.fsf@topspin.com> <1119526607.29410.6.camel@localhost> Message-ID: <52u0jpnjyy.fsf@topspin.com> Josh> I get a completion with failed status 13. Josh> Again, this works just fine with IBV_WR_RDMA_WRITE. I'm Josh> using 2.6.11.11 with SVN rev 2606. I'll update stuff and Josh> try again. status 13 is "RNR retries exceeded." This means that the remote side did not have a receive ready to handle this work request. What are you trying to do here? The between RDMA write and RDMA write with immediate operations is sending immediate data consumes a receive request on the remote system and generates a completion with the immediate data. So you need to post a receive to handle RDMA write with immediate but not regular RDMA write. Did you modify your app to post the receive here? - R. From roland at topspin.com Thu Jun 23 09:04:36 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 09:04:36 -0700 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <1119539738.4617.2.camel@localhost.localdomain> (Hal Rosenstock's message of "23 Jun 2005 11:15:39 -0400") References: <1119539738.4617.2.camel@localhost.localdomain> Message-ID: <52psudnjor.fsf@topspin.com> Hal> Fix ib_pack/unpack for 64 bits Can you be more specific about what this is fixing? - R. From halr at voltaire.com Thu Jun 23 09:03:09 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 12:03:09 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <527jgloz2d.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> Message-ID: <1119542588.4477.2.camel@localhost.localdomain> On Thu, 2005-06-23 at 11:47, Roland Dreier wrote: > James> The IBAT API provides the two key services needed by kDAPL: > James> the ability to obtain a route based on an IP address and > James> the ability to map a GID to an IP address. > > James> Is there agreement that an IB address translation service > James> must provide these two services? > > I think the first service is reasonable, although one has to be > careful to design it so it works with things like IPv6 link-local > addresses. The second service, mapping a GID to an IP address, > doesn't make sense to me: for a given GID, there may be no associated > IPoIB address, or there may be many IPoIB addresses -- in fact, it is > difficult to configure an IPoIB interface so that it doesn't have both > IPv4 and IPv6 addresses. In terms of mapping a GID to an IP address, the administrator can choose whether to run IPoIB or not and how it is configured. In the case of multiple addresses, the set of addresses is returned. In the case of no associated addresses, then an error is returned. I -- Hal From roland at topspin.com Thu Jun 23 09:08:40 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 09:08:40 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050623055835.GA10456@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 23 Jun 2005 08:58:35 +0300") References: <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> Message-ID: <52k6klnjhz.fsf@topspin.com> Michael> Currently I want to be able to script linking mthca.a Michael> statically. Michael> In the future I can imagine someone might want to script Michael> installing additional plugins into an existing system. I'm not sure this makes sense. libibverbs has a certain default path but users can easily override it by setting the OPENIB_DRIVER_PATH environment variable. For example, I often compile libmthca but don't install it, and then point OPENIB_DRIVER_PATH at my source directory. Is there any way we can use an existing tool like pkg-config to help with this situation? - R. From iod00d at hp.com Thu Jun 23 09:15:42 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 23 Jun 2005 09:15:42 -0700 Subject: [openib-general] Re: page allocation failure with kdapl/mthca In-Reply-To: <523br9oyul.fsf@topspin.com> References: <1119366787.4477.26.camel@localhost.localdomain> <52d5qfwtrz.fsf@topspin.com> <52d5qeqksl.fsf@topspin.com> <20050623064208.GE21363@esmail.cup.hp.com> <523br9oyul.fsf@topspin.com> Message-ID: <20050623161542.GE22934@esmail.cup.hp.com> On Thu, Jun 23, 2005 at 08:51:46AM -0700, Roland Dreier wrote: > Grant> On ia64 I expect both functions to land in the same bit of > Grant> code. For ZX1 platforms, that is sba_alloc_coherent(). > Grant> See arch/ia64/hp/common/sba_iommu.c since the behavior is > Grant> different depending on CONFIG_NUMA setting. > > Sure, but follow the code all the way through. ia64 uses > , which defines pci_alloc_consistent() > to be dma_alloc_coherent(..., GFP_ATOMIC). Ah ok - that's the bit I skipped over last night. > You're right that both > will end up in sba_alloc_coherent(), but the difference is the flags > parameter. GFP_ATOMIC allocations can easily fail even when the > system has plenty of pages available for GFP_KERNEL allocations. Ah! I had forgotten dma_alloc_coherent() takes a GFP_* flag parameter. nevermind... thanks, grant From halr at voltaire.com Thu Jun 23 09:14:53 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 12:14:53 -0400 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <52psudnjor.fsf@topspin.com> References: <1119539738.4617.2.camel@localhost.localdomain> <52psudnjor.fsf@topspin.com> Message-ID: <1119543293.4477.10.camel@localhost.localdomain> On Thu, 2005-06-23 at 12:04, Roland Dreier wrote: > Hal> Fix ib_pack/unpack for 64 bits > > Can you be more specific about what this is fixing? It fixes 64 bit sizes (at least on 32 bit machines). The problem was with the mask computation when size_bits = 64. mask = cpu_to_be64(((1ull << desc[i].size_bits) - 1) << shift); yielded a mask of 0 (for size_bits 64 and shift of 0). The 32 bit mask computation relies on the 33rd bit. In this case, shifting the 1 64 bits does not put it in the 65th bit which is needed for proper mask calculation. I may not have handled the shift part properly in my patch. -- Hal From roland at topspin.com Thu Jun 23 09:26:37 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 09:26:37 -0700 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <1119543293.4477.10.camel@localhost.localdomain> (Hal Rosenstock's message of "23 Jun 2005 12:14:53 -0400") References: <1119539738.4617.2.camel@localhost.localdomain> <52psudnjor.fsf@topspin.com> <1119543293.4477.10.camel@localhost.localdomain> Message-ID: <52aclhnio2.fsf@topspin.com> Hal> It fixes 64 bit sizes (at least on 32 bit machines). The Hal> problem was with the mask computation when size_bits = 64. Hal> mask = cpu_to_be64(((1ull << desc[i].size_bits) - 1) << shift); Hal> yielded a mask of 0 (for size_bits 64 and shift of 0). The 32 Hal> bit mask computation relies on the 33rd bit. In this case, Hal> shifting the 1 64 bits does not put it in the 65th bit which Hal> is needed for proper mask calculation. Hmm, is there anywhere that actually sets size_bits to 64? It's odd that you saw the expession end up as 0. It seems that on every system I have, 1ull << 64 is 0, and 0 - 1 == -1 == 0xffffffffffffffff. So although it isn't correct C, it should work. It is true that left shifting a 64-bit type by 64 bits is undefined according to standard C. I'll fix this up. - R. From mst at mellanox.co.il Thu Jun 23 09:40:02 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 23 Jun 2005 19:40:02 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <52k6klnjhz.fsf@topspin.com> References: <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> Message-ID: <20050623164002.GA12420@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] rdma_lat-09 and results > > Michael> Currently I want to be able to script linking mthca.a > Michael> statically. At least for this usage, I think it would be best to rename mthca.a (and mthca.so) to libmthca.a (and libmthca.so), respectively, and put them in libpath, then softlink libmthca.so to infiniband directory. This avoids the need for such utilities. Would you accept such a change? > Michael> In the future I can imagine someone might want to script > Michael> installing additional plugins into an existing system. > > I'm not sure this makes sense. libibverbs has a certain default path > but users can easily override it by setting the OPENIB_DRIVER_PATH > environment variable. For example, I often compile libmthca but don't > install it, and then point OPENIB_DRIVER_PATH at my source directory. Hmm, this would be a security problem if someone sets suid bit on programs using uverbs. > Is there any way we can use an existing tool like pkg-config to help > with this situation? > > - R. > Donnu, that would be distro-specific, wont it? E.g. perl and tcl do their own. -- MST From roland at topspin.com Thu Jun 23 09:41:09 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 09:41:09 -0700 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <52aclhnio2.fsf@topspin.com> (Roland Dreier's message of "Thu, 23 Jun 2005 09:26:37 -0700") References: <1119539738.4617.2.camel@localhost.localdomain> <52psudnjor.fsf@topspin.com> <1119543293.4477.10.camel@localhost.localdomain> <52aclhnio2.fsf@topspin.com> Message-ID: <521x6tnhzu.fsf@topspin.com> Roland> It's odd that you saw the expession end up as 0. It seems Roland> that on every system I have, 1ull << 64 is 0, and 0 - 1 == Roland> -1 == 0xffffffffffffffff. So although it isn't correct C, Roland> it should work. Oh, I see. A constant left shift of 64 is evaluated to 0 at compile time. But a variable left shift that ends up as 64 indeed gives the wrong answer. So you're right, packer will get the wrong answer. I think this should fix it properly: --- core/packer.c (revision 2665) +++ core/packer.c (working copy) @@ -96,7 +96,7 @@ void ib_pack(const struct ib_field else val = 0; - mask = cpu_to_be64(((1ull << desc[i].size_bits) - 1) << shift); + mask = cpu_to_be64((~0ull >> (64 - desc[i].size_bits)) << shift); addr = (__be64 *) ((__be32 *) buf + desc[i].offset_words); *addr = (*addr & ~mask) | (cpu_to_be64(val) & mask); } else { @@ -176,7 +176,7 @@ void ib_unpack(const struct ib_field __be64 *addr; shift = 64 - desc[i].offset_bits - desc[i].size_bits; - mask = ((1ull << desc[i].size_bits) - 1) << shift; + mask = (~0ull >> (64 - desc[i].size_bits)) << shift; addr = (__be64 *) buf + desc[i].offset_words; val = (be64_to_cpup(addr) & mask) >> shift; value_write(desc[i].struct_offset_bytes, From halr at voltaire.com Thu Jun 23 09:40:28 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 12:40:28 -0400 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <52aclhnio2.fsf@topspin.com> References: <1119539738.4617.2.camel@localhost.localdomain> <52psudnjor.fsf@topspin.com> <1119543293.4477.10.camel@localhost.localdomain> <52aclhnio2.fsf@topspin.com> Message-ID: <1119544826.4477.16.camel@localhost.localdomain> On Thu, 2005-06-23 at 12:26, Roland Dreier wrote: > Hal> It fixes 64 bit sizes (at least on 32 bit machines). The > Hal> problem was with the mask computation when size_bits = 64. > > Hal> mask = cpu_to_be64(((1ull << desc[i].size_bits) - 1) << shift); > > Hal> yielded a mask of 0 (for size_bits 64 and shift of 0). The 32 > Hal> bit mask computation relies on the 33rd bit. In this case, > Hal> shifting the 1 64 bits does not put it in the 65th bit which > Hal> is needed for proper mask calculation. > > Hmm, is there anywhere that actually sets size_bits to 64? Not currently but hopefully soon. > It's odd that you saw the expession end up as 0. It seems that on every > system I have, 1ull << 64 is 0, and 0 - 1 == -1 == 0xffffffffffffffff. > So although it isn't correct C, it should work. > > It is true that left shifting a 64-bit type by 64 bits is undefined > according to standard C. Maybe this is a compiler difference. > I'll fix this up. Thanks. -- Hal From mst at mellanox.co.il Thu Jun 23 09:45:55 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 23 Jun 2005 19:45:55 +0300 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <52aclhnio2.fsf@topspin.com> References: <1119539738.4617.2.camel@localhost.localdomain> <52psudnjor.fsf@topspin.com> <1119543293.4477.10.camel@localhost.localdomain> <52aclhnio2.fsf@topspin.com> Message-ID: <20050623164555.GB12420@mellanox.co.il> Quoting r. Roland Dreier : > It's odd that you saw the expession end up as 0. It seems that on every > system I have, 1ull << 64 is 0, and 0 - 1 == -1 == 0xffffffffffffffff. > So although it isn't correct C, it should work. It seems that compiler generates a single instruction for this shift. And CPU designers apparently have cut a corner by simply using only low 6 bits in the operand of a shift, so that 1ull<<64 is same as 1ull<<(64&0x37), or 1. -- MST From jlentini at netapp.com Thu Jun 23 09:49:30 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 12:49:30 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] remove NULL check for input params In-Reply-To: <20050622143635.GA24010@mellanox.co.il> References: <20050622143635.GA24010@mellanox.co.il> Message-ID: Committed in revision 2688 (minus the error Tom Duffy pointed out). On Wed, 22 Jun 2005, Itamar Rabenstein wrote: itamar> Apply This patch on top of the the previous patch set (5 patches) itamar> itamar> dapl_cr.c | 23 ---------- itamar> dapl_ep.c | 134 ------------------------------------------------------------- itamar> dapl_evd.c | 36 ---------------- itamar> dapl_ia.c | 24 ---------- itamar> dapl_lmr.c | 37 ---------------- itamar> dapl_pz.c | 36 ---------------- itamar> dapl_rmr.c | 29 ------------- itamar> dapl_sp.c | 86 --------------------------------------- itamar> dapl_srq.c | 58 -------------------------- itamar> 9 files changed, 4 insertions(+), 459 deletions(-) itamar> itamar> Delete Null input param check in dapl functions. itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_cr.c dat-provider/dapl_cr.c itamar> --- dat-provider_del_hca/dapl_cr.c Mon Jun 20 08:40:28 2005 itamar> +++ dat-provider/dapl_cr.c Wed Jun 22 11:37:50 2005 itamar> @@ -84,11 +84,6 @@ u32 dapl_cr_accept(struct dat_cr *dat_cr itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_accept (%p, %p, %d, %p)\n", itamar> dat_cr, dat_ep, priv_size, priv_data); itamar> itamar> - if (!dat_cr) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); itamar> - goto bail; itamar> - } itamar> - itamar> cr = (struct dapl_cr *)dat_cr; itamar> itamar> /* itamar> @@ -569,16 +564,6 @@ u32 dapl_cr_query(struct dat_cr *dat_cr, itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", itamar> dat_cr, cr_param); itamar> itamar> - if (dat_cr == NULL) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); itamar> - goto bail; itamar> - } itamar> - itamar> - if (NULL == cr_param) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> cr = (struct dapl_cr *)dat_cr; itamar> itamar> /* obtain the remote IP address */ itamar> @@ -590,7 +575,6 @@ u32 dapl_cr_query(struct dat_cr *dat_cr, itamar> itamar> status = DAT_SUCCESS; itamar> itamar> -bail: itamar> return status; itamar> } itamar> itamar> @@ -609,11 +593,6 @@ u32 dapl_cr_reject(struct dat_cr *dat_cr itamar> itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", dat_cr); itamar> itamar> - if (!dat_cr) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_CR); itamar> - goto bail; itamar> - } itamar> - itamar> cr = (struct dapl_cr *)dat_cr; itamar> itamar> /* itamar> @@ -654,7 +633,5 @@ u32 dapl_cr_reject(struct dat_cr *dat_cr itamar> itamar> dapl_cr_free(cr); itamar> } itamar> - itamar> -bail: itamar> return status; itamar> } itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_ep.c dat-provider/dapl_ep.c itamar> --- dat-provider_del_hca/dapl_ep.c Mon Jun 20 08:40:28 2005 itamar> +++ dat-provider/dapl_ep.c Wed Jun 22 11:38:46 2005 itamar> @@ -175,12 +175,6 @@ static u32 dapl_ep_post_send_req(struct itamar> struct dapl_cookie *cookie; itamar> u32 dat_status; itamar> itamar> - if (!ep) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> ep_ptr = (struct dapl_ep *)ep; itamar> itamar> /* itamar> @@ -301,15 +295,6 @@ u32 dapl_ep_create(struct dat_ia *ia, st itamar> ia_ptr = (struct dapl_ia *)ia; itamar> itamar> /* itamar> - * Verify parameters itamar> - */ itamar> - if (!ia_ptr) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - itamar> - /* itamar> * Verify non-required parameters. itamar> * N.B. Assumption: any parameter that can be itamar> * modified by dat_ep_modify() is not strictly itamar> @@ -342,15 +327,6 @@ u32 dapl_ep_create(struct dat_ia *ia, st itamar> goto bail; itamar> } itamar> itamar> - if (ep == NULL) { itamar> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); itamar> - goto bail; itamar> - } itamar> - if ((unsigned long)ep_attr & 3) { itamar> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG6); itamar> - goto bail; itamar> - } itamar> - itamar> /* itamar> * Qualify EP Attributes are legal and make sense. Note that if one itamar> * or both of the DTO handles are NULL, then the corresponding itamar> @@ -495,15 +471,6 @@ u32 dapl_ep_create_with_srq(struct dat_i itamar> ia_ptr = (struct dapl_ia *)ia; itamar> itamar> /* itamar> - * Verify parameters itamar> - */ itamar> - if (!ia_ptr) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - itamar> - /* itamar> * Verify non-required parameters. itamar> * N.B. Assumption: any parameter that can be itamar> * modified by dat_ep_modify() is not strictly itamar> @@ -537,25 +504,6 @@ u32 dapl_ep_create_with_srq(struct dat_i itamar> } itamar> itamar> /* itamar> - * Verify the SRQ handle. It is an error to invoke this call with itamar> - * a NULL handle itamar> - */ itamar> - if (!srq) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); itamar> - goto bail; itamar> - } itamar> - itamar> - if (ep == NULL) { itamar> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG7); itamar> - goto bail; itamar> - } itamar> - if ((unsigned long)ep_attr & 3) { itamar> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG6); itamar> - goto bail; itamar> - } itamar> - itamar> - /* itamar> * Qualify EP Attributes are legal and make sense. Note that if one itamar> * or both of the DTO handles are NULL, then the corresponding itamar> * max_*_dtos must 0 as the user will not be able to post dto ops on itamar> @@ -703,15 +651,6 @@ u32 dapl_ep_free(struct dat_ep *ep) itamar> ep_ptr = (struct dapl_ep *)ep; itamar> param = &ep_ptr->param; itamar> itamar> - /* itamar> - * Verify parameter & state itamar> - */ itamar> - if (!ep_ptr) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> if (ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED || itamar> ep_ptr->param.ep_state == DAT_EP_STATE_PASSIVE_CONNECTION_PENDING || itamar> ep_ptr->param.ep_state == DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) itamar> @@ -834,12 +773,6 @@ u32 dapl_ep_connect(struct dat_ep *ep, itamar> * Verify parameter & state. The connection handle must be good itamar> * at this point. itamar> */ itamar> - if (!ep_ptr) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> if (!ep_ptr->param.connect_evd) { itamar> dat_status = itamar> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); itamar> @@ -1036,16 +969,6 @@ u32 dapl_ep_dup_connect(struct dat_ep *e itamar> itamar> ep_dup_ptr = (struct dapl_ep *)ep_dup; itamar> itamar> - /* itamar> - * Verify the dup handle, which must be connected. All other itamar> - * parameters will be verified by dapl_ep_connect itamar> - */ itamar> - if (!ep_dup) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> /* Can't do a connection in 0 time, reject outright */ itamar> if (timeout == 0) { itamar> dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> @@ -1096,15 +1019,6 @@ u32 dapl_ep_disconnect(struct dat_ep *da itamar> ep = (struct dapl_ep *)dat_ep; itamar> itamar> /* itamar> - * Verify parameter & state itamar> - */ itamar> - if (!ep) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> - /* itamar> * Do the verification of parameters and the state change itamar> * atomically. itamar> */ itamar> @@ -1875,11 +1789,6 @@ u32 dapl_ep_post_recv(struct dat_ep *ep, itamar> ep, num_segments, local_iov, user_cookie.as_64, itamar> completion_flags); itamar> itamar> - if (!ep) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> ep_ptr = (struct dapl_ep *) ep; itamar> itamar> /* itamar> @@ -1950,19 +1859,6 @@ u32 dapl_ep_query(struct dat_ep *dat_ep, itamar> ep = (struct dapl_ep *)dat_ep; itamar> itamar> /* itamar> - * Verify parameter & state itamar> - */ itamar> - if (!ep) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> - if (ep_param == NULL) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> - /* itamar> * Fill in according to user request itamar> * itamar> * N.B. Just slam all values into the user structure, there itamar> @@ -1978,7 +1874,6 @@ u32 dapl_ep_query(struct dat_ep *dat_ep, itamar> (struct sockaddr *)&ep->remote_ia_address; itamar> *ep_param = ep->param; itamar> itamar> -bail: itamar> return status; itamar> } itamar> itamar> @@ -1993,18 +1888,8 @@ u32 dapl_ep_recv_query(struct dat_ep *ep itamar> itamar> ep_ptr = (struct dapl_ep *)ep; itamar> itamar> - /* itamar> - * Verify parameter & state itamar> - */ itamar> - if (!ep_ptr) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); itamar> itamar> -bail: itamar> return dat_status; itamar> itamar> } itamar> @@ -2016,15 +1901,6 @@ u32 dapl_ep_reset(struct dat_ep *ep) itamar> itamar> ep_ptr = (struct dapl_ep *)ep; itamar> itamar> - /* itamar> - * Verify parameter & state itamar> - */ itamar> - if (!ep_ptr) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED itamar> && ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECTED) { itamar> dat_status = itamar> @@ -2053,18 +1929,8 @@ u32 dapl_ep_set_watermark(struct dat_ep itamar> itamar> ep_ptr = (struct dapl_ep *)ep; itamar> itamar> - /* itamar> - * Verify parameter & state itamar> - */ itamar> - if (!ep_ptr) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - itamar> dat_status = DAT_NOT_IMPLEMENTED; itamar> itamar> -bail: itamar> return dat_status; itamar> itamar> } itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_evd.c dat-provider/dapl_evd.c itamar> --- dat-provider_del_hca/dapl_evd.c Mon Jun 20 08:40:28 2005 itamar> +++ dat-provider/dapl_evd.c Wed Jun 22 11:39:25 2005 itamar> @@ -1013,11 +1013,6 @@ u32 dapl_evd_kcreate(struct dat_ia *ia_h itamar> evd = NULL; itamar> *evd_handle = NULL; itamar> itamar> - if (!ia_handle) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - itamar> if (min_qlen <= 0) { itamar> status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); itamar> goto bail; itamar> @@ -1060,11 +1055,6 @@ u32 dapl_evd_free(struct dat_evd *evd_ha itamar> itamar> evd = (struct dapl_evd *)evd_handle; itamar> itamar> - if (!evd_handle) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, 0); itamar> - goto bail; itamar> - } itamar> - itamar> if (atomic_read(&evd->evd_ref_count) != 0) { itamar> status = itamar> DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_EVD_IN_USE); itamar> @@ -1090,11 +1080,6 @@ u32 dapl_evd_kquery(struct dat_evd *evd_ itamar> struct dapl_evd *evd; itamar> u32 status = DAT_SUCCESS; itamar> itamar> - if (NULL == evd_param) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> /* Note: the spec. allows for events to be directed to a NULL EVD */ itamar> /* with handle of type NULL. See 6.3.1 */ itamar> if (NULL == evd_handle) { itamar> @@ -1138,15 +1123,9 @@ u32 dapl_evd_modify_upcall(struct dat_ev itamar> itamar> evd = (struct dapl_evd *)evd_handle; itamar> itamar> - if (!evd_handle) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, 0); itamar> - goto bail; itamar> - } itamar> - itamar> evd->upcall_policy = upcall_policy; itamar> evd->upcall = *upcall; itamar> itamar> -bail: itamar> return status; itamar> } itamar> itamar> @@ -1195,16 +1174,6 @@ u32 dapl_evd_dequeue(struct dat_evd *evd itamar> itamar> evd = (struct dapl_evd *)evd_handle; itamar> itamar> - if (!evd_handle) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, 0); itamar> - goto bail; itamar> - } itamar> - itamar> - if (event == NULL) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); itamar> - goto bail; itamar> - } itamar> - itamar> /* itamar> * We need to dequeue under lock, as the IB OS Access API itamar> * restricts us from having multiple threads in CQ poll, and the itamar> @@ -1255,11 +1224,6 @@ u32 dapl_evd_resize(struct dat_evd *evd_ itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", itamar> evd_handle, evd_qlen); itamar> itamar> - if (!evd_handle) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, 0); itamar> - goto bail; itamar> - } itamar> - itamar> evd = (struct dapl_evd *)evd_handle; itamar> ia = evd->common.owner_ia; itamar> itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_ia.c dat-provider/dapl_ia.c itamar> --- dat-provider_del_hca/dapl_ia.c Mon Jun 20 08:40:28 2005 itamar> +++ dat-provider/dapl_ia.c Wed Jun 22 11:40:02 2005 itamar> @@ -611,16 +611,6 @@ u32 dapl_ia_open(const char *name, int a itamar> goto bail; itamar> } itamar> itamar> - /* ia_ptr and async_evd cannot be NULL */ itamar> - if (ia_ptr == NULL) { itamar> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); itamar> - goto bail; itamar> - } itamar> - if (async_evd == NULL) { itamar> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> /* initialize the caller's param */ itamar> *ia_ptr = NULL; itamar> itamar> @@ -710,12 +700,6 @@ u32 dapl_ia_close(struct dat_ia *ia_ptr, itamar> itamar> ia = (struct dapl_ia *)ia_ptr; itamar> itamar> - if (!ia) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - itamar> if (DAT_CLOSE_ABRUPT_FLAG == ia_flags) itamar> dat_status = dapl_ia_abrupt_close(ia); itamar> else if (DAT_CLOSE_GRACEFUL_FLAG == ia_flags) itamar> @@ -723,7 +707,6 @@ u32 dapl_ia_close(struct dat_ia *ia_ptr, itamar> else itamar> dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); itamar> itamar> -bail: itamar> return dat_status; itamar> } itamar> itamar> @@ -783,12 +766,6 @@ u32 dapl_ia_query(struct dat_ia *ia_ptr, itamar> itamar> ia = (struct dapl_ia *)ia_ptr; itamar> itamar> - if (!ia) { itamar> - dat_status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - itamar> if (NULL != async_evd) { itamar> *async_evd = (struct dat_evd *)ia->async_error_evd; itamar> } itamar> @@ -862,7 +839,6 @@ u32 dapl_ia_query(struct dat_ia *ia_ptr, itamar> #endif /* DAPL_MERGE_CM_DTO */ itamar> } itamar> itamar> -bail: itamar> dapl_dbg_log(DAPL_DBG_TYPE_RTN, itamar> "dapl_ia_query () returns 0x%x\n", dat_status); itamar> itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_lmr.c dat-provider/dapl_lmr.c itamar> --- dat-provider_del_hca/dapl_lmr.c Mon Jun 20 08:40:28 2005 itamar> +++ dat-provider/dapl_lmr.c Wed Jun 22 11:45:13 2005 itamar> @@ -241,15 +241,6 @@ u32 dapl_lmr_kcreate(struct dat_ia *ia, itamar> "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", itamar> ia, mem_type); itamar> itamar> - if (!ia) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - if (!pz) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); itamar> - goto bail; itamar> - } itamar> - itamar> dapl_ia = (struct dapl_ia *)ia; itamar> dapl_pz = (struct dapl_pz *)pz; itamar> itamar> @@ -309,11 +300,6 @@ u32 dapl_lmr_free(struct dat_lmr *lmr) itamar> itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr); itamar> itamar> - if (!lmr) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); itamar> - goto error; itamar> - } itamar> - itamar> dapl_lmr = (struct dapl_lmr *)lmr; itamar> itamar> switch (dapl_lmr->param.mem_type) { itamar> @@ -343,7 +329,6 @@ u32 dapl_lmr_free(struct dat_lmr *lmr) itamar> status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); itamar> break; itamar> } itamar> -error: itamar> return status; itamar> } itamar> itamar> @@ -355,19 +340,9 @@ u32 dapl_lmr_query(struct dat_lmr *lmr, itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", itamar> lmr, lmr_param); itamar> itamar> - if (!lmr) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); itamar> - goto error; itamar> - } itamar> - if (NULL == lmr_param) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto error; itamar> - } itamar> - itamar> dapl_lmr = (struct dapl_lmr *)lmr; itamar> memcpy(lmr_param, &dapl_lmr->param, sizeof *lmr_param); itamar> status = DAT_SUCCESS; itamar> -error: itamar> return status; itamar> } itamar> itamar> @@ -388,12 +363,6 @@ u32 dapl_lmr_sync_rdma_read(struct dat_i itamar> itamar> ia_ptr = (struct dapl_ia *)ia; itamar> itamar> - if (!ia_ptr) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto error; itamar> - } itamar> - itamar> -error: itamar> return status; itamar> } itamar> itamar> @@ -409,11 +378,5 @@ u32 dapl_lmr_sync_rdma_write(struct dat_ itamar> itamar> ia_ptr = (struct dapl_ia *)ia; itamar> itamar> - if (!ia_ptr) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto error; itamar> - } itamar> - itamar> -error: itamar> return status; itamar> } itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_pz.c dat-provider/dapl_pz.c itamar> --- dat-provider_del_hca/dapl_pz.c Mon Jun 20 08:40:28 2005 itamar> +++ dat-provider/dapl_pz.c Wed Jun 22 11:41:13 2005 itamar> @@ -75,40 +75,23 @@ u32 dapl_pz_create(struct dat_ia *ia, st itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create(%p, %p)\n", itamar> ia, pz); itamar> itamar> - if (!ia) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto error1; itamar> - } itamar> - if (NULL == pz) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); itamar> - goto error1; itamar> - } itamar> - itamar> dapl_ia = (struct dapl_ia *)ia; itamar> itamar> dapl_pz = dapl_pz_alloc(dapl_ia); itamar> - if (!dapl_pz) { itamar> - status = itamar> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); itamar> - goto error1; itamar> - } itamar> - itamar> dapl_pz->pd = ib_alloc_pd(dapl_ia->hca->ib_hca_handle); itamar> if (IS_ERR(dapl_pz->pd)) { itamar> ib_status = PTR_ERR(dapl_pz->pd); itamar> dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_alloc_pd failed: %X\n", itamar> ib_status); itamar> status = dapl_ib_status_convert(ib_status); itamar> - goto error2; itamar> + goto error; itamar> } itamar> itamar> *pz = (struct dat_pz *)dapl_pz; itamar> return DAT_SUCCESS; itamar> itamar> -error2: itamar> +error: itamar> dapl_pz_dealloc(dapl_pz); itamar> - itamar> -error1: itamar> *pz = NULL; itamar> return status; itamar> } itamar> @@ -121,11 +104,6 @@ u32 dapl_pz_free(struct dat_pz *pz) itamar> itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free(%p)\n", pz); itamar> itamar> - if (!pz) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); itamar> - goto error; itamar> - } itamar> - itamar> dapl_pz = (struct dapl_pz *)pz; itamar> itamar> if (0 != atomic_read(&dapl_pz->pz_ref_count)) { itamar> @@ -157,19 +135,9 @@ u32 dapl_pz_query(struct dat_pz *pz, str itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query(%p, %x, %p)\n", itamar> pz, pz_param); itamar> itamar> - if (!pz) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); itamar> - goto error; itamar> - } itamar> - if (NULL == pz_param) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto error; itamar> - } itamar> - itamar> dapl_pz = (struct dapl_pz *)pz; itamar> pz_param->ia = (struct dat_ia *)dapl_pz->common.owner_ia; itamar> return DAT_SUCCESS; itamar> itamar> -error: itamar> return status; itamar> } itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_rmr.c dat-provider/dapl_rmr.c itamar> --- dat-provider_del_hca/dapl_rmr.c Mon Jun 20 08:40:28 2005 itamar> +++ dat-provider/dapl_rmr.c Wed Jun 22 11:41:36 2005 itamar> @@ -354,15 +354,6 @@ u32 dapl_rmr_bind(struct dat_rmr *rmr_ha itamar> struct dapl_lmr *lmr; itamar> struct dapl_ep *ep_ptr; itamar> itamar> - if (!rmr_handle) itamar> - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); itamar> - itamar> - if (!lmr_handle) itamar> - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); itamar> - itamar> - if (!ep) itamar> - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - itamar> rmr = (struct dapl_rmr *)rmr_handle; itamar> lmr = (struct dapl_lmr *)lmr_handle; itamar> ep_ptr = (struct dapl_ep *)ep; itamar> @@ -399,11 +390,6 @@ u32 dapl_rmr_create(struct dat_pz *pz, s itamar> struct dapl_rmr *dapl_rmr; itamar> u32 status = DAT_SUCCESS; itamar> itamar> - if (!pz) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); itamar> - goto bail; itamar> - } itamar> - itamar> dapl_pz = (struct dapl_pz *)pz; itamar> itamar> dapl_rmr = dapl_rmr_alloc(dapl_pz); itamar> @@ -451,11 +437,6 @@ u32 dapl_rmr_free(struct dat_rmr *rmr_ha itamar> struct dapl_rmr *rmr; itamar> u32 status = DAT_SUCCESS; itamar> itamar> - if (!rmr_handle) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); itamar> - goto bail; itamar> - } itamar> - itamar> rmr = (struct dapl_rmr *)rmr_handle; itamar> itamar> /* itamar> @@ -485,19 +466,9 @@ u32 dapl_rmr_query(struct dat_rmr *rmr_h itamar> struct dapl_rmr *rmr; itamar> u32 status = DAT_SUCCESS; itamar> itamar> - if (!rmr_handle) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); itamar> - goto bail; itamar> - } itamar> - if (NULL == rmr_param) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> rmr = (struct dapl_rmr *)rmr_handle; itamar> itamar> memcpy(rmr_param, &rmr->param, sizeof *rmr_param); itamar> itamar> -bail: itamar> return status; itamar> } itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_sp.c dat-provider/dapl_sp.c itamar> --- dat-provider_del_hca/dapl_sp.c Mon Jun 20 08:40:28 2005 itamar> +++ dat-provider/dapl_sp.c Wed Jun 22 11:41:57 2005 itamar> @@ -145,25 +145,6 @@ u32 dapl_rsp_create(struct dat_ia *dat_i itamar> ">>> dapl_rsp_free conn_qual: %x EP: %p\n", itamar> conn_qual, dat_ep); itamar> itamar> - if (!dat_ia) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - if (!dat_ep) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); itamar> - goto bail; itamar> - } itamar> - if (!dat_evd) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, itamar> - DAT_INVALID_HANDLE_EVD_CR); itamar> - goto bail; itamar> - } itamar> - itamar> - if (rsp == NULL) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); itamar> - goto bail; itamar> - } itamar> - itamar> ia = (struct dapl_ia *)dat_ia; itamar> itamar> ep = (struct dapl_ep *)dat_ep; itamar> @@ -263,14 +244,8 @@ u32 dapl_rsp_free(struct dat_sp *dat_rsp itamar> u32 status = DAT_SUCCESS; itamar> itamar> sp = (struct dapl_sp *)dat_rsp; itamar> - /* itamar> - * Verify handle itamar> - */ itamar> + itamar> dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", dat_rsp); itamar> - if (!sp) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); itamar> - goto bail; itamar> - } itamar> itamar> ia = sp->common.owner_ia; itamar> itamar> @@ -341,16 +316,6 @@ u32 dapl_rsp_query(struct dat_sp *dat_rs itamar> struct dapl_sp *sp; itamar> u32 status; itamar> itamar> - if (!dat_rsp) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); itamar> - goto bail; itamar> - } itamar> - itamar> - if (NULL == rsp_param) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> sp = (struct dapl_sp *)dat_rsp; itamar> itamar> /* itamar> @@ -363,7 +328,6 @@ u32 dapl_rsp_query(struct dat_sp *dat_rs itamar> itamar> status = DAT_SUCCESS; itamar> itamar> -bail: itamar> return status; itamar> } itamar> itamar> @@ -390,25 +354,6 @@ u32 dapl_psp_create_any(struct dat_ia *d itamar> "dapl_psp_create_anny(%p, %p, %p, %X, %p)\n", itamar> dat_ia, conn_qual, dat_evd, psp_flags, dat_psp); itamar> itamar> - if (!dat_ia) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - if (!dat_evd) { itamar> - status = itamar> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); itamar> - goto bail; itamar> - } itamar> - itamar> - if (!dat_psp) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); itamar> - goto bail; itamar> - } itamar> - if (conn_qual == NULL) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); itamar> - goto bail; itamar> - } itamar> - itamar> ia = (struct dapl_ia *)dat_ia; itamar> evd = (struct dapl_evd *)dat_evd; itamar> if (!(evd->evd_flags & DAT_EVD_CR_FLAG)) { itamar> @@ -510,21 +455,6 @@ u32 dapl_psp_create(struct dat_ia *dat_i itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_psp_create(%p, %X, %p, %X, %p)\n", itamar> dat_ia, conn_qual, dat_evd, psp_flags, dat_psp); itamar> itamar> - if (!dat_ia) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - if (!dat_evd) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, itamar> - DAT_INVALID_HANDLE_EVD_CR); itamar> - goto bail; itamar> - } itamar> - itamar> - if (!dat_psp) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); itamar> - goto bail; itamar> - } itamar> - itamar> ia = (struct dapl_ia *)dat_ia; itamar> evd = (struct dapl_evd *)dat_evd; itamar> if (!(evd->evd_flags & DAT_EVD_CR_FLAG)) { itamar> @@ -628,15 +558,8 @@ u32 dapl_psp_free(struct dat_sp *dat_psp itamar> u32 status = DAT_SUCCESS; itamar> itamar> sp = (struct dapl_sp *)dat_psp; itamar> - /* itamar> - * Verify handle itamar> - */ itamar> - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", dat_psp); itamar> itamar> - if (!sp) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); itamar> - goto bail; itamar> - } itamar> + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", dat_psp); itamar> itamar> ia = sp->common.owner_ia; itamar> /* itamar> @@ -712,11 +635,6 @@ u32 dapl_psp_query(struct dat_sp *dat_ps itamar> goto bail; itamar> } itamar> itamar> - if (NULL == psp_param) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> sp = (struct dapl_sp *)dat_psp; itamar> itamar> psp_param->ia = sp->ia; itamar> diff -Nurp -X dontdiff dat-provider_del_hca/dapl_srq.c dat-provider/dapl_srq.c itamar> --- dat-provider_del_hca/dapl_srq.c Mon Jun 20 08:40:27 2005 itamar> +++ dat-provider/dapl_srq.c Wed Jun 22 11:42:23 2005 itamar> @@ -149,30 +149,6 @@ u32 dapl_srq_create(struct dat_ia *ia, s itamar> itamar> ia_ptr = (struct dapl_ia *)ia; itamar> itamar> - /* itamar> - * Verify parameters itamar> - */ itamar> - if (!ia_ptr) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); itamar> - goto bail; itamar> - } itamar> - itamar> - /* itamar> - * Verify non-required parameters. itamar> - * N.B. Assumption: any parameter that can be itamar> - * modified by dat_ep_modify() is not strictly itamar> - * required when the EP is created itamar> - */ itamar> - if (srq == NULL) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); itamar> - goto bail; itamar> - } itamar> - if ((unsigned long)srq_attr & 3) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> - /* Allocate SRQ */ itamar> srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); itamar> if (srq_ptr == NULL) { itamar> status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, itamar> @@ -228,14 +204,6 @@ u32 dapl_srq_free(struct dat_srq *srq) itamar> srq_ptr = (struct dapl_srq *)srq; itamar> param = &srq_ptr->param; itamar> itamar> - /* itamar> - * Verify parameter & state itamar> - */ itamar> - if (!srq_ptr) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); itamar> - goto bail; itamar> - } itamar> - itamar> if (atomic_read(&srq_ptr->srq_ref_count) != 0) { itamar> /* itamar> * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does itamar> @@ -313,11 +281,6 @@ u32 dapl_srq_post_recv(struct dat_srq *s itamar> "dapl_srq_post_recv (%p, %d, %p, %P)\n", itamar> srq, num_segments, local_iov, user_cookie.as_64); itamar> itamar> - if (!srq) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); itamar> - goto bail; itamar> - } itamar> - itamar> srq_ptr = (struct dapl_srq *)srq; itamar> itamar> /* itamar> @@ -360,15 +323,6 @@ u32 dapl_srq_query(struct dat_srq *srq, itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", srq, itamar> srq_param); itamar> itamar> - if (!srq) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); itamar> - goto bail; itamar> - } itamar> - if (srq_param == NULL) { itamar> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); itamar> - goto bail; itamar> - } itamar> - itamar> srq_ptr = (struct dapl_srq *)srq; itamar> itamar> /* itamar> @@ -379,7 +333,6 @@ u32 dapl_srq_query(struct dat_srq *srq, itamar> itamar> *srq_param = srq_ptr->param; itamar> itamar> -bail: itamar> return status; itamar> } itamar> itamar> @@ -412,11 +365,6 @@ u32 dapl_srq_resize(struct dat_srq *srq, itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", itamar> srq, srq_max_recv_dto); itamar> itamar> - if (!srq) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); itamar> - goto bail; itamar> - } itamar> - itamar> srq_ptr = (struct dapl_srq *)srq; itamar> ia_ptr = srq_ptr->common.owner_ia; itamar> itamar> @@ -466,17 +414,11 @@ u32 dapl_srq_set_lw(struct dat_srq *srq, itamar> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq, itamar> low_watermark); itamar> itamar> - if (!srq) { itamar> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); itamar> - goto bail; itamar> - } itamar> - itamar> srq_ptr = (struct dapl_srq *)srq; itamar> itamar> /* XXX Put implementation here XXX */ itamar> itamar> /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); itamar> itamar> -bail: itamar> return status; itamar> } itamar> -- itamar> Itamar itamar> From Thomas.Talpey at netapp.com Thu Jun 23 09:52:27 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Thu, 23 Jun 2005 12:52:27 -0400 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <42B99546.2070703@linuxmachines.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> Message-ID: <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> At 12:43 PM 6/22/2005, Jeff Carr wrote: >On 06/21/2005 12:50 PM, Roland Dreier wrote: > >> What happens if you try replacing the send_flags line with the one you >> have commented out? >> >> + // send_wr.send_flags = IB_SEND_SIGNALED; > >Thanks, you are correct. IB_SEND_SIGNALED gives me the behavior I was >expecting. By the way, unsignaled sends can work very well indeed, but you have to be sure to poll for completions at regular intervals. Basically, what you're doing is ensuring that software (mthca) gets control from time to time, either via an interrupt (signaled) or poll (unsignaled). It's quite a challenge to get the polling right, but the reduction in interrupts can be a win. The NFS/RDMA module does this, but it takes the approach of "occasionally" posting a signaled send. The trick is getting the value of "occasionally" right. :-) Tom. From roland at topspin.com Thu Jun 23 10:06:08 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 10:06:08 -0700 Subject: [openib-general] Re: [PATCH] ib_verbs.h ib_send_wr imm_data endianness In-Reply-To: <20050622141238.GC20041@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 22 Jun 2005 17:12:38 +0300") References: <20050622141238.GC20041@mellanox.co.il> Message-ID: <52psudm29r.fsf@topspin.com> Michael> Mark imm_data as big endian in ib_send_wr, same as we do Michael> in ib_wc. Thanks, applied. Have you tried using -Wbitwise when building drivers/infiniband? How bad is it? - R. From jcarr at linuxmachines.com Thu Jun 23 10:11:47 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 23 Jun 2005 10:11:47 -0700 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> Message-ID: <42BAED53.80206@linuxmachines.com> On 06/23/2005 09:52 AM, Talpey, Thomas wrote: > At 12:43 PM 6/22/2005, Jeff Carr wrote: > >>On 06/21/2005 12:50 PM, Roland Dreier wrote: >> >> >>>What happens if you try replacing the send_flags line with the one you >>>have commented out? >>> >>>+ // send_wr.send_flags = IB_SEND_SIGNALED; >> >>Thanks, you are correct. IB_SEND_SIGNALED gives me the behavior I was >>expecting. > > > By the way, unsignaled sends can work very well indeed, but you have to be > sure to poll for completions at regular intervals. Basically, what you're doing > is ensuring that software (mthca) gets control from time to time, either via > an interrupt (signaled) or poll (unsignaled). > > It's quite a challenge to get the polling right, but the reduction in interrupts > can be a win. The NFS/RDMA module does this, but it takes the approach I didn't know there was a nfs/rmda module? root at jcarr:/test/gen2# find . |grep -i nfs root at jcarr:/test/gen2# > of "occasionally" posting a signaled send. The trick is getting the value of > "occasionally" right. :-) Yep, I understand the tuning problems; that is what I need. Jeff From jlentini at netapp.com Thu Jun 23 10:12:58 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 13:12:58 -0400 (EDT) Subject: [openib-general] mapping between IP address and device name In-Reply-To: <1119542588.4477.2.camel@localhost.localdomain> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> Message-ID: On Thu, 23 Jun 2005, Hal Rosenstock wrote: > On Thu, 2005-06-23 at 11:47, Roland Dreier wrote: >> James> The IBAT API provides the two key services needed by kDAPL: >> James> the ability to obtain a route based on an IP address and >> James> the ability to map a GID to an IP address. >> >> James> Is there agreement that an IB address translation service >> James> must provide these two services? >> >> I think the first service is reasonable, although one has to be >> careful to design it so it works with things like IPv6 link-local >> addresses. The second service, mapping a GID to an IP address, >> doesn't make sense to me: for a given GID, there may be no associated >> IPoIB address, or there may be many IPoIB addresses -- in fact, it is >> difficult to configure an IPoIB interface so that it doesn't have both >> IPv4 and IPv6 addresses. > > In terms of mapping a GID to an IP address, the administrator can choose > whether to run IPoIB or not and how it is configured. > > In the case of multiple addresses, the set of addresses is returned. > In the case of no associated addresses, then an error is returned. I Perhaps a bit of motivation of how the GID->IP service can be used is in order. kDAPL uses this feature to provide the passive side of a connection with the IP address of the remote peer. kDAPL consumers can use this information as a weak authentication mechanism. For example, an NFS server can be configured to export file systems to certain client systems. An NFS-RDMA server using kDAPL will use the GID-IP service to implement this feature. Could SDP make use of this service to validate a connection request's source IP address? From roland at topspin.com Thu Jun 23 10:21:23 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 10:21:23 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050623164002.GA12420@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 23 Jun 2005 19:40:02 +0300") References: <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> Message-ID: <52k6klm1kc.fsf@topspin.com> Michael> At least for this usage, I think it would be best to Michael> rename mthca.a (and mthca.so) to libmthca.a (and Michael> libmthca.so), respectively, and put them in libpath, then Michael> softlink libmthca.so to infiniband directory. Michael> This avoids the need for such utilities. Would you Michael> accept such a change? That makes sense, although I don't see why we want libmthca.so in the regular library directory. It seems like the best would be to have libmthca.a in $(libdir) and mthca.so in $(libdir)/infiniband. That way we make it easy to do static linking and avoid having two names for the same object. Michael> Hmm, this would be a security problem if someone sets Michael> suid bit on programs using uverbs. True, good point. To me being able to set the search path is much more useful than being able to have secure suid verbs programs, so I don't want to get rid of the OPENIB_DRIVER_PATH. Does something like the following seem good enough? --- libibverbs/src/init.c (revision 2656) +++ libibverbs/src/init.c (working copy) @@ -41,6 +41,8 @@ #include #include #include +#include +#include #include "ibverbs.h" @@ -207,11 +209,17 @@ static void INIT ibverbs_init(void) */ load_driver(NULL); - user_path = getenv(OPENIB_DRIVER_PATH_ENV); - if (user_path) { - wr_path = strdupa(user_path); - while ((dir = strsep(&wr_path, ";:"))) - find_drivers(dir); + /* + * Only follow the path passed in through the calling user's + * environment if we're not running SUID. + */ + if (getuid() == geteuid()) { + user_path = getenv(OPENIB_DRIVER_PATH_ENV); + if (user_path) { + wr_path = strdupa(user_path); + while ((dir = strsep(&wr_path, ";:"))) + find_drivers(dir); + } } find_drivers(default_path); From mst at mellanox.co.il Thu Jun 23 10:22:33 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 23 Jun 2005 20:22:33 +0300 Subject: [openib-general] Re: [PATCH] ib_verbs.h ib_send_wr imm_data endianness In-Reply-To: <52psudm29r.fsf@topspin.com> References: <20050622141238.GC20041@mellanox.co.il> <52psudm29r.fsf@topspin.com> Message-ID: <20050623172233.GC12420@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] ib_verbs.h ib_send_wr imm_data endianness > > Michael> Mark imm_data as big endian in ib_send_wr, same as we do > Michael> in ib_wc. > > Thanks, applied. > > Have you tried using -Wbitwise when building drivers/infiniband? How > bad is it? > > - R. > Not, yet. I'm reading up a bit on sparse now. BTW, from what I'm reading it seems we need to mark flag enums with the bitwise flag. But we'll see about that. -- MST From roland at topspin.com Thu Jun 23 10:31:30 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 10:31:30 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: (James Lentini's message of "Thu, 23 Jun 2005 13:12:58 -0400 (EDT)") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> Message-ID: <52ekatm13h.fsf@topspin.com> James> Perhaps a bit of motivation of how the GID->IP service can James> be used is in order. James> kDAPL uses this feature to provide the passive side of a James> connection with the IP address of the remote peer. kDAPL James> consumers can use this information as a weak authentication James> mechanism. This seems so weak as to be not useful, and rather expensive to boot. To implement this, a system receiving a connection request would have to perform an SA query to map the remote LID back to a GuidInfo record, and then for each GID attached to the remote LID, somehow retrieve the set of IP addresses configured for that GID (assuming that is somehow even possible). James> Could SDP make use of this service to validate a connection James> request's source IP address? No, SDP passes the remote peer's IP address directly as part of its connection establishment. In fact, the SDP annex in the IBA spec contains this rather enlightening passage: IP over InfiniBand does not define a mechanism to perform an inverse lookup (from an InfiniBand address to an IP address). It is also possible for a single InfiniBand address to have many IP addresses, providing a one-to-many mapping when attempting to perform an inverse lookup. To resolve these issues, the complete source and destination IP address is provided during connection setup to enable mapping the destination and source LID/GID to an IP address at the accepting peer of the connection. - R. From tduffy at sun.com Thu Jun 23 10:33:35 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 23 Jun 2005 10:33:35 -0700 Subject: [openib-general] [PATCH] kDAPL: use Grant's suggestions In-Reply-To: <20050623062020.GB21363@esmail.cup.hp.com> References: <1119460629.26537.21.camel@duffman> <20050623062020.GB21363@esmail.cup.hp.com> Message-ID: <1119548015.9388.2.camel@duffman> On Wed, 2005-06-22 at 23:20 -0700, Grant Grundler wrote: > On Wed, Jun 22, 2005 at 10:17:09AM -0700, Tom Duffy wrote: > > This patch removes the function dapl_os_panic() in favor of calling > > panic() directly. > > just some nits... > > > if (!evd) > > - dapl_os_panic("NULL == context\n"); > > + panic("NULL == context\n"); > > Isn't this a bit silly? > > > async_evd = evd->common.owner_ia->async_error_evd; > > The system is going to panic anyway here if evd is null. > I don't see any advantage to testing evd if the only action is to panic. > > > - if (NULL == context) { > > - dapl_os_panic("NULL == context\n"); > > - return; > > - } > > + if (NULL == context) > > + panic("NULL == context\n"); > > Would this be better as "BUG_ON(NULL == context)"? I agree with both points. Here is a new patch based off of r2689. Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2689) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -680,9 +680,6 @@ static void dapl_evd_cq_async_error_call "dapl_evd_cq_async_error_callback (%p, %p)\n", cause, context); - if (!evd) - panic("NULL == context\n"); - async_evd = evd->common.owner_ia->async_error_evd; status = dapl_evd_post_async_error_event(async_evd, @@ -706,8 +703,7 @@ void dapl_evd_un_async_error_callback(st "dapl_evd_un_async_error_callback (%p, %p)\n", cause, context); - if (NULL == context) - panic("NULL == context\n"); + BUG_ON(context == NULL); async_evd = (struct dapl_evd *)context; From ftillier at silverstorm.com Thu Jun 23 10:43:06 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 23 Jun 2005 10:43:06 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52ekatm13h.fsf@topspin.com> Message-ID: <000101c5781b$040936d0$9c5aa8c0@infiniconsys.com> > From: Roland Dreier [mailto:roland at topspin.com] > Sent: Thursday, June 23, 2005 10:32 AM > > James> Perhaps a bit of motivation of how the GID->IP service can > James> be used is in order. > > James> kDAPL uses this feature to provide the passive side of a > James> connection with the IP address of the remote peer. kDAPL > James> consumers can use this information as a weak authentication > James> mechanism. > > This seems so weak as to be not useful, and rather expensive to boot. > To implement this, a system receiving a connection request would have > to perform an SA query to map the remote LID back to a GuidInfo > record, and then for each GID attached to the remote LID, somehow > retrieve the set of IP addresses configured for that GID (assuming > that is somehow even possible). This reverse lookup was something that I worked to accommodate in my proposed changes to expand DAPL ATS to support multiple IP addresses. The revised DAPL ATS proposal establishes the notion of a primary IP address that would be used for such validation. However, I still think the reverse lookup (GID->IP) is weak as there is no way to tell which IP the source really used. IMO it would be much better to put the source and destination addresses into the CM private data, but this supposedly creates a wire protocol which the DAT collaborative wants to avoid at all costs. - Fab From jjengla at sandia.gov Thu Jun 23 10:45:27 2005 From: jjengla at sandia.gov (Josh England) Date: Thu, 23 Jun 2005 10:45:27 -0700 Subject: [openib-general] RDMA write with immediate In-Reply-To: <1119531169.4481.15.camel@localhost.localdomain> References: <1119480693.24922.33.camel@localhost> <52k6klq184.fsf@topspin.com> <1119530877.29406.29.camel@localhost> <1119531169.4481.15.camel@localhost.localdomain> Message-ID: <1119548727.31016.2.camel@localhost> On Thu, 2005-06-23 at 08:56 -0400, Hal Rosenstock wrote: > On Thu, 2005-06-23 at 08:47, Josh England wrote: > > Alright, I'm probably being retarded, but just in case: > > uverbs doesn't seem to work anymore. Simple pingpong uverbs test gives: > > > > Couldn't get context for mthca0 > > > > This is with rev 2685 on 2.6.12.1 under RHEL3. Fresh rebuild/install of > > libibverbs, libmthca, and libibcm. Ports are ACTIVE and ipoib ping > > works. > > > > # lsmod > > ib_ucm 15448 0 > > ib_cm 33904 1 ib_ucm > > ib_at 13704 0 > > ib_uverbs 24088 0 > > ib_ipoib 38920 0 > > ib_sa 11660 2 ib_at,ib_ipoib > > ib_mthca 90768 0 > > ib_mad 38056 3 ib_cm,ib_sa,ib_mthca > > ib_core 41088 6 > > ib_cm,ib_uverbs,ib_ipoib,ib_sa,ib_mthca,ib_mad > > > > # ll /dev/infiniband > > crw-r--r-- 1 root root 231, 255 Jun 23 05:31 ucm > > crw-rw-rw- 1 root root 231, 128 Jun 23 05:31 uverbs0 > > crw-rw-rw- 1 root root 231, 129 Jun 23 05:31 uverbs1 > > > > > > Did I miss something? I think this is the same environment that was > > working before. > > Are you sure this is 2685 ? As of 2658, I think uverbs now starts at > minor 192 rather than 128. That's probably it. I'm not using udev for this testing, and had done mknods by hand more or less. I'll try the new minor number. -JE > 2658 > Modified: > gen2/trunk/src/linux-kernel/infiniband/core/uverbs_main.c > Log: > Fix order of cleanup in module unload. Move first minor from 128 to > 192. > -- Hal > > > > > -JE > > > > On Wed, 2005-06-22 at 19:02 -0700, Roland Dreier wrote: > > > Josh> I'm trying to figure out how to do an RDMA write with > > > Josh> immediate data. I've got a regular RDMA write working and > > > Josh> tried changing wr.opcode from IBV_WR_RDMA_WRITE to > > > Josh> IBV_WR_RDMA_WRITE_WITH_IMM and stuffing a value in > > > Josh> wr.imm_data to no avail. The send fails. > > > > > > This should work, and in fact I tested it a while ago, although it may > > > have broken in the meantime. > > > > > > When you say that your send fails, what do you mean? Does > > > ibv_post_send() return immediately with an error? Or do you get a > > > completion for your send request with a failed status? If so, what is > > > the status? Or does the send just never complete and/or never show up > > > on the remote system? > > > > > > - R. > > > > > > > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From Thomas.Talpey at netapp.com Thu Jun 23 10:54:17 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Thu, 23 Jun 2005 13:54:17 -0400 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <42BAED53.80206@linuxmachines.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> <42BAED53.80206@linuxmachines.com> Message-ID: <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> At 01:11 PM 6/23/2005, Jeff Carr wrote: >I didn't know there was a nfs/rmda module? > >root at jcarr:/test/gen2# find . |grep -i nfs >root at jcarr:/test/gen2# Brief intro to NFS/RDMA work: Current client version is on Sourceforge, supporting various flavors of 2.4. I'm preparing a new release for 2.6. The client needs a patched version of Sunrpc in order to hook in as an NFS transport. This patch is being used by the NFS/IPv6 project as well, and it's planned for future kernel.org integration. (There are also patch sets for earlier Linux revs) Server version for 2.6 is under development at UMich CITI, scroll down to Documents and Code sections of this page: Finally, NetApp and Sun have demonstrated implementations, Solaris 10 has support for it. We look forward to running NFS/RDMA over OpenIB, when its kDAPL is ready. Tom. From jjengla at sandia.gov Thu Jun 23 10:54:39 2005 From: jjengla at sandia.gov (Josh England) Date: Thu, 23 Jun 2005 10:54:39 -0700 Subject: [openib-general] RDMA write with immediate In-Reply-To: <52u0jpnjyy.fsf@topspin.com> References: <1119480693.24922.33.camel@localhost> <52k6klq184.fsf@topspin.com> <1119526607.29410.6.camel@localhost> <52u0jpnjyy.fsf@topspin.com> Message-ID: <1119549279.31017.5.camel@localhost> On Thu, 2005-06-23 at 08:58 -0700, Roland Dreier wrote: > Josh> I get a completion with failed status 13. > > Josh> Again, this works just fine with IBV_WR_RDMA_WRITE. I'm > Josh> using 2.6.11.11 with SVN rev 2606. I'll update stuff and > Josh> try again. > > status 13 is "RNR retries exceeded." This means that the remote side > did not have a receive ready to handle this work request. > > What are you trying to do here? The between RDMA write and RDMA write > with immediate operations is sending immediate data consumes a receive > request on the remote system and generates a completion with the > immediate data. So you need to post a receive to handle RDMA write > with immediate but not regular RDMA write. There's my problem. I was just polling on the receive buffer before...not posting a receive. > Did you modify your app to post the receive here? I'll do that. Thanks, -JE > - R. > From jlentini at netapp.com Thu Jun 23 11:00:50 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 14:00:50 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: use Grant's suggestions In-Reply-To: <1119548015.9388.2.camel@duffman> References: <1119460629.26537.21.camel@duffman> <20050623062020.GB21363@esmail.cup.hp.com> <1119548015.9388.2.camel@duffman> Message-ID: Committed in revision 2691. On Thu, 23 Jun 2005, Tom Duffy wrote: tduffy> On Wed, 2005-06-22 at 23:20 -0700, Grant Grundler wrote: tduffy> > On Wed, Jun 22, 2005 at 10:17:09AM -0700, Tom Duffy wrote: tduffy> > > This patch removes the function dapl_os_panic() in favor of calling tduffy> > > panic() directly. tduffy> > tduffy> > just some nits... tduffy> > tduffy> > > if (!evd) tduffy> > > - dapl_os_panic("NULL == context\n"); tduffy> > > + panic("NULL == context\n"); tduffy> > tduffy> > Isn't this a bit silly? tduffy> > tduffy> > > async_evd = evd->common.owner_ia->async_error_evd; tduffy> > tduffy> > The system is going to panic anyway here if evd is null. tduffy> > I don't see any advantage to testing evd if the only action is to panic. tduffy> > tduffy> > > - if (NULL == context) { tduffy> > > - dapl_os_panic("NULL == context\n"); tduffy> > > - return; tduffy> > > - } tduffy> > > + if (NULL == context) tduffy> > > + panic("NULL == context\n"); tduffy> > tduffy> > Would this be better as "BUG_ON(NULL == context)"? tduffy> tduffy> I agree with both points. tduffy> tduffy> Here is a new patch based off of r2689. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.c (revision 2689) tduffy> +++ linux-kernel/dat-provider/dapl_evd.c (working copy) tduffy> @@ -680,9 +680,6 @@ static void dapl_evd_cq_async_error_call tduffy> "dapl_evd_cq_async_error_callback (%p, %p)\n", tduffy> cause, context); tduffy> tduffy> - if (!evd) tduffy> - panic("NULL == context\n"); tduffy> - tduffy> async_evd = evd->common.owner_ia->async_error_evd; tduffy> tduffy> status = dapl_evd_post_async_error_event(async_evd, tduffy> @@ -706,8 +703,7 @@ void dapl_evd_un_async_error_callback(st tduffy> "dapl_evd_un_async_error_callback (%p, %p)\n", tduffy> cause, context); tduffy> tduffy> - if (NULL == context) tduffy> - panic("NULL == context\n"); tduffy> + BUG_ON(context == NULL); tduffy> tduffy> async_evd = (struct dapl_evd *)context; tduffy> tduffy> tduffy> _______________________________________________ tduffy> openib-general mailing list tduffy> openib-general at openib.org tduffy> http://openib.org/mailman/listinfo/openib-general tduffy> tduffy> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general tduffy> From jlentini at netapp.com Thu Jun 23 11:08:51 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 14:08:51 -0400 (EDT) Subject: [openib-general] Re: [PATCH] cleanup dat provider registratio n In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35D07@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35D07@mtvex01.mtv.mtl.com> Message-ID: Making it a pointer would save memory, but it would open up some difficult questions. If it is a pointer, how is the underlying memory managed? This structure is returned by dat_registry_list_providers(). When would the registry be allowed to free this data? If we leave it as is, we will avoid these types of ambiguities. On Thu, 23 Jun 2005, Itamar Rabenstein wrote: > i have a question: > after we have cleaned up the dat_provider_info struct and now it is define > as : > > struct dat_provider_info { > char ia_name[DAT_NAME_MAX_LENGTH]; > }; > > why cant we replace it with a simple char * > it is only a string that the only thing we do with it > is to store it and compare with others strings. > > Itamar > >> -----Original Message----- >> From: James Lentini [mailto:jlentini at netapp.com] >> Sent: Thursday, June 23, 2005 12:13 AM >> To: Christoph Hellwig >> Cc: openib-general at openib.org >> Subject: [openib-general] Re: [PATCH] cleanup dat provider >> registration >> >> >> >> This is an excellent simplification. Committed in revision >> 2682 with a >> few minor modifications: >> >> - kept printouts in dat_registry_add_provider, >> dat_registry_remove_provider, and dat_registry_list_providers >> >> - updated printout in dat_ia_close (this wasn't something you >> changed) >> >> - removed parens around sizeof >> >> - removed space in front of labels >> >> The last two are for consistency with the coding style we've been >> using. If we've deviated from what is acceptable, let us know. >> >> Given this simplification, I can think of a few more changes: >> >> - rename api.c to registry.c >> - remove all "dictionary" references: rename >> dat_dictionary_search to >> dat_provider_list_search, rename struct dat_dictionary_entry to >> struct dat_provider_list_entry, rename >> dat_dictionary_key_is_equal() to dat_provider_info_is_equal() >> >> Anyone have opinions for or against? >> >> On Fri, 17 Jun 2005, Christoph Hellwig wrote: >> >> hch> This huge patch revamps almost everything under dat, and >> merges the three >> hch> layers of external API, dynamic registry and dictinary >> into a single >> hch> one, greatly streamlining all of the code. >> hch> >> hch> Note that I don't have any RDMA hardware to actually >> takes this, so it >> hch> probably needs some testing first. >> hch> >> hch> >> hch> Signed-off-by: Christoph Hellwig >> hch> >> hch> Index: linux-kernel/dat/dr.c >> hch> >> =================================================================== >> hch> --- linux-kernel/dat/dr.c (revision 2647) >> hch> +++ linux-kernel/dat/dr.c (working copy) >> hch> @@ -1,242 +0,0 @@ >> hch> -/* >> hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All >> rights reserved. >> hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights >> reserved. >> hch> - * >> hch> - * This Software is licensed under one of the following >> licenses: >> hch> - * >> hch> - * 1) under the terms of the "Common Public License >> 1.0" a copy of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/cpl.php. >> hch> - * >> hch> - * 2) under the terms of the "The BSD License" a copy >> of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/bsd-license.php. >> hch> - * >> hch> - * 3) under the terms of the "GNU General Public >> License (GPL) Version 2" a >> hch> - * copy of which is available from the Open Source >> Initiative, see >> hch> - * http://www.opensource.org/licenses/gpl-license.php. >> hch> - * >> hch> - * Licensee has the right to choose one of the above licenses. >> hch> - * >> hch> - * Redistributions of source code must retain the above >> copyright >> hch> - * notice and one of the license notices. >> hch> - * >> hch> - * Redistributions in binary form must reproduce both >> the above copyright >> hch> - * notice, one of the license notices in the documentation >> hch> - * and/or other materials provided with the distribution. >> hch> - */ >> hch> - >> hch> -/* >> hch> - * Dynamic Registry (DR) functions and data structures >> hch> - * >> hch> - * $Id$ >> hch> - */ >> hch> - >> hch> -#include "core.h" >> hch> -#include "dictionary.h" >> hch> -#include "dr.h" >> hch> - >> hch> -struct dat_dr_entry { >> hch> - int ref_count; >> hch> - struct dat_provider_info info; >> hch> - DAT_IA_OPEN_FUNC ia_open_func; >> hch> -}; >> hch> - >> hch> -static spinlock_t g_dr_lock = SPIN_LOCK_UNLOCKED; >> hch> -static struct dat_dictionary g_dr_dictionary = >> hch> - DAT_DICTIONARY_INIT(g_dr_dictionary); >> hch> - >> hch> -void dat_dr_fini(void) >> hch> -{ >> hch> - dat_dictionary_destroy(&g_dr_dictionary); >> hch> -} >> hch> - >> hch> -u32 dat_dr_insert(const struct dat_provider_info *info, >> hch> - const DAT_IA_OPEN_FUNC ia_open_func) >> hch> -{ >> hch> - struct dat_dr_entry *data; >> hch> - struct dat_dictionary_entry *dict_entry; >> hch> - unsigned long flags; >> hch> - u32 status; >> hch> - >> hch> - data = kmalloc(sizeof *data, GFP_ATOMIC); >> hch> - if (!data) { >> hch> - status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, >> hch> - DAT_RESOURCE_MEMORY); >> hch> - goto bail; >> hch> - } >> hch> - >> hch> - data->ref_count = 0; >> hch> - data->info = *info; >> hch> - data->ia_open_func = ia_open_func; >> hch> - >> hch> - dict_entry = NULL; >> hch> - status = dat_dictionary_entry_create(&dict_entry); >> hch> - if (DAT_SUCCESS != status) >> hch> - goto bail; >> hch> - >> hch> - spin_lock_irqsave(&g_dr_lock, flags); >> hch> - >> hch> - status = >> dat_dictionary_insert(&g_dr_dictionary, info, dict_entry, >> hch> - data); >> hch> - >> hch> - spin_unlock_irqrestore(&g_dr_lock, flags); >> hch> - >> hch> -bail: >> hch> - if (DAT_SUCCESS != status) { >> hch> - if (NULL != data) >> hch> - kfree(data); >> hch> - >> hch> - if (NULL != dict_entry) >> hch> - >> dat_dictionary_entry_destroy(dict_entry); >> hch> - } >> hch> - >> hch> - return status; >> hch> -} >> hch> - >> hch> -u32 dat_dr_remove(const struct dat_provider_info *info) >> hch> -{ >> hch> - struct dat_dr_entry *data; >> hch> - struct dat_dictionary_entry *dict_entry; >> hch> - unsigned long flags; >> hch> - u32 status; >> hch> - >> hch> - dict_entry = NULL; >> hch> - spin_lock_irqsave(&g_dr_lock, flags); >> hch> - >> hch> - status = >> dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); >> hch> - >> hch> - if (DAT_SUCCESS != status) >> hch> - goto bail; /* return status from >> dat_dictionary_search() */ >> hch> - >> hch> - if (0 != data->ref_count) { >> hch> - status = DAT_ERROR(DAT_PROVIDER_IN_USE, 0); >> hch> - goto bail; >> hch> - } >> hch> - >> hch> - status = >> dat_dictionary_remove(&g_dr_dictionary, info, &dict_entry, >> hch> - (void **) &data); >> hch> - if (DAT_SUCCESS != status) >> hch> - goto bail; /* return status from >> dat_dictionary_remove() */ >> hch> - >> hch> - kfree(data); >> hch> - >> hch> -bail: >> hch> - spin_unlock_irqrestore(&g_dr_lock, flags); >> hch> - >> hch> - if (NULL != dict_entry) >> hch> - dat_dictionary_entry_destroy(dict_entry); >> hch> - >> hch> - return status; >> hch> -} >> hch> - >> hch> -u32 dat_dr_provider_open(const struct dat_provider_info *info, >> hch> - DAT_IA_OPEN_FUNC >> *p_ia_open_func) >> hch> -{ >> hch> - struct dat_dr_entry *data; >> hch> - unsigned long flags; >> hch> - u32 status; >> hch> - >> hch> - spin_lock_irqsave(&g_dr_lock, flags); >> hch> - >> hch> - status = >> dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); >> hch> - >> hch> - spin_unlock_irqrestore(&g_dr_lock, flags); >> hch> - >> hch> - if (DAT_SUCCESS == status) { >> hch> - data->ref_count++; >> hch> - *p_ia_open_func = data->ia_open_func; >> hch> - } >> hch> - >> hch> - return status; >> hch> -} >> hch> - >> hch> -u32 dat_dr_provider_close(const struct dat_provider_info *info) >> hch> -{ >> hch> - struct dat_dr_entry *data; >> hch> - unsigned long flags; >> hch> - u32 status; >> hch> - >> hch> - spin_lock_irqsave(&g_dr_lock, flags); >> hch> - >> hch> - status = >> dat_dictionary_search(&g_dr_dictionary, info, (void **) &data); >> hch> - >> hch> - spin_unlock_irqrestore(&g_dr_lock, flags); >> hch> - >> hch> - if (DAT_SUCCESS == status) >> hch> - data->ref_count--; >> hch> - >> hch> - return status; >> hch> -} >> hch> - >> hch> -u32 dat_dr_size(int *size) >> hch> -{ >> hch> - return dat_dictionary_size(&g_dr_dictionary, size); >> hch> -} >> hch> - >> hch> -u32 dat_dr_list(int max_to_return, int *entries_returned, >> hch> - struct dat_provider_info *(dat_provider_list[])) >> hch> -{ >> hch> - struct dat_dr_entry **array = NULL; >> hch> - int i, array_size; >> hch> - unsigned long flags; >> hch> - u32 status = DAT_SUCCESS; >> hch> - >> hch> - /* >> hch> - * The dictionary size may increase between the call to >> hch> - * dat_dictionary_size() and dat_dictionary_enumerate(). >> hch> - * Therefore we loop until a successful >> enumeration is made. >> hch> - */ >> hch> - *entries_returned = 0; >> hch> - for (;;) { >> hch> - status = >> dat_dictionary_size(&g_dr_dictionary, &array_size); >> hch> - if (DAT_SUCCESS != status) >> hch> - goto bail; >> hch> - >> hch> - if (0 == array_size) { >> hch> - status = DAT_SUCCESS; >> hch> - goto bail; >> hch> - } >> hch> - >> hch> - array = kmalloc(array_size * sizeof >> *array, GFP_ATOMIC); >> hch> - if (!array) { >> hch> - status = >> hch> - >> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, >> hch> - DAT_RESOURCE_MEMORY); >> hch> - goto bail; >> hch> - } >> hch> - >> hch> - spin_lock_irqsave(&g_dr_lock, flags); >> hch> - >> hch> - status = >> dat_dictionary_enumerate(&g_dr_dictionary, >> hch> - (void >> **) array, array_size); >> hch> - >> hch> - spin_unlock_irqrestore(&g_dr_lock, flags); >> hch> - >> hch> - if (DAT_SUCCESS == status) >> hch> - break; >> hch> - else { >> hch> - kfree(array); >> hch> - array = NULL; >> hch> - continue; >> hch> - } >> hch> - } >> hch> - >> hch> - for (i = 0; (i < max_to_return) && (i < >> array_size); i++) { >> hch> - if (NULL == dat_provider_list[i]) { >> hch> - status = >> hch> - >> DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); >> hch> - goto bail; >> hch> - } >> hch> - >> hch> - *dat_provider_list[i] = array[i]->info; >> hch> - } >> hch> - >> hch> - *entries_returned = i; >> hch> - >> hch> -bail: >> hch> - if (NULL != array) >> hch> - kfree(array); >> hch> - >> hch> - return status; >> hch> -} >> hch> Index: linux-kernel/dat/dictionary.c >> hch> >> =================================================================== >> hch> --- linux-kernel/dat/dictionary.c (revision 2647) >> hch> +++ linux-kernel/dat/dictionary.c (working copy) >> hch> @@ -1,229 +0,0 @@ >> hch> -/* >> hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All >> rights reserved. >> hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights >> reserved. >> hch> - * >> hch> - * This Software is licensed under one of the following >> licenses: >> hch> - * >> hch> - * 1) under the terms of the "Common Public License >> 1.0" a copy of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/cpl.php. >> hch> - * >> hch> - * 2) under the terms of the "The BSD License" a copy >> of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/bsd-license.php. >> hch> - * >> hch> - * 3) under the terms of the "GNU General Public >> License (GPL) Version 2" a >> hch> - * copy of which is available from the Open Source >> Initiative, see >> hch> - * http://www.opensource.org/licenses/gpl-license.php. >> hch> - * >> hch> - * Licensee has the right to choose one of the above licenses. >> hch> - * >> hch> - * Redistributions of source code must retain the above >> copyright >> hch> - * notice and one of the license notices. >> hch> - * >> hch> - * Redistributions in binary form must reproduce both >> the above copyright >> hch> - * notice, one of the license notices in the documentation >> hch> - * and/or other materials provided with the distribution. >> hch> - */ >> hch> - >> hch> -/* >> hch> - * A dictionary data structure implemented with a linked list >> hch> - * >> hch> - * $Id$ >> hch> - */ >> hch> - >> hch> -#include "dictionary.h" >> hch> - >> hch> -static u32 dat_dictionary_key_dup(const struct >> dat_provider_info *old_key, >> hch> - struct >> dat_provider_info *new_key); >> hch> - >> hch> -static boolean_t dat_dictionary_key_is_equal(const >> struct dat_provider_info *a, >> hch> - const >> struct dat_provider_info *b); >> hch> - >> hch> -void dat_dictionary_destroy(struct dat_dictionary *dictionary) >> hch> -{ >> hch> - struct list_head *cur_list, *next_list; >> hch> - struct dat_dictionary_entry *cur_entry; >> hch> - >> hch> - BUG_ON(NULL == dictionary); >> hch> - >> hch> - list_for_each_safe(cur_list, next_list, >> &dictionary->list) { >> hch> - cur_entry = list_entry(cur_list, struct >> dat_dictionary_entry, >> hch> - list); >> hch> - list_del(&cur_entry->list); >> hch> - kfree(cur_entry); >> hch> - } >> hch> -} >> hch> - >> hch> -u32 dat_dictionary_size(const struct dat_dictionary >> *dictionary, int *size) >> hch> -{ >> hch> - BUG_ON(NULL == dictionary); >> hch> - BUG_ON(NULL == size); >> hch> - >> hch> - *size = dictionary->size; >> hch> - >> hch> - return DAT_SUCCESS; >> hch> -} >> hch> - >> hch> -u32 dat_dictionary_entry_create(struct >> dat_dictionary_entry **p_entry) >> hch> -{ >> hch> - struct dat_dictionary_entry *entry; >> hch> - u32 status = DAT_SUCCESS; >> hch> - >> hch> - BUG_ON(NULL == p_entry); >> hch> - >> hch> - entry = kmalloc(sizeof *entry, GFP_ATOMIC); >> hch> - if (!entry) { >> hch> - status = >> hch> - >> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); >> hch> - goto bail; >> hch> - } >> hch> - >> hch> - *p_entry = entry; >> hch> - >> hch> -bail: >> hch> - return status; >> hch> -} >> hch> - >> hch> -void dat_dictionary_entry_destroy(const struct >> dat_dictionary_entry *entry) >> hch> -{ >> hch> - kfree(entry); >> hch> -} >> hch> - >> hch> -u32 dat_dictionary_insert(struct dat_dictionary *dictionary, >> hch> - const struct dat_provider_info *key, >> hch> - struct dat_dictionary_entry >> *entry, void *data) >> hch> -{ >> hch> - u32 status; >> hch> - >> hch> - BUG_ON(NULL == dictionary); >> hch> - BUG_ON(NULL == entry); >> hch> - >> hch> - if (DAT_SUCCESS == >> dat_dictionary_search(dictionary, key, NULL)) { >> hch> - status = >> DAT_ERROR(DAT_PROVIDER_ALREADY_REGISTERED, 0); >> hch> - goto bail; >> hch> - } >> hch> - >> hch> - status = dat_dictionary_key_dup(key, &entry->key); >> hch> - if (DAT_SUCCESS != status) >> hch> - goto bail; >> hch> - >> hch> - entry->data = data; >> hch> - >> hch> - /* insert node at end of list to preserve >> registration order */ >> hch> - list_add_tail(&entry->list, &dictionary->list); >> hch> - >> hch> - dictionary->size++; >> hch> - >> hch> -bail: >> hch> - return status; >> hch> -} >> hch> - >> hch> -u32 dat_dictionary_search(struct dat_dictionary *dictionary, >> hch> - const struct >> dat_provider_info *key, void **p_data) >> hch> -{ >> hch> - struct dat_dictionary_entry *cur_entry; >> hch> - u32 status; >> hch> - >> hch> - BUG_ON(NULL == dictionary); >> hch> - BUG_ON(NULL == key); >> hch> - >> hch> - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, >> DAT_NAME_NOT_REGISTERED); >> hch> - >> hch> - list_for_each_entry(cur_entry, >> &dictionary->list, list) { >> hch> - if (TRUE == >> dat_dictionary_key_is_equal(&cur_entry->key, key)) { >> hch> - if (NULL != p_data) >> hch> - *p_data = cur_entry->data; >> hch> - >> hch> - status = DAT_SUCCESS; >> hch> - goto bail; >> hch> - } >> hch> - } >> hch> - >> hch> -bail: >> hch> - return status; >> hch> -} >> hch> - >> hch> -u32 dat_dictionary_enumerate(struct dat_dictionary >> *dictionary, void *array[], >> hch> - int array_size) >> hch> -{ >> hch> - struct dat_dictionary_entry *cur_entry; >> hch> - int i; >> hch> - u32 status; >> hch> - >> hch> - BUG_ON(NULL == dictionary); >> hch> - BUG_ON(NULL == array); >> hch> - >> hch> - status = DAT_SUCCESS; >> hch> - >> hch> - if (array_size < dictionary->size) { >> hch> - status = >> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); >> hch> - goto bail; >> hch> - } >> hch> - >> hch> - i = 0; >> hch> - list_for_each_entry(cur_entry, &dictionary->list, list) >> hch> - array[i++] = cur_entry->data; >> hch> - >> hch> -bail: >> hch> - return status; >> hch> -} >> hch> - >> hch> -u32 dat_dictionary_remove(struct dat_dictionary *dictionary, >> hch> - const struct dat_provider_info *key, >> hch> - struct dat_dictionary_entry >> **p_entry, void **p_data) >> hch> -{ >> hch> - struct list_head *cur_list, *next_list; >> hch> - struct dat_dictionary_entry *cur_entry; >> hch> - u32 status; >> hch> - >> hch> - BUG_ON(NULL == dictionary); >> hch> - BUG_ON(NULL == key); >> hch> - BUG_ON(NULL == p_entry); >> hch> - >> hch> - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, >> DAT_NAME_NOT_REGISTERED); >> hch> - >> hch> - list_for_each_safe(cur_list, next_list, >> &dictionary->list) { >> hch> - cur_entry = list_entry(cur_list, struct >> dat_dictionary_entry, >> hch> - list); >> hch> - if (TRUE == >> dat_dictionary_key_is_equal(&cur_entry->key, key)) { >> hch> - if (NULL != p_data) >> hch> - *p_data = cur_entry->data; >> hch> - >> hch> - list_del(&cur_entry->list); >> hch> - >> hch> - *p_entry = cur_entry; >> hch> - >> hch> - dictionary->size--; >> hch> - >> hch> - status = DAT_SUCCESS; >> hch> - goto bail; >> hch> - } >> hch> - } >> hch> - >> hch> -bail: >> hch> - return status; >> hch> -} >> hch> - >> hch> -static u32 dat_dictionary_key_dup(const struct >> dat_provider_info *old_key, >> hch> - struct >> dat_provider_info *new_key) >> hch> -{ >> hch> - BUG_ON(NULL == old_key); >> hch> - BUG_ON(NULL == new_key); >> hch> - >> hch> - strncpy(new_key->ia_name, old_key->ia_name, >> DAT_NAME_MAX_LENGTH); >> hch> - return DAT_SUCCESS; >> hch> -} >> hch> - >> hch> -static boolean_t dat_dictionary_key_is_equal(const >> struct dat_provider_info *a, >> hch> - const >> struct dat_provider_info *b) >> hch> -{ >> hch> - BUG_ON(NULL == a); >> hch> - BUG_ON(NULL == b); >> hch> - >> hch> - if ((strlen(a->ia_name) == strlen(b->ia_name)) && >> hch> - (!strncmp(a->ia_name, b->ia_name, >> strlen(a->ia_name)))) >> hch> - return TRUE; >> hch> - else >> hch> - return FALSE; >> hch> -} >> hch> Index: linux-kernel/dat/core.c >> hch> >> =================================================================== >> hch> --- linux-kernel/dat/core.c (revision 2647) >> hch> +++ linux-kernel/dat/core.c (working copy) >> hch> @@ -1,75 +0,0 @@ >> hch> -/* >> hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All >> rights reserved. >> hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights >> reserved. >> hch> - * >> hch> - * This Software is licensed under one of the following >> licenses: >> hch> - * >> hch> - * 1) under the terms of the "Common Public License >> 1.0" a copy of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/cpl.php. >> hch> - * >> hch> - * 2) under the terms of the "The BSD License" a copy >> of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/bsd-license.php. >> hch> - * >> hch> - * 3) under the terms of the "GNU General Public >> License (GPL) Version 2" a >> hch> - * copy of which is available from the Open Source >> Initiative, see >> hch> - * http://www.opensource.org/licenses/gpl-license.php. >> hch> - * >> hch> - * Licensee has the right to choose one of the above licenses. >> hch> - * >> hch> - * Redistributions of source code must retain the above >> copyright >> hch> - * notice and one of the license notices. >> hch> - * >> hch> - * Redistributions in binary form must reproduce both >> the above copyright >> hch> - * notice, one of the license notices in the documentation >> hch> - * and/or other materials provided with the distribution. >> hch> - */ >> hch> - >> hch> -/* >> hch> - * Core DAT registry functions and data structures >> hch> - * >> hch> - * $Id$ >> hch> - */ >> hch> - >> hch> -#include "core.h" >> hch> -#include "dr.h" >> hch> - >> hch> -MODULE_LICENSE("Dual BSD/GPL"); >> hch> -MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); >> hch> -MODULE_AUTHOR("James Lentini"); >> hch> - >> hch> -static DAT_DBG_MASK g_dbg_mask = DAT_DBG_TYPE_ERROR; >> hch> -module_param_named(dbg_mask, g_dbg_mask, int, 0644); >> hch> -MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug >> message types."); >> hch> - >> hch> -void dat_dbg_print(DAT_DBG_TYPE type, const char *fmt, ...) >> hch> -{ >> hch> - static char buf[1024]; >> hch> - >> hch> - if (type & g_dbg_mask) { >> hch> - va_list args; >> hch> - >> hch> - va_start(args, fmt); >> hch> - vsnprintf(buf, sizeof buf, fmt, args); >> hch> - printk(KERN_ALERT "DAT: %s", buf); >> hch> - va_end(args); >> hch> - } >> hch> -} >> hch> - >> hch> -static int __init dat_init(void) >> hch> -{ >> hch> - dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry >> started\n"); >> hch> - return 0; >> hch> -} >> hch> - >> hch> -module_init(dat_init); >> hch> - >> hch> -static void __exit dat_fini(void) >> hch> -{ >> hch> - dat_dr_fini(); >> hch> - >> hch> - dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry >> stopped\n"); >> hch> -} >> hch> - >> hch> -module_exit(dat_fini); >> hch> Index: linux-kernel/dat/api.c >> hch> >> =================================================================== >> hch> --- linux-kernel/dat/api.c (revision 2647) >> hch> +++ linux-kernel/dat/api.c (working copy) >> hch> @@ -26,15 +26,53 @@ >> hch> * and/or other materials provided with the distribution. >> hch> */ >> hch> >> hch> -/* >> hch> - * DAT functions for consumers >> hch> - * >> hch> - * $Id: consumer.c 2251 2005-05-04 05:57:41Z jlentini $ >> hch> - */ >> hch> +#include >> hch> +#include >> hch> +#include >> hch> +#include >> hch> +#include >> hch> >> hch> -#include "core.h" >> hch> -#include "dr.h" >> hch> +#include >> hch> >> hch> + >> hch> +enum dat_dbg_type { >> hch> + DAT_DBG_TYPE_ERROR = (1 << 0), >> hch> + DAT_DBG_TYPE_GENERIC = (1 << 1), >> hch> + DAT_DBG_TYPE_DR = (1 << 2), >> hch> + DAT_DBG_TYPE_PROVIDER_API = (1 << 3), >> hch> + DAT_DBG_TYPE_CONSUMER_API = (1 << 4), >> hch> + DAT_DBG_TYPE_ALL = 0x1f >> hch> +}; >> hch> + >> hch> +static int dat_dbg_mask = DAT_DBG_TYPE_ERROR; >> hch> +module_param_named(dbg_mask, dat_dbg_mask, int, 0644); >> hch> +MODULE_PARM_DESC(dbg_mask, "Bitmask to enable debug >> message types."); >> hch> + >> hch> +struct dat_dictionary_entry { >> hch> + struct list_head list; >> hch> + struct dat_provider_info info; >> hch> + DAT_IA_OPEN_FUNC ia_open_func; >> hch> + int ref_count; >> hch> +}; >> hch> + >> hch> +static LIST_HEAD(dat_provider_list); >> hch> +static int dat_provider_list_size; >> hch> +static DEFINE_SPINLOCK(dat_provider_list_lock); >> hch> + >> hch> +static void dat_dbg_print(enum dat_dbg_type type, const >> char *fmt, ...) >> hch> +{ >> hch> + static char buf[1024]; >> hch> + >> hch> + if (type & dat_dbg_mask) { >> hch> + va_list args; >> hch> + >> hch> + va_start(args, fmt); >> hch> + vsnprintf(buf, sizeof buf, fmt, args); >> hch> + printk(KERN_ALERT "DAT: %s", buf); >> hch> + va_end(args); >> hch> + } >> hch> +} >> hch> + >> hch> static u32 dat_strerror_major(u32 value, const char **message) >> hch> { >> hch> switch (DAT_GET_TYPE(value)) { >> hch> @@ -350,21 +388,39 @@ >> hch> } >> hch> } >> hch> >> hch> +static boolean_t dat_dictionary_key_is_equal(const >> struct dat_provider_info *a, >> hch> + const >> struct dat_provider_info *b) >> hch> +{ >> hch> + if (strlen(a->ia_name) == strlen(b->ia_name) && >> hch> + !strncmp(a->ia_name, b->ia_name, >> strlen(a->ia_name))) >> hch> + return 1; >> hch> + return 0; >> hch> +} >> hch> + >> hch> +static struct dat_dictionary_entry *dat_dictionary_search( >> hch> + struct list_head *list, const struct >> dat_provider_info *key) >> hch> +{ >> hch> + struct dat_dictionary_entry *entry; >> hch> + >> hch> + list_for_each_entry(entry, list, list) >> hch> + if >> (dat_dictionary_key_is_equal(&entry->info, key)) >> hch> + return entry; >> hch> + >> hch> + return NULL; >> hch> +} >> hch> + >> hch> u32 dat_ia_open(const char *name, int async_event_qlen, >> hch> struct dat_evd **async_event_handle, >> struct dat_ia **ia) >> hch> { >> hch> - DAT_IA_OPEN_FUNC ia_open_func; >> hch> struct dat_provider_info info; >> hch> - u32 status; >> hch> + struct dat_dictionary_entry *data; >> hch> + unsigned long flags; >> hch> size_t len; >> hch> >> hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, >> hch> "%s(name:%s, async_qlen:%i, >> ver:%x.%x, thrd_safe:%x)\n", >> hch> __func__, name, async_event_qlen); >> hch> >> hch> - if (DAT_IS_BAD_POINTER(name)) >> hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, >> DAT_INVALID_ARG1); >> hch> - >> hch> len = strlen(name); >> hch> >> hch> if (DAT_NAME_MAX_LENGTH < len) >> hch> @@ -373,62 +429,67 @@ >> hch> strncpy(info.ia_name, name, len); >> hch> info.ia_name[len] = '\0'; >> hch> >> hch> - status = dat_dr_provider_open(&info, &ia_open_func); >> hch> - if (status != DAT_SUCCESS) { >> hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); >> hch> + data = dat_dictionary_search(&dat_provider_list, &info); >> hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); >> hch> + >> hch> + if (!data) { >> hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, >> hch> "%s: IA [%s] not found in >> dynamic registry\n", >> hch> __func__, name); >> hch> >> hch> - return status; >> hch> + return DAT_ERROR(DAT_PROVIDER_NOT_FOUND, >> hch> + DAT_NAME_NOT_REGISTERED); >> hch> } >> hch> >> hch> - return ia_open_func(name, async_event_qlen, >> async_event_handle, ia); >> hch> + data->ref_count++; >> hch> + return data->ia_open_func(name, >> async_event_qlen, async_event_handle, >> hch> + ia); >> hch> } >> hch> - >> hch> EXPORT_SYMBOL(dat_ia_open); >> hch> >> hch> -u32 dat_ia_close(struct dat_ia *ia, enum dat_close_flags flags) >> hch> +u32 dat_ia_close(struct dat_ia *ia, enum >> dat_close_flags close_flags) >> hch> { >> hch> struct dat_provider *provider; >> hch> - struct dat_provider_attr provider_attr; >> hch> - u32 status; >> hch> + struct dat_provider_info info; >> hch> + u32 status = DAT_SUCCESS; >> hch> const char *ia_name; >> hch> + struct dat_dictionary_entry *data; >> hch> + unsigned long flags; >> hch> + size_t len; >> hch> >> hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, >> hch> - "%s(ia:%x, flags:%x)\n", >> __func__, ia, flags); >> hch> + "%s(ia:%x, flags:%x)\n", >> __func__, ia, close_flags); >> hch> >> hch> provider = ia->common.provider; >> hch> ia_name = provider->device_name; >> hch> >> hch> - if (DAT_SUCCESS != (status = dat_ia_query(ia, >> NULL, NULL, >> hch> - >> &provider_attr))) >> hch> + status = provider->ia_close_func(ia, close_flags); >> hch> + if (status != DAT_SUCCESS) { >> hch> dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, >> hch> - "dat_ia_query for IA %s >> failed\n", ia_name); >> hch> - else if (DAT_SUCCESS != (status = >> provider->ia_close_func(ia, flags))) >> hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, >> hch> "close function for IA %s >> failed\n", ia_name); >> hch> - else { >> hch> - struct dat_provider_info info; >> hch> - size_t len; >> hch> + return status; >> hch> + } >> hch> >> hch> - len = strlen(ia_name); >> hch> + len = strlen(ia_name); >> hch> >> hch> - BUG_ON(DAT_NAME_MAX_LENGTH < len); >> hch> + BUG_ON(DAT_NAME_MAX_LENGTH < len); >> hch> >> hch> - strncpy(info.ia_name, ia_name, len); >> hch> - info.ia_name[len] = '\0'; >> hch> + strncpy(info.ia_name, ia_name, len); >> hch> + info.ia_name[len] = '\0'; >> hch> >> hch> - status = dat_dr_provider_close(&info); >> hch> - if (DAT_SUCCESS != status) >> hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, >> hch> - "dynamic registry >> unable to close " >> hch> - "provider for IA %s\n", >> hch> - ia_name); >> hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); >> hch> + data = dat_dictionary_search(&dat_provider_list, &info); >> hch> + if (data) { >> hch> + data->ref_count--; >> hch> + } else { >> hch> + status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, >> hch> + DAT_NAME_NOT_REGISTERED); >> hch> } >> hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); >> hch> >> hch> return status; >> hch> } >> hch> - >> hch> EXPORT_SYMBOL(dat_ia_close); >> hch> >> hch> u32 dat_strerror(u32 value, const char **major_message, >> hch> @@ -454,69 +515,138 @@ >> hch> >> hch> return DAT_SUCCESS; >> hch> } >> hch> - >> hch> EXPORT_SYMBOL(dat_strerror); >> hch> >> hch> u32 dat_registry_add_provider(const struct dat_provider >> *provider, >> hch> - const struct >> dat_provider_info *provider_info) >> hch> + const struct >> dat_provider_info *info) >> hch> { >> hch> - dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, >> hch> - "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", >> hch> - __func__, provider_info->ia_name); >> hch> + struct dat_dictionary_entry *entry; >> hch> + unsigned long flags; >> hch> + u32 status = DAT_SUCCESS; >> hch> >> hch> - if (DAT_IS_BAD_POINTER(provider)) >> hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, >> DAT_INVALID_ARG1); >> hch> + entry = kmalloc(sizeof(*entry), GFP_ATOMIC); >> hch> + if (!entry) { >> hch> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, >> hch> + DAT_RESOURCE_MEMORY); >> hch> + goto out; >> hch> + } >> hch> >> hch> - if (DAT_IS_BAD_POINTER(provider_info)) >> hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, >> DAT_INVALID_ARG2); >> hch> + entry->ref_count = 0; >> hch> + entry->info = *info; >> hch> + entry->ia_open_func = provider->ia_open_func; >> hch> >> hch> - return dat_dr_insert(provider_info, >> provider->ia_open_func); >> hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); >> hch> + if (dat_dictionary_search(&dat_provider_list, info)) { >> hch> + status = >> DAT_ERROR(DAT_PROVIDER_ALREADY_REGISTERED, 0); >> hch> + goto out_unlock; >> hch> + } >> hch> + >> hch> + strncpy(entry->info.ia_name, info->ia_name, >> DAT_NAME_MAX_LENGTH); >> hch> + >> hch> + /* insert node at end of list to preserve >> registration order */ >> hch> + list_add_tail(&entry->list, &dat_provider_list); >> hch> + dat_provider_list_size++; >> hch> + >> hch> + out_unlock: >> hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); >> hch> + out: >> hch> + if (status != DAT_SUCCESS) >> hch> + kfree(entry); >> hch> + return status; >> hch> } >> hch> +EXPORT_SYMBOL(dat_registry_add_provider); >> hch> >> hch> -EXPORT_SYMBOL(dat_registry_remove_provider); >> hch> - >> hch> u32 dat_registry_remove_provider(const struct >> dat_provider *provider, >> hch> - const struct >> dat_provider_info *provider_info) >> hch> + const struct >> dat_provider_info *info) >> hch> { >> hch> - dat_dbg_print(DAT_DBG_TYPE_PROVIDER_API, >> "%s(ia:%s)\n", __func__, >> hch> - provider_info->ia_name); >> hch> + struct dat_dictionary_entry *cur, *next; >> hch> + unsigned long flags; >> hch> + u32 status = DAT_SUCCESS; >> hch> >> hch> - if (DAT_IS_BAD_POINTER(provider)) >> hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, >> DAT_INVALID_ARG1); >> hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); >> hch> + list_for_each_entry_safe(cur, next, >> &dat_provider_list, list) { >> hch> + if >> (dat_dictionary_key_is_equal(&cur->info, info)) { >> hch> + if (cur->ref_count) { >> hch> + status = >> DAT_ERROR(DAT_PROVIDER_IN_USE, 0); >> hch> + goto out; >> hch> + } >> hch> >> hch> - return dat_dr_remove(provider_info); >> hch> + list_del(&cur->list); >> hch> + dat_provider_list_size--; >> hch> + >> hch> + kfree(cur); >> hch> + goto out; >> hch> + } >> hch> + } >> hch> + >> hch> + status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, >> DAT_NAME_NOT_REGISTERED); >> hch> + out: >> hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); >> hch> + return status; >> hch> } >> hch> +EXPORT_SYMBOL(dat_registry_remove_provider); >> hch> >> hch> -EXPORT_SYMBOL(dat_registry_add_provider); >> hch> - >> hch> u32 dat_registry_list_providers(int max_to_return, int >> *entries_returned, >> hch> - struct >> dat_provider_info *(dat_provider_list[])) >> hch> + struct >> dat_provider_info *provider_list[]) >> hch> { >> hch> + struct dat_dictionary_entry *cur; >> hch> + unsigned long flags; >> hch> + int i = 0; >> hch> u32 status = DAT_SUCCESS; >> hch> >> hch> - dat_dbg_print(DAT_DBG_TYPE_CONSUMER_API, >> hch> - "%s(max_to_return:%i, >> entries_rtrn:%p, prvdr_list:%p)\n", >> hch> - __func__, max_to_return, >> entries_returned, >> hch> - dat_provider_list); >> hch> - >> hch> - if ((DAT_IS_BAD_POINTER(entries_returned))) >> hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, >> DAT_INVALID_ARG2); >> hch> - >> hch> - if (0 != max_to_return && >> (DAT_IS_BAD_POINTER(dat_provider_list))) >> hch> - return DAT_ERROR(DAT_INVALID_PARAMETER, >> DAT_INVALID_ARG3); >> hch> - >> hch> - if (0 == max_to_return) { >> hch> + if (!max_to_return) { >> hch> /* the user is allowed to call with >> max_to_return set to zero. >> hch> * in which case we simply return (in >> *entries_returned) the >> hch> * number of providers currently >> installed. We must also >> hch> * (per spec) return an error >> hch> */ >> hch> - (void)dat_dr_size(entries_returned); >> hch> + *entries_returned = dat_provider_list_size; >> hch> return DAT_ERROR(DAT_INVALID_PARAMETER, >> DAT_INVALID_ARG1); >> hch> - } else >> hch> - status = dat_dr_list(max_to_return, >> entries_returned, >> hch> - dat_provider_list); >> hch> + } >> hch> + >> hch> + *entries_returned = 0; >> hch> + >> hch> + spin_lock_irqsave(&dat_provider_list_lock, flags); >> hch> + if (!dat_provider_list_size) >> hch> + goto out_unlock; >> hch> + >> hch> + list_for_each_entry(cur, &dat_provider_list, list) { >> hch> + if (i >= max_to_return) >> hch> + break; >> hch> + if (!provider_list[i]) { >> hch> + status = >> hch> + >> DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); >> hch> + goto out_unlock; >> hch> + } >> hch> + provider_list[i++] = &cur->info; >> hch> + } >> hch> + *entries_returned = i; >> hch> + >> hch> + out_unlock: >> hch> + spin_unlock_irqrestore(&dat_provider_list_lock, flags); >> hch> return status; >> hch> } >> hch> +EXPORT_SYMBOL(dat_registry_list_providers); >> hch> >> hch> -EXPORT_SYMBOL(dat_registry_list_providers); >> hch> +static int __init dat_init(void) >> hch> +{ >> hch> + dat_dbg_print(DAT_DBG_TYPE_GENERIC, "registry >> started\n"); >> hch> + return 0; >> hch> +} >> hch> + >> hch> +static void __exit dat_fini(void) >> hch> +{ >> hch> + struct dat_dictionary_entry *cur, *next; >> hch> + >> hch> + list_for_each_entry_safe(cur, next, >> &dat_provider_list, list) { >> hch> + list_del(&cur->list); >> hch> + kfree(cur); >> hch> + } >> hch> +} >> hch> + >> hch> +MODULE_LICENSE("Dual BSD/GPL"); >> hch> +MODULE_DESCRIPTION("Direct Access Transport (DAT) API"); >> hch> +MODULE_AUTHOR("James Lentini"); >> hch> + >> hch> +module_init(dat_init); >> hch> +module_exit(dat_fini); >> hch> Index: linux-kernel/dat/dictionary.h >> hch> >> =================================================================== >> hch> --- linux-kernel/dat/dictionary.h (revision 2647) >> hch> +++ linux-kernel/dat/dictionary.h (working copy) >> hch> @@ -1,98 +0,0 @@ >> hch> -/* >> hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All >> rights reserved. >> hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights >> reserved. >> hch> - * >> hch> - * This Software is licensed under one of the following >> licenses: >> hch> - * >> hch> - * 1) under the terms of the "Common Public License >> 1.0" a copy of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/cpl.php. >> hch> - * >> hch> - * 2) under the terms of the "The BSD License" a copy >> of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/bsd-license.php. >> hch> - * >> hch> - * 3) under the terms of the "GNU General Public >> License (GPL) Version 2" a >> hch> - * copy of which is available from the Open Source >> Initiative, see >> hch> - * http://www.opensource.org/licenses/gpl-license.php. >> hch> - * >> hch> - * Licensee has the right to choose one of the above licenses. >> hch> - * >> hch> - * Redistributions of source code must retain the above >> copyright >> hch> - * notice and one of the license notices. >> hch> - * >> hch> - * Redistributions in binary form must reproduce both >> the above copyright >> hch> - * notice, one of the license notices in the documentation >> hch> - * and/or other materials provided with the distribution. >> hch> - */ >> hch> - >> hch> -/* >> hch> - * dictionary data structure >> hch> - * >> hch> - * $Id$ >> hch> - */ >> hch> - >> hch> -#ifndef DICTIONARY_H >> hch> -#define DICTIONARY_H >> hch> - >> hch> -#include "core.h" >> hch> - >> hch> >> -/************************************************************ >> ********* >> hch> - * >> * >> hch> - * Structures >> * >> hch> - * >> * >> hch> - >> *********************************************************************/ >> hch> - >> hch> -struct dat_dictionary_entry { >> hch> - struct list_head list; >> hch> - struct dat_provider_info key; >> hch> - void *data; >> hch> -}; >> hch> - >> hch> -struct dat_dictionary { >> hch> - struct list_head list; >> hch> - int size; >> hch> -}; >> hch> - >> hch> >> -/************************************************************ >> ********* >> hch> - * >> * >> hch> - * Macros >> * >> hch> - * >> * >> hch> - >> *********************************************************************/ >> hch> - >> hch> -#define DAT_DICTIONARY_INIT(name) { >> LIST_HEAD_INIT((name).list), 0 } >> hch> - >> hch> >> -/************************************************************ >> ********* >> hch> - * >> * >> hch> - * Function Prototypes >> * >> hch> - * >> * >> hch> - >> *********************************************************************/ >> hch> - >> hch> -extern void dat_dictionary_destroy(struct >> dat_dictionary *dictionary); >> hch> - >> hch> -extern u32 dat_dictionary_size(const struct >> dat_dictionary *dictionary, >> hch> - int *size); >> hch> - >> hch> -extern u32 dat_dictionary_entry_create(struct >> dat_dictionary_entry **p_entry); >> hch> - >> hch> -extern void dat_dictionary_entry_destroy( >> hch> - const struct dat_dictionary_entry *entry); >> hch> - >> hch> -extern u32 dat_dictionary_insert(struct dat_dictionary >> *dictionary, >> hch> - const struct >> dat_provider_info *key, >> hch> - struct >> dat_dictionary_entry *entry, >> hch> - void *data); >> hch> - >> hch> -extern u32 dat_dictionary_search(struct dat_dictionary >> *dictionary, >> hch> - const struct >> dat_provider_info *key, >> hch> - void **p_data); >> hch> - >> hch> -extern u32 dat_dictionary_enumerate(struct >> dat_dictionary *dictionary, >> hch> - void *array[], >> const int array_size); >> hch> - >> hch> -extern u32 dat_dictionary_remove(struct dat_dictionary >> *dictionary, >> hch> - const struct >> dat_provider_info *key, >> hch> - struct >> dat_dictionary_entry **p_entry, >> hch> - void **p_data); >> hch> - >> hch> -#endif /* DICTIONARY_H */ >> hch> Index: linux-kernel/dat/dr.h >> hch> >> =================================================================== >> hch> --- linux-kernel/dat/dr.h (revision 2647) >> hch> +++ linux-kernel/dat/dr.h (working copy) >> hch> @@ -1,57 +0,0 @@ >> hch> -/* >> hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All >> rights reserved. >> hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights >> reserved. >> hch> - * >> hch> - * This Software is licensed under one of the following >> licenses: >> hch> - * >> hch> - * 1) under the terms of the "Common Public License >> 1.0" a copy of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/cpl.php. >> hch> - * >> hch> - * 2) under the terms of the "The BSD License" a copy >> of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/bsd-license.php. >> hch> - * >> hch> - * 3) under the terms of the "GNU General Public >> License (GPL) Version 2" a >> hch> - * copy of which is available from the Open Source >> Initiative, see >> hch> - * http://www.opensource.org/licenses/gpl-license.php. >> hch> - * >> hch> - * Licensee has the right to choose one of the above licenses. >> hch> - * >> hch> - * Redistributions of source code must retain the above >> copyright >> hch> - * notice and one of the license notices. >> hch> - * >> hch> - * Redistributions in binary form must reproduce both >> the above copyright >> hch> - * notice, one of the license notices in the documentation >> hch> - * and/or other materials provided with the distribution. >> hch> - */ >> hch> - >> hch> -/* >> hch> - * Dynamic Registry (DR) functions and data stucture >> hch> - * >> hch> - * $Id$ >> hch> - */ >> hch> - >> hch> -#ifndef DR_H >> hch> -#define DR_H >> hch> - >> hch> -#include >> hch> - >> hch> -extern void dat_dr_fini(void); >> hch> - >> hch> -extern u32 dat_dr_insert(const struct dat_provider_info *info, >> hch> - const DAT_IA_OPEN_FUNC ia_open_func); >> hch> - >> hch> -extern u32 dat_dr_remove(const struct dat_provider_info *info); >> hch> - >> hch> -extern u32 dat_dr_provider_open(const struct >> dat_provider_info *info, >> hch> - DAT_IA_OPEN_FUNC >> *p_ia_open_func); >> hch> - >> hch> -extern u32 dat_dr_provider_close(const struct >> dat_provider_info *info); >> hch> - >> hch> -extern u32 dat_dr_size(int *size); >> hch> - >> hch> -extern u32 dat_dr_list(int max_to_return, int *entries_returned, >> hch> - struct dat_provider_info >> *(dat_provider_list[])); >> hch> - >> hch> -#endif /* DR_H */ >> hch> Index: linux-kernel/dat/Makefile >> hch> >> =================================================================== >> hch> --- linux-kernel/dat/Makefile (revision 2647) >> hch> +++ linux-kernel/dat/Makefile (working copy) >> hch> @@ -5,8 +5,4 @@ >> hch> >> hch> obj-$(CONFIG_DAT) += dat.o >> hch> >> hch> -dat-y := \ >> hch> - api.o \ >> hch> - core.o \ >> hch> - dictionary.o \ >> hch> - dr.o >> hch> +dat-y += api.o >> hch> Index: linux-kernel/dat/core.h >> hch> >> =================================================================== >> hch> --- linux-kernel/dat/core.h (revision 2647) >> hch> +++ linux-kernel/dat/core.h (working copy) >> hch> @@ -1,61 +0,0 @@ >> hch> -/* >> hch> - * Copyright (c) 2002-2005, Network Appliance, Inc. All >> rights reserved. >> hch> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights >> reserved. >> hch> - * >> hch> - * This Software is licensed under one of the following >> licenses: >> hch> - * >> hch> - * 1) under the terms of the "Common Public License >> 1.0" a copy of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/cpl.php. >> hch> - * >> hch> - * 2) under the terms of the "The BSD License" a copy >> of which is >> hch> - * available from the Open Source Initiative, see >> hch> - * http://www.opensource.org/licenses/bsd-license.php. >> hch> - * >> hch> - * 3) under the terms of the "GNU General Public >> License (GPL) Version 2" a >> hch> - * copy of which is available from the Open Source >> Initiative, see >> hch> - * http://www.opensource.org/licenses/gpl-license.php. >> hch> - * >> hch> - * Licensee has the right to choose one of the above licenses. >> hch> - * >> hch> - * Redistributions of source code must retain the above >> copyright >> hch> - * notice and one of the license notices. >> hch> - * >> hch> - * Redistributions in binary form must reproduce both >> the above copyright >> hch> - * notice, one of the license notices in the documentation >> hch> - * and/or other materials provided with the distribution. >> hch> - */ >> hch> - >> hch> -/* >> hch> - * core DAT registry functions and data structures >> hch> - * >> hch> - * $Id$ >> hch> - */ >> hch> - >> hch> -#ifndef CORE_H >> hch> -#define CORE_H >> hch> - >> hch> -#include >> hch> -#include >> hch> -#include >> hch> -#include >> hch> -#include >> hch> - >> hch> -#include >> hch> - >> hch> -#define DAT_IS_BAD_POINTER(p) ( NULL == (p) ) >> hch> - >> hch> -typedef enum { >> hch> - DAT_DBG_TYPE_ERROR = (1 << 0), >> hch> - DAT_DBG_TYPE_GENERIC = (1 << 1), >> hch> - DAT_DBG_TYPE_DR = (1 << 2), >> hch> - DAT_DBG_TYPE_PROVIDER_API = (1 << 3), >> hch> - DAT_DBG_TYPE_CONSUMER_API = (1 << 4), >> hch> - DAT_DBG_TYPE_ALL = 0x1f >> hch> -} DAT_DBG_TYPE; >> hch> - >> hch> -typedef int DAT_DBG_MASK; >> hch> - >> hch> -extern void dat_dbg_print(DAT_DBG_TYPE type, const char >> *fmt, ...); >> hch> - >> hch> -#endif /* CORE_H */ >> hch> >> _______________________________________________ >> openib-general mailing list >> openib-general at openib.org >> http://openib.org/mailman/listinfo/openib-general >> >> To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From iod00d at hp.com Thu Jun 23 12:01:38 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 23 Jun 2005 12:01:38 -0700 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <20050623164555.GB12420@mellanox.co.il> References: <1119539738.4617.2.camel@localhost.localdomain> <52psudnjor.fsf@topspin.com> <1119543293.4477.10.camel@localhost.localdomain> <52aclhnio2.fsf@topspin.com> <20050623164555.GB12420@mellanox.co.il> Message-ID: <20050623190138.GH22934@esmail.cup.hp.com> On Thu, Jun 23, 2005 at 07:45:55PM +0300, Michael S. Tsirkin wrote: > It seems that compiler generates a single instruction for this shift. If so, then gcc or the code gen backend should generate a warning about overflowing the encoding. This sounds like a toolchain bug. > And CPU designers apparently have cut a corner by simply using only low 6 bits > in the operand of a shift, so that 1ull<<64 is same as 1ull<<(64&0x37), or 1. It might be the instruction encoding is the limiting factor and not the CPU implementation. I don't have my dead-tree IA32 arch books to look up the encoding. Ie gcc might need to learn a new instruction for 64-bit platforms. I've run into this on the parisc support in the gnu tool chain. grant From jlentini at netapp.com Thu Jun 23 12:04:36 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 15:04:36 -0400 (EDT) Subject: [openib-general] Re: Setting kdapl ep_state relative to calling disconnect In-Reply-To: <1119538391.4476.24.camel@localhost.localdomain> References: <1119538391.4476.24.camel@localhost.localdomain> Message-ID: We should make them consistent. The state should be updated before the disconnect is sent. On Thu, 23 Jun 2005, Hal Rosenstock wrote: halr> Hi, halr> halr> In dapl_evd.c, there are cases where the ep_state is updated before halr> calling disconnect and others where this is done after. Should these be halr> made consistent ? halr> halr> void dapl_evd_connection_callback(...) halr> { halr> ... halr> case DAT_CONNECTION_EVENT_DISCONNECTED: halr> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> dapl_ib_disconnect_clean(ep); halr> break; halr> case DAT_CONNECTION_EVENT_PEER_REJECTED: halr> case DAT_CONNECTION_EVENT_UNREACHABLE: halr> case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: halr> case DAT_CONNECTION_EVENT_BROKEN: halr> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); halr> dapl_ib_disconnect_clean(ep); halr> break; halr> halr> void dapl_evd_connection_callback(...) halr> { halr> dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); halr> spin_lock_irqsave(&ep->common.lock, ep->common.flags); halr> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> spin_unlock_irqrestore(&ep->common.lock, halr> ep->common.flags); halr> halr> -- Hal halr> From tduffy at sun.com Thu Jun 23 12:11:16 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 23 Jun 2005 12:11:16 -0700 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> <42BAED53.80206@linuxmachines.com> <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> Message-ID: <1119553876.15654.1.camel@duffman> On Thu, 2005-06-23 at 13:54 -0400, Talpey, Thomas wrote: > We look forward to running NFS/RDMA over OpenIB, when its kDAPL > is ready. Let's get NFSoRDMA going sooner rather than later on James's kDAPL. I think this will both be a good test case as well as a vehicle to demonstrate the functionality of kDAPL. What can I do to help? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Thu Jun 23 12:14:34 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 15:14:34 -0400 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <521x6tnhzu.fsf@topspin.com> References: <1119539738.4617.2.camel@localhost.localdomain> <52psudnjor.fsf@topspin.com> <1119543293.4477.10.camel@localhost.localdomain> <52aclhnio2.fsf@topspin.com> <521x6tnhzu.fsf@topspin.com> Message-ID: <1119554073.4477.174.camel@localhost.localdomain> On Thu, 2005-06-23 at 12:41, Roland Dreier wrote: > Roland> It's odd that you saw the expession end up as 0. It seems > Roland> that on every system I have, 1ull << 64 is 0, and 0 - 1 == > Roland> -1 == 0xffffffffffffffff. So although it isn't correct C, > Roland> it should work. > > Oh, I see. A constant left shift of 64 is evaluated to 0 at compile > time. But a variable left shift that ends up as 64 indeed gives the > wrong answer. So you're right, packer will get the wrong answer. > > I think this should fix it properly: It does. Thanks. -- Hal From Thomas.Talpey at netapp.com Thu Jun 23 12:22:44 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Thu, 23 Jun 2005 15:22:44 -0400 Subject: [openib-general] NFS/RDMA/kDAPL In-Reply-To: <1119553876.15654.1.camel@duffman> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> <42BAED53.80206@linuxmachines.com> <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> <1119553876.15654.1.camel@duffman> Message-ID: <6.2.1.2.2.20050623151939.03d80900@exnane01.nane.netapp.com> At 03:11 PM 6/23/2005, Tom Duffy wrote: >On Thu, 2005-06-23 at 13:54 -0400, Talpey, Thomas wrote: >> We look forward to running NFS/RDMA over OpenIB, when its kDAPL >> is ready. > >Let's get NFSoRDMA going sooner rather than later on James's kDAPL. I >think this will both be a good test case as well as a vehicle to >demonstrate the functionality of kDAPL. > >What can I do to help? Can the Linux OpenIB client connect to Solaris 10? If so, we might consider using Sol10's NFS/RDMA server. If not, we'll have to use a NetApp filer (which is fine by me but maybe hard for you), because the CITI NFS/RDMA server is accepting connections but not yet processing RPCs. Tom. From tduffy at sun.com Thu Jun 23 12:33:12 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 23 Jun 2005 12:33:12 -0700 Subject: [openib-general] Re: NFS/RDMA/kDAPL In-Reply-To: <6.2.1.2.2.20050623151939.03d80900@exnane01.nane.netapp.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> <42BAED53.80206@linuxmachines.com> <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> <1119553876.15654.1.camel@duffman> <6.2.1.2.2.20050623151939.03d80900@exnane01.nane.netapp.com> Message-ID: <1119555192.15654.5.camel@duffman> On Thu, 2005-06-23 at 15:22 -0400, Talpey, Thomas wrote: > Can the Linux OpenIB client connect to Solaris 10? Yes. IPoIB works now. You can connect a Solaris 10 client to an OpenSM managed network. > If so, we might > consider using Sol10's NFS/RDMA server. OpenSolaris would be more accessible to the wider community. > If not, we'll have to use a > NetApp filer (which is fine by me but maybe hard for you), I have a really old NetApp (F840) in my lab that probably doens't support this. > because > the CITI NFS/RDMA server is accepting connections but not yet > processing RPCs. Bummer. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Thu Jun 23 12:37:50 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 12:37:50 -0700 Subject: [openib-general] Re: [PATCH] Fix ib_pack/unpack for 64 bits In-Reply-To: <20050623190138.GH22934@esmail.cup.hp.com> (Grant Grundler's message of "Thu, 23 Jun 2005 12:01:38 -0700") References: <1119539738.4617.2.camel@localhost.localdomain> <52psudnjor.fsf@topspin.com> <1119543293.4477.10.camel@localhost.localdomain> <52aclhnio2.fsf@topspin.com> <20050623164555.GB12420@mellanox.co.il> <20050623190138.GH22934@esmail.cup.hp.com> Message-ID: <52slz8lv8x.fsf@topspin.com> Grant> If so, then gcc or the code gen backend should generate a Grant> warning about overflowing the encoding. This sounds like a Grant> toolchain bug. gcc does generate a warning for constant shifts that are too big, as in: bit = 1ull << 64; No diagnostic is generated when the shift is a variable, as in bit = 1ull << shift; If at runtime, shift is bigger than the width of the unsigned long long type, then this expression is undefined according to the C spec. So the gcc behavior is perfectly correct. - R. From tduffy at sun.com Thu Jun 23 12:41:41 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 23 Jun 2005 12:41:41 -0700 Subject: [openib-general] [PATCH] kDAPL: add a couple more static Message-ID: <1119555701.18281.1.camel@duffman> Signed-off-by: Tom Duffy Index: linux-kernel-static/dat-provider/dapl_cr.c =================================================================== --- linux-kernel-static/dat-provider/dapl_cr.c (revision 2695) +++ linux-kernel-static/dat-provider/dapl_cr.c (working copy) @@ -41,7 +41,7 @@ /* * Create a CR. Part of the passive side of a connection */ -struct dapl_cr *dapl_cr_alloc(struct dapl_ia *ia) +static struct dapl_cr *dapl_cr_alloc(struct dapl_ia *ia) { struct dapl_cr *cr; @@ -305,9 +305,9 @@ static u32 dapl_connection_request(struc * up resources and obtain the EP pointer associated with a CR in * the SP. */ -struct dapl_ep *dapl_get_sp_ep(struct dapl_cm_ctx *cm_ctx, - struct dapl_sp *sp, - enum dat_event_number event) +static struct dapl_ep *dapl_get_sp_ep(struct dapl_cm_ctx *cm_ctx, + struct dapl_sp *sp, + enum dat_event_number event) { struct dapl_cr *cr; struct dapl_ep *ep; From tomduffy at gmail.com Thu Jun 23 13:06:47 2005 From: tomduffy at gmail.com (Tom Duffy) Date: Thu, 23 Jun 2005 13:06:47 -0700 Subject: [openib-general] SDP: still getting sk_alloc() panic, any ideas? Message-ID: <9d3b7de7050623130658aa92@mail.gmail.com> I am still getting the panic when you try to connect to a machine and it is not listening (but has ib_sdp loaded): [root at sins-stinger-10 ~]# ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at "/build1/tduffy/openib-work/linux-2.6.12-openib/in:352 invalid operand: 0000 [1] SMP CPU 1 Modules linked in: ib_sdp kdapltest ib_dat_provider dat ib_at ib_cm md5 ipv6 parport_pc lp parport autofs4 nfs lockd rfcomm l2cap bluetooth pcmcia yenta_socket rsrc_nonstatic pcmcia_core sunrpc ext3 jbd dm_mod video container button battery ac ohci_hcd tpm_atmel tpm i2c_amd756 i2c_core shpchp ib_mthca ib_ipoib ib_sa ib_mad ib_core tg3 floppy xfs exportfs mptscsih mptbase sd_mod scsi_mod Pid: 3683, comm: ib_cm/1 Not tainted 2.6.12openib RIP: 0010:[] {sk_alloc+297} RSP: 0018:ffff81003c7bdc68 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8100668bf750 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 000000000000001b RDI: ffffffff8834f200 RBP: ffffffff8834ee00 R08: 0000000000000000 R09: ffff8100668bf750 R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8834e680 R13: 00000000000000d0 R14: 0000000000000001 R15: 000000000000001b FS: 00002aaaaaf01d00(0000) GS:ffffffff804e7000(0000) knlGS:0000000055587260 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00007fffffd9815c CR3: 000000007cade000 CR4: 00000000000006e0 Process ib_cm/1 (pid: 3683, threadinfo ffff81003c7bc000, task ffff8100416a8720) Stack: ffff81007e24d910 ffff81006f2f9c58 0000000100000001 ffff810037d81aec 0000000000000000 ffff81005132e330 ffff81000adf52d8 ffff81005132e330 ffff81005132e388 ffffffff8833dc93 Call Trace:{:ib_sdp:sdp_conn_alloc+35} {:ib_sdp:sdp_cm_req_handler+2111} {check_spinlock_acquired+24} {:ib_sdp:sdp_cm_event_handler+175} {:ib_cm:cm_process_work+50} {:ib_cm:cm_work_handler+1881} {:ib_cm:cm_work_handler+0} {worker_thread+476} {default_wake_function+0} {keventd_create_kthread+0} {worker_thread+0} {keventd_create_kthread+0} {kthread+217} {schedule_tail+64} {child_rip+8} {keventd_create_kthread+0} {flat_send_IPI_mask+0} {kthread+0} {child_rip+0} Code: 0f 0b b8 03 37 80 ff ff ff ff 60 01 65 8b 04 25 34 00 00 00 RIP {sk_alloc+297} RSP Message from syslogd at sins-stinger-10 at Thu Jun 23 11:53:12 2005 ... sins-stinger-10 kernel: invalid operand: 0000 [1] SMP Any idea about this? -tduffy From tomduffy at gmail.com Thu Jun 23 13:11:17 2005 From: tomduffy at gmail.com (Tom Duffy) Date: Thu, 23 Jun 2005 13:11:17 -0700 Subject: [openib-general] NFS/RDMA/kDAPL In-Reply-To: <6.2.1.2.2.20050623151939.03d80900@exnane01.nane.netapp.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> <42BAED53.80206@linuxmachines.com> <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> <1119553876.15654.1.camel@duffman> <6.2.1.2.2.20050623151939.03d80900@exnane01.nane.netapp.com> Message-ID: <9d3b7de705062313111bf32bef@mail.gmail.com> On 6/23/05, Talpey, Thomas wrote: > Can the Linux OpenIB client connect to Solaris 10? If so, we might > consider using Sol10's NFS/RDMA server. If not, we'll have to use a > NetApp filer (which is fine by me but maybe hard for you), because > the CITI NFS/RDMA server is accepting connections but not yet > processing RPCs. I am new to NFSoRDMA. Where do I start? Is there a good doc about how to set it up? Tom, would you be so kind as to start a tiki page about it? Just click on NFSoRDMA on this page and go: https://openib.org/tiki/tiki-index.php Thanks, -tduffy From halr at voltaire.com Thu Jun 23 13:11:30 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 Jun 2005 16:11:30 -0400 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> <42BAED53.80206@linuxmachines.com> <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> Message-ID: <1119557114.4477.334.camel@localhost.localdomain> > Brief intro to NFS/RDMA work: > > Current client version is on Sourceforge, supporting various flavors of > 2.4. I'm preparing a new release for 2.6. > > > The client needs a patched version of Sunrpc in order to hook in as > an NFS transport. This patch is being used by the NFS/IPv6 > project as well, and it's planned for future kernel.org integration. > > (There are also patch sets for earlier Linux revs) > > Server version for 2.6 is under development at UMich CITI, scroll > down to Documents and Code sections of this page: > This seems like good info for the OpenIB wiki page on NFSoRDMA. -- Hal > Finally, NetApp and Sun have demonstrated implementations, > Solaris 10 has support for it. > > We look forward to running NFS/RDMA over OpenIB, when its kDAPL > is ready. > > Tom. From jlentini at netapp.com Thu Jun 23 13:17:36 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 16:17:36 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: add a couple more static In-Reply-To: <1119555701.18281.1.camel@duffman> References: <1119555701.18281.1.camel@duffman> Message-ID: Committed in revision 2703. On Thu, 23 Jun 2005, Tom Duffy wrote: tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-static/dat-provider/dapl_cr.c tduffy> =================================================================== tduffy> --- linux-kernel-static/dat-provider/dapl_cr.c (revision 2695) tduffy> +++ linux-kernel-static/dat-provider/dapl_cr.c (working copy) tduffy> @@ -41,7 +41,7 @@ tduffy> /* tduffy> * Create a CR. Part of the passive side of a connection tduffy> */ tduffy> -struct dapl_cr *dapl_cr_alloc(struct dapl_ia *ia) tduffy> +static struct dapl_cr *dapl_cr_alloc(struct dapl_ia *ia) tduffy> { tduffy> struct dapl_cr *cr; tduffy> tduffy> @@ -305,9 +305,9 @@ static u32 dapl_connection_request(struc tduffy> * up resources and obtain the EP pointer associated with a CR in tduffy> * the SP. tduffy> */ tduffy> -struct dapl_ep *dapl_get_sp_ep(struct dapl_cm_ctx *cm_ctx, tduffy> - struct dapl_sp *sp, tduffy> - enum dat_event_number event) tduffy> +static struct dapl_ep *dapl_get_sp_ep(struct dapl_cm_ctx *cm_ctx, tduffy> + struct dapl_sp *sp, tduffy> + enum dat_event_number event) tduffy> { tduffy> struct dapl_cr *cr; tduffy> struct dapl_ep *ep; tduffy> From jlentini at netapp.com Thu Jun 23 13:23:14 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 16:23:14 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] Integrate dapl_hca_link_ia/dapl_hca_unlink_ia into dapl_ia.c In-Reply-To: <20050620084756.GA7088@mellanox.co.il> References: <20050620084756.GA7088@mellanox.co.il> Message-ID: Do we even need to link each IA onto an HCA? I don't see where in the code we use an HCA's ia_list. Is this just dead code? On Mon, 20 Jun 2005, Itamar Rabenstein wrote: itamar> Integrate dapl_hca_link_ia/dapl_hca_unlink_ia into dapl_ia.c itamar> (no need for functions that just call LIST_ADD and LIST_DEL) itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> diff -Nurp -X dontdiff dat-provider_simp/dapl_hca_util.c dat-provider/dapl_hca_util.c itamar> --- dat-provider_simp/dapl_hca_util.c Sun Jun 19 18:36:31 2005 itamar> +++ dat-provider/dapl_hca_util.c Sun Jun 19 18:38:33 2005 itamar> @@ -88,49 +88,3 @@ void dapl_hca_free(struct dapl_hca *hca) itamar> { itamar> kfree(hca); itamar> } itamar> - itamar> -/* itamar> - * dapl_hca_link_ia itamar> - * itamar> - * Add an ia to the HCA structure itamar> - * itamar> - * Input: itamar> - * hca itamar> - * ia_ptr itamar> - * itamar> - * Output: itamar> - * none itamar> - * itamar> - * Returns: itamar> - * none itamar> - * itamar> - */ itamar> -void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia_ptr) itamar> -{ itamar> - spin_lock_irqsave(&hca->lock, hca->flags); itamar> - list_add(&ia_ptr->list, &hca->ia_list); itamar> - spin_unlock_irqrestore(&hca->lock, hca->flags); itamar> -} itamar> - itamar> -/* itamar> - * dapl_hca_unlink_ia itamar> - * itamar> - * Remove an ia from the hca info structure itamar> - * itamar> - * Input: itamar> - * hca itamar> - * ia_ptr itamar> - * itamar> - * Output: itamar> - * none itamar> - * itamar> - * Returns: itamar> - * none itamar> - * itamar> - */ itamar> -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia) itamar> -{ itamar> - spin_lock_irqsave(&hca->lock, hca->flags); itamar> - list_del(&ia->list); itamar> - spin_unlock_irqrestore(&hca->lock, hca->flags); itamar> -} itamar> diff -Nurp -X dontdiff dat-provider_simp/dapl_hca_util.h dat-provider/dapl_hca_util.h itamar> --- dat-provider_simp/dapl_hca_util.h Sun Jun 19 18:36:31 2005 itamar> +++ dat-provider/dapl_hca_util.h Sun Jun 19 18:36:45 2005 itamar> @@ -39,8 +39,4 @@ struct dapl_hca *dapl_hca_alloc(char *na itamar> itamar> void dapl_hca_free(struct dapl_hca *hca); itamar> itamar> -void dapl_hca_link_ia(struct dapl_hca *hca, struct dapl_ia *ia); itamar> - itamar> -void dapl_hca_unlink_ia(struct dapl_hca *hca, struct dapl_ia *ia); itamar> - itamar> #endif itamar> diff -Nurp -X dontdiff dat-provider_simp/dapl_ia.c dat-provider/dapl_ia.c itamar> --- dat-provider_simp/dapl_ia.c Sun Jun 19 18:36:31 2005 itamar> +++ dat-provider/dapl_ia.c Sun Jun 19 18:39:03 2005 itamar> @@ -34,7 +34,6 @@ itamar> #include "dapl_ia.h" itamar> #include "dapl_provider.h" itamar> #include "dapl_evd.h" itamar> -#include "dapl_hca_util.h" itamar> #include "dapl_openib_util.h" itamar> #include "dapl_sp.h" itamar> #include "dapl_cr.h" itamar> @@ -71,7 +70,9 @@ static struct dapl_ia *dapl_ia_alloc(str itamar> INIT_LIST_HEAD(&ia->psp_list); itamar> INIT_LIST_HEAD(&ia->srq_list); itamar> itamar> - dapl_hca_link_ia(hca, ia); itamar> + spin_lock_irqsave(&hca->lock, hca->flags); itamar> + list_add(&ia->list, &hca->ia_list); itamar> + spin_unlock_irqrestore(&hca->lock, hca->flags); itamar> itamar> return ia; itamar> } itamar> @@ -379,7 +380,9 @@ void dapl_ia_free(struct dapl_ia *ia) itamar> dapl_os_assert(list_empty(&ia->psp_list)); itamar> dapl_os_assert(list_empty(&ia->rsp_list)); itamar> itamar> - dapl_hca_unlink_ia(ia->hca, ia); itamar> + spin_lock_irqsave(&ia->hca->lock, ia->hca->flags); itamar> + list_del(&ia->list); itamar> + spin_unlock_irqrestore(&ia->hca->lock, ia->hca->flags); itamar> /* no need to destroy ia->common.lock */ itamar> itamar> kfree(ia); itamar> -- itamar> Itamar itamar> From tduffy at sun.com Thu Jun 23 13:25:25 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 23 Jun 2005 13:25:25 -0700 Subject: [openib-general] [PATCH] kDAPL: remove dapl_os_assert() Message-ID: <1119558325.21702.1.camel@duffman> This patch removes dapl_os_assert(). In most cases, replacing with BUG_ON(!). Some cases, I just called panic() where others I removed the assert all together because the next line dereferences the pointer anyways. Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel/dat-provider/dapl_cookie.c (revision 2701) +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) @@ -184,7 +184,7 @@ u32 dapl_cb_get(struct dapl_cookie_buffe u32 dat_status; int new_head; - dapl_os_assert(NULL != cookie_ptr); + BUG_ON(cookie_ptr == NULL); new_head = (atomic_read(&buffer->head) + 1) % buffer->pool_size; Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2701) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -783,7 +783,7 @@ u32 dapl_ib_get_gid(struct ib_device *hc { int status; - dapl_os_assert(hca); + BUG_ON(!hca); if (gid) { status = ib_query_gid(hca, port, 0, gid); Index: linux-kernel/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2701) +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) @@ -76,7 +76,6 @@ u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ attr = &ep_ptr->param.ep_attr; - dapl_os_assert(ep_ptr->param.pz != NULL); ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz)->pd; ib_hca_handle = ia_ptr->hca->ib_hca_handle; Index: linux-kernel/dat-provider/dapl_openib_dto.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_dto.h (revision 2701) +++ linux-kernel/dat-provider/dapl_openib_dto.h (working copy) @@ -43,8 +43,6 @@ static inline u32 dapl_ib_post_recv(stru struct ib_sge *sg_list; int status, i, total_len = 0; - dapl_os_assert(NULL != cookie); - sg_list = ep->recv_iov; for (i = 0; i < num_segments; i++, sg_list++) { sg_list->addr = local_iov[i].virtual_address; @@ -84,8 +82,6 @@ static inline u32 dapl_ib_post_send(stru struct ib_sge *sg_list; int status, i, total_len = 0; - dapl_os_assert(NULL != cookie); - sg_list = ep->send_iov; for (i = 0; i < num_segments; i++, sg_list++) { sg_list->addr = local_iov[i].virtual_address; Index: linux-kernel/dat-provider/dapl_ia.c =================================================================== --- linux-kernel/dat-provider/dapl_ia.c (revision 2701) +++ linux-kernel/dat-provider/dapl_ia.c (working copy) @@ -371,13 +371,13 @@ bail: */ void dapl_ia_free(struct dapl_ia *ia) { - dapl_os_assert(ia->async_error_evd == NULL); - dapl_os_assert(list_empty(&ia->lmr_list)); - dapl_os_assert(list_empty(&ia->rmr_list)); - dapl_os_assert(list_empty(&ia->ep_list)); - dapl_os_assert(list_empty(&ia->evd_list)); - dapl_os_assert(list_empty(&ia->psp_list)); - dapl_os_assert(list_empty(&ia->rsp_list)); + BUG_ON(ia->async_error_evd != NULL); + BUG_ON(!list_empty(&ia->lmr_list)); + BUG_ON(!list_empty(&ia->rmr_list)); + BUG_ON(!list_empty(&ia->ep_list)); + BUG_ON(!list_empty(&ia->evd_list)); + BUG_ON(!list_empty(&ia->psp_list)); + BUG_ON(!list_empty(&ia->rsp_list)); dapl_hca_unlink_ia(ia->hca, ia); /* no need to destroy ia->common.lock */ Index: linux-kernel/dat-provider/dapl_rmr.c =================================================================== --- linux-kernel/dat-provider/dapl_rmr.c (revision 2701) +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) @@ -126,8 +126,7 @@ static u64 dapl_rmr_get_address(DAT_REGI * DAT_MEM_TYPE_IA * DAT_MEM_TYPE_BYPASS */ - dapl_os_assert(0); - return 0; + panic("unimplemented or unknown memory type\n"); } } Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2701) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -675,8 +675,8 @@ u32 dapl_ep_free(struct dat_ep *ep) */ (void)dapl_ep_disconnect((struct dat_ep *)ep_ptr, DAT_CLOSE_ABRUPT_FLAG); - dapl_os_assert(ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || - ep_ptr->param.ep_state == DAT_EP_STATE_UNCONNECTED); + BUG_ON(ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECTED && + ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED); /* * Do verification of parameters and the state change atomically. @@ -1538,11 +1538,11 @@ u32 dapl_ep_modify(struct dat_ep *ep, en * occurred. But they're important to the logic of this routine, * so we check. */ - dapl_os_assert(ep1 == ep2); - dapl_os_assert(ep_attr2.max_recv_dtos == ep_attr1.max_recv_dtos); - dapl_os_assert(ep_attr2.max_request_dtos == ep_attr1.max_request_dtos); - dapl_os_assert(ep_attr2.max_recv_iov == ep_attr1.max_recv_iov); - dapl_os_assert(ep_attr2.max_request_iov == ep_attr1.max_request_iov); + BUG_ON(ep1 != ep2); + BUG_ON(ep_attr2.max_recv_dtos != ep_attr1.max_recv_dtos); + BUG_ON(ep_attr2.max_request_dtos != ep_attr1.max_request_dtos); + BUG_ON(ep_attr2.max_recv_iov != ep_attr1.max_recv_iov); + BUG_ON(ep_attr2.max_request_iov != ep_attr1.max_request_iov); copy_of_old_ep = *ep2; @@ -1607,10 +1607,10 @@ u32 dapl_ep_modify(struct dat_ep *ep, en * because the parameter validate routine should protect us, * but it's an important enough point that we assert it. */ - dapl_os_assert((ep2->param.ep_state - != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING) - && (ep2->param.ep_state - != DAT_EP_STATE_ACTIVE_CONNECTION_PENDING)); + BUG_ON(ep2->param.ep_state == + DAT_EP_STATE_PASSIVE_CONNECTION_PENDING || + ep2->param.ep_state == + DAT_EP_STATE_ACTIVE_CONNECTION_PENDING); new_ep.qp = alloc_ep.qp; } Index: linux-kernel/dat-provider/dapl_cr.c =================================================================== --- linux-kernel/dat-provider/dapl_cr.c (revision 2701) +++ linux-kernel/dat-provider/dapl_cr.c (working copy) @@ -271,7 +271,7 @@ static u32 dapl_connection_request(struc DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING; } else { /* RSP */ - dapl_os_assert(sp->sp.type == DAT_SP_TYPE_RSP); + BUG_ON(sp->sp.type != DAT_SP_TYPE_RSP); ep->param.ep_state = DAT_EP_STATE_PASSIVE_CONNECTION_PENDING; } @@ -516,8 +516,7 @@ void dapl_cr_callback(struct dapl_cm_ctx break; default: evd = NULL; - dapl_os_assert(0); /* shouldn't happen */ - break; + panic("unknown callback event\n"); } if (evd != NULL) Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2701) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -243,7 +243,7 @@ static u32 dapl_evd_dealloc(struct dapl_ u32 status = DAT_SUCCESS; struct dapl_ia *ia; - dapl_os_assert(atomic_read(&evd->evd_ref_count) == 0); + BUG_ON(atomic_read(&evd->evd_ref_count) != 0); /* * Destroy the CQ first, to keep any more callbacks from coming @@ -330,9 +330,9 @@ static void dapl_evd_post_event(struct d event->event_number); status = dapl_rbuf_add(&evd->pending_event_queue, event); - dapl_os_assert(status == DAT_SUCCESS); + BUG_ON(status != DAT_SUCCESS); - dapl_os_assert(evd->evd_state == DAPL_EVD_STATE_OPEN); + BUG_ON(evd->evd_state != DAPL_EVD_STATE_OPEN); if (evd->evd_producer_locking_needed) spin_unlock_irqrestore(&evd->common.lock, @@ -522,10 +522,9 @@ static void dapl_evd_wc_to_event(struct dto_status = dapl_ib_get_dto_status(wc); cookie = (struct dapl_cookie *) (unsigned long) wc->wr_id; - dapl_os_assert(NULL != cookie); ep = cookie->ep; - dapl_os_assert(NULL != ep); + BUG_ON(ep == NULL); event->evd = (struct dat_evd *)evd; @@ -550,26 +549,6 @@ static void dapl_evd_wc_to_event(struct event_data->user_cookie = cookie->val.dto.cookie; event_data->status = dto_status; -#if 0 - /* Currently mthca is not setting the opcode in */ - /* succesful wc. The opcode will be IB_WC_SEND or */ - /* IB_WC_RECV according the is_send bit */ - /* We can not check the following assert for now */ - if (dto_status == DAT_DTO_SUCCESS) { - enum ib_wc_opcode ib_opcode = wc->opcode; - enum dapl_dto_type dto_type = cookie->val.dto.type; - dapl_os_assert((ib_opcode == IB_WC_SEND && - dto_type == DAPL_DTO_TYPE_SEND) || - (ib_opcode == IB_WC_RECV && - dto_type == DAPL_DTO_TYPE_RECV) || - (ib_opcide == IB_WC_RDMA_WRITE && - dto_type == DAPL_DTO_TYPE_RDMA_WRITE) || - (ib_opcode == IB_WC_RDMA_READ && - dto_type == DAPL_DTO_TYPE_RDMA_READ)); - - } -#endif - if (cookie->val.dto.type == DAPL_DTO_TYPE_SEND || cookie->val.dto.type == DAPL_DTO_TYPE_RDMA_WRITE) { /* Get size from DTO; CQE value may be off. */ @@ -593,7 +572,7 @@ static void dapl_evd_wc_to_event(struct event_data->user_cookie = cookie->val.rmr.cookie; if (dto_status == DAT_DTO_SUCCESS) { - dapl_os_assert(wc->opcode == IB_WC_BIND_MW); + BUG_ON(wc->opcode != IB_WC_BIND_MW); event_data->status = DAT_RMR_BIND_SUCCESS; } else { dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, @@ -607,10 +586,7 @@ static void dapl_evd_wc_to_event(struct break; } default: - { - dapl_os_assert(!"Invalid Operation type"); - break; - } + panic("Invalid Operation type\n"); } /* end switch */ } @@ -652,7 +628,7 @@ void dapl_evd_qp_async_error_callback(st ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; } - dapl_os_assert(async_evd != NULL); + BUG_ON(async_evd == NULL); status = dapl_ib_get_async_event(cause, &async_event); if (status == DAT_SUCCESS) { @@ -792,9 +768,7 @@ void dapl_evd_connection_callback(struct default: spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); evd = NULL; - - dapl_os_assert(0); /* shouldn't happen */ - break; + panic("bad event\n"); } /* @@ -838,7 +812,7 @@ static void dapl_evd_dto_callback(struct evd = (struct dapl_evd *)user_context; - dapl_os_assert(evd->cq == cq); + BUG_ON(evd->cq != cq); /* Read once. */ state = *(volatile enum dapl_evd_state *)&evd->evd_state; Index: linux-kernel/dat-provider/dapl_util.h =================================================================== --- linux-kernel/dat-provider/dapl_util.h (revision 2701) +++ linux-kernel/dat-provider/dapl_util.h (working copy) @@ -48,15 +48,6 @@ #include #endif -#define dapl_os_assert(expression) \ - do { \ - if (!(expression)) { \ - panic("ASSERTION fail in %s:%i:%s\n", \ - __FILE__, __LINE__, __func__); \ - } \ - } while (0) - - /* dapl_os_atomic_assign * * assign 'new_value' to '*v' if the current value Index: linux-kernel/dat-provider/dapl_sp.c =================================================================== --- linux-kernel/dat-provider/dapl_sp.c (revision 2701) +++ linux-kernel/dat-provider/dapl_sp.c (working copy) @@ -64,7 +64,7 @@ static struct dapl_sp *dapl_sp_alloc(str void dapl_sp_dealloc(struct dapl_sp *sp) { - dapl_os_assert(list_empty(&sp->cr_list)); + BUG_ON(!list_empty(&sp->cr_list)); kfree(sp); } From jlentini at netapp.com Thu Jun 23 13:27:09 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 16:27:09 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c In-Reply-To: <20050620085010.GA7206@mellanox.co.il> References: <20050620085010.GA7206@mellanox.co.il> Message-ID: For consistency, I think we should keep the HCA object in its own file. However, I'm not sure we need an HCA object. Is there a better way to organize the data being stored in the dapl_hca structure? I have this feeling that we should merge all the structures that we create on a per-hca basis (the dapl_provider_entry, dat_provider table, and dapl_hca). james On Mon, 20 Jun 2005, Itamar Rabenstein wrote: itamar> ntegrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c itamar> (no need for 2 files just for 2 simple function that kmalloc and kfree. itamar> There is not any special logic in this functions that need to separate itamar> them into different files) itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> diff -Nurp -X dontdiff dat-provider_link_ia/Makefile dat-provider/Makefile itamar> --- dat-provider_link_ia/Makefile Sun Jun 19 18:43:16 2005 itamar> +++ dat-provider/Makefile Sun Jun 19 19:45:15 2005 itamar> @@ -20,7 +20,6 @@ PROVIDER_MODULES := \ itamar> dapl_cr \ itamar> dapl_ep \ itamar> dapl_evd \ itamar> - dapl_hca_util \ itamar> dapl_ia \ itamar> dapl_lmr \ itamar> dapl_provider \ itamar> diff -Nurp -X dontdiff dat-provider_link_ia/dapl_hca_util.c dat-provider/dapl_hca_util.c itamar> --- dat-provider_link_ia/dapl_hca_util.c Sun Jun 19 18:43:16 2005 itamar> +++ dat-provider/dapl_hca_util.c Thu Jan 1 02:00:00 1970 itamar> @@ -1,90 +0,0 @@ itamar> -/* itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. itamar> - * itamar> - * This Software is licensed under one of the following licenses: itamar> - * itamar> - * 1) under the terms of the "Common Public License 1.0" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/cpl.php. itamar> - * itamar> - * 2) under the terms of the "The BSD License" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/bsd-license.php. itamar> - * itamar> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a itamar> - * copy of which is available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/gpl-license.php. itamar> - * itamar> - * Licensee has the right to choose one of the above licenses. itamar> - * itamar> - * Redistributions of source code must retain the above copyright itamar> - * notice and one of the license notices. itamar> - * itamar> - * Redistributions in binary form must reproduce both the above copyright itamar> - * notice, one of the license notices in the documentation itamar> - * and/or other materials provided with the distribution. itamar> - */ itamar> - itamar> -/* itamar> - * $Id: dapl_hca_util.c 2640 2005-06-16 16:22:46Z jlentini $ itamar> - */ itamar> - itamar> -#include "dapl.h" itamar> -#include "dapl_openib_util.h" itamar> -#include "dapl_provider.h" itamar> -#include "dapl_hca_util.h" itamar> - itamar> -/* itamar> - * dapl_hca_alloc itamar> - * itamar> - * alloc and initialize an HCA struct itamar> - * itamar> - * Input: itamar> - * name itamar> - * port itamar> - * itamar> - * Output: itamar> - * hca itamar> - * itamar> - * Returns: itamar> - * none itamar> - * itamar> - */ itamar> -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) itamar> -{ itamar> - struct dapl_hca *hca; itamar> - int malloc_size = sizeof *hca + strlen(name) + 1; itamar> - itamar> - hca = kmalloc(malloc_size, GFP_ATOMIC); itamar> - if (hca) { itamar> - memset(hca, 0, malloc_size); itamar> - spin_lock_init(&hca->lock); itamar> - INIT_LIST_HEAD(&hca->ia_list); itamar> - hca->name = (char *)hca + sizeof *hca; itamar> - strcpy(hca->name, name); itamar> - hca->ib_hca_handle = device; itamar> - hca->port_num = port; itamar> - } itamar> - return hca; itamar> -} itamar> - itamar> -/* itamar> - * dapl_hca_free itamar> - * itamar> - * free an IA INFO struct itamar> - * itamar> - * Input: itamar> - * hca itamar> - * itamar> - * Output: itamar> - * none itamar> - * itamar> - * Returns: itamar> - * none itamar> - * itamar> - */ itamar> -void dapl_hca_free(struct dapl_hca *hca) itamar> -{ itamar> - kfree(hca); itamar> -} itamar> diff -Nurp -X dontdiff dat-provider_link_ia/dapl_hca_util.h dat-provider/dapl_hca_util.h itamar> --- dat-provider_link_ia/dapl_hca_util.h Sun Jun 19 18:43:16 2005 itamar> +++ dat-provider/dapl_hca_util.h Thu Jan 1 02:00:00 1970 itamar> @@ -1,42 +0,0 @@ itamar> -/* itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. itamar> - * itamar> - * This Software is licensed under one of the following licenses: itamar> - * itamar> - * 1) under the terms of the "Common Public License 1.0" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/cpl.php. itamar> - * itamar> - * 2) under the terms of the "The BSD License" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/bsd-license.php. itamar> - * itamar> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a itamar> - * copy of which is available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/gpl-license.php. itamar> - * itamar> - * Licensee has the right to choose one of the above licenses. itamar> - * itamar> - * Redistributions of source code must retain the above copyright itamar> - * notice and one of the license notices. itamar> - * itamar> - * Redistributions in binary form must reproduce both the above copyright itamar> - * notice, one of the license notices in the documentation itamar> - * and/or other materials provided with the distribution. itamar> - */ itamar> - itamar> -/* itamar> - * $Id: dapl_hca_util.h 2640 2005-06-16 16:22:46Z jlentini $ itamar> - */ itamar> - itamar> -#ifndef DAPL_HCA_UTIL_H itamar> -#define DAPL_HCA_UTIL_H itamar> - itamar> -#include "dapl.h" itamar> - itamar> -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port); itamar> - itamar> -void dapl_hca_free(struct dapl_hca *hca); itamar> - itamar> -#endif itamar> diff -Nurp -X dontdiff dat-provider_link_ia/dapl_provider.c dat-provider/dapl_provider.c itamar> --- dat-provider_link_ia/dapl_provider.c Sun Jun 19 18:43:16 2005 itamar> +++ dat-provider/dapl_provider.c Sun Jun 19 19:47:16 2005 itamar> @@ -35,7 +35,6 @@ itamar> #include itamar> itamar> #include "dapl.h" itamar> -#include "dapl_hca_util.h" itamar> #include "dapl_provider.h" itamar> #include "dapl_util.h" itamar> #include "dapl_openib_util.h" itamar> @@ -246,6 +245,24 @@ static void dapl_provider_info_init(stru itamar> provider_info->ia_name[i+1] = '\0'; itamar> } itamar> itamar> +static struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) itamar> +{ itamar> + struct dapl_hca *hca; itamar> + int malloc_size = sizeof *hca + strlen(name) + 1; itamar> + itamar> + hca = kmalloc(malloc_size, GFP_ATOMIC); itamar> + if (hca) { itamar> + memset(hca, 0, malloc_size); itamar> + spin_lock_init(&hca->lock); itamar> + INIT_LIST_HEAD(&hca->ia_list); itamar> + hca->name = (char *)hca + sizeof *hca; itamar> + strcpy(hca->name, name); itamar> + hca->ib_hca_handle = device; itamar> + hca->port_num = port; itamar> + } itamar> + return hca; itamar> +} itamar> + itamar> static void dapl_add_port(struct ib_device *device, u8 port) itamar> { itamar> struct dat_provider_info provider_info; itamar> @@ -305,7 +322,7 @@ error: itamar> (void)dapl_provider_list_remove(provider_info.ia_name); itamar> itamar> if (NULL != hca) itamar> - dapl_hca_free(hca); itamar> + kfree(hca); itamar> } itamar> } itamar> itamar> @@ -338,7 +355,7 @@ static void dapl_remove_port(struct ib_d itamar> provider_info.ia_name); itamar> } itamar> itamar> - dapl_hca_free(provider->extension); itamar> + kfree(provider->extension); itamar> itamar> dapl_provider_list_remove(provider_info.ia_name); itamar> } itamar> -- itamar> Itamar itamar> From jlentini at netapp.com Thu Jun 23 13:28:11 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 16:28:11 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c In-Reply-To: References: <20050620085010.GA7206@mellanox.co.il> Message-ID: One more thing, if we decide to keep these files, I will rename them dapl_hca.[ch] for consistency. On Thu, 23 Jun 2005, James Lentini wrote: > > For consistency, I think we should keep the HCA object in its own > file. > > However, I'm not sure we need an HCA object. Is there a better way to > organize the data being stored in the dapl_hca structure? I have this > feeling that we should merge all the structures that we create on a > per-hca basis (the dapl_provider_entry, dat_provider table, and > dapl_hca). > > james > > On Mon, 20 Jun 2005, Itamar Rabenstein wrote: > > itamar> ntegrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c > itamar> (no need for 2 files just for 2 simple function that kmalloc and kfree. > itamar> There is not any special logic in this functions that need to separate > itamar> them into different files) > itamar> > itamar> Signed-off-by: Itamar Rabenstein > itamar> > itamar> diff -Nurp -X dontdiff dat-provider_link_ia/Makefile dat-provider/Makefile > itamar> --- dat-provider_link_ia/Makefile Sun Jun 19 18:43:16 2005 > itamar> +++ dat-provider/Makefile Sun Jun 19 19:45:15 2005 > itamar> @@ -20,7 +20,6 @@ PROVIDER_MODULES := \ > itamar> dapl_cr \ > itamar> dapl_ep \ > itamar> dapl_evd \ > itamar> - dapl_hca_util \ > itamar> dapl_ia \ > itamar> dapl_lmr \ > itamar> dapl_provider \ > itamar> diff -Nurp -X dontdiff dat-provider_link_ia/dapl_hca_util.c dat-provider/dapl_hca_util.c > itamar> --- dat-provider_link_ia/dapl_hca_util.c Sun Jun 19 18:43:16 2005 > itamar> +++ dat-provider/dapl_hca_util.c Thu Jan 1 02:00:00 1970 > itamar> @@ -1,90 +0,0 @@ > itamar> -/* > itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. > itamar> - * > itamar> - * This Software is licensed under one of the following licenses: > itamar> - * > itamar> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > itamar> - * available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/cpl.php. > itamar> - * > itamar> - * 2) under the terms of the "The BSD License" a copy of which is > itamar> - * available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/bsd-license.php. > itamar> - * > itamar> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > itamar> - * copy of which is available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/gpl-license.php. > itamar> - * > itamar> - * Licensee has the right to choose one of the above licenses. > itamar> - * > itamar> - * Redistributions of source code must retain the above copyright > itamar> - * notice and one of the license notices. > itamar> - * > itamar> - * Redistributions in binary form must reproduce both the above copyright > itamar> - * notice, one of the license notices in the documentation > itamar> - * and/or other materials provided with the distribution. > itamar> - */ > itamar> - > itamar> -/* > itamar> - * $Id: dapl_hca_util.c 2640 2005-06-16 16:22:46Z jlentini $ > itamar> - */ > itamar> - > itamar> -#include "dapl.h" > itamar> -#include "dapl_openib_util.h" > itamar> -#include "dapl_provider.h" > itamar> -#include "dapl_hca_util.h" > itamar> - > itamar> -/* > itamar> - * dapl_hca_alloc > itamar> - * > itamar> - * alloc and initialize an HCA struct > itamar> - * > itamar> - * Input: > itamar> - * name > itamar> - * port > itamar> - * > itamar> - * Output: > itamar> - * hca > itamar> - * > itamar> - * Returns: > itamar> - * none > itamar> - * > itamar> - */ > itamar> -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) > itamar> -{ > itamar> - struct dapl_hca *hca; > itamar> - int malloc_size = sizeof *hca + strlen(name) + 1; > itamar> - > itamar> - hca = kmalloc(malloc_size, GFP_ATOMIC); > itamar> - if (hca) { > itamar> - memset(hca, 0, malloc_size); > itamar> - spin_lock_init(&hca->lock); > itamar> - INIT_LIST_HEAD(&hca->ia_list); > itamar> - hca->name = (char *)hca + sizeof *hca; > itamar> - strcpy(hca->name, name); > itamar> - hca->ib_hca_handle = device; > itamar> - hca->port_num = port; > itamar> - } > itamar> - return hca; > itamar> -} > itamar> - > itamar> -/* > itamar> - * dapl_hca_free > itamar> - * > itamar> - * free an IA INFO struct > itamar> - * > itamar> - * Input: > itamar> - * hca > itamar> - * > itamar> - * Output: > itamar> - * none > itamar> - * > itamar> - * Returns: > itamar> - * none > itamar> - * > itamar> - */ > itamar> -void dapl_hca_free(struct dapl_hca *hca) > itamar> -{ > itamar> - kfree(hca); > itamar> -} > itamar> diff -Nurp -X dontdiff dat-provider_link_ia/dapl_hca_util.h dat-provider/dapl_hca_util.h > itamar> --- dat-provider_link_ia/dapl_hca_util.h Sun Jun 19 18:43:16 2005 > itamar> +++ dat-provider/dapl_hca_util.h Thu Jan 1 02:00:00 1970 > itamar> @@ -1,42 +0,0 @@ > itamar> -/* > itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. > itamar> - * > itamar> - * This Software is licensed under one of the following licenses: > itamar> - * > itamar> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > itamar> - * available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/cpl.php. > itamar> - * > itamar> - * 2) under the terms of the "The BSD License" a copy of which is > itamar> - * available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/bsd-license.php. > itamar> - * > itamar> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > itamar> - * copy of which is available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/gpl-license.php. > itamar> - * > itamar> - * Licensee has the right to choose one of the above licenses. > itamar> - * > itamar> - * Redistributions of source code must retain the above copyright > itamar> - * notice and one of the license notices. > itamar> - * > itamar> - * Redistributions in binary form must reproduce both the above copyright > itamar> - * notice, one of the license notices in the documentation > itamar> - * and/or other materials provided with the distribution. > itamar> - */ > itamar> - > itamar> -/* > itamar> - * $Id: dapl_hca_util.h 2640 2005-06-16 16:22:46Z jlentini $ > itamar> - */ > itamar> - > itamar> -#ifndef DAPL_HCA_UTIL_H > itamar> -#define DAPL_HCA_UTIL_H > itamar> - > itamar> -#include "dapl.h" > itamar> - > itamar> -struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port); > itamar> - > itamar> -void dapl_hca_free(struct dapl_hca *hca); > itamar> - > itamar> -#endif > itamar> diff -Nurp -X dontdiff dat-provider_link_ia/dapl_provider.c dat-provider/dapl_provider.c > itamar> --- dat-provider_link_ia/dapl_provider.c Sun Jun 19 18:43:16 2005 > itamar> +++ dat-provider/dapl_provider.c Sun Jun 19 19:47:16 2005 > itamar> @@ -35,7 +35,6 @@ > itamar> #include > itamar> > itamar> #include "dapl.h" > itamar> -#include "dapl_hca_util.h" > itamar> #include "dapl_provider.h" > itamar> #include "dapl_util.h" > itamar> #include "dapl_openib_util.h" > itamar> @@ -246,6 +245,24 @@ static void dapl_provider_info_init(stru > itamar> provider_info->ia_name[i+1] = '\0'; > itamar> } > itamar> > itamar> +static struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) > itamar> +{ > itamar> + struct dapl_hca *hca; > itamar> + int malloc_size = sizeof *hca + strlen(name) + 1; > itamar> + > itamar> + hca = kmalloc(malloc_size, GFP_ATOMIC); > itamar> + if (hca) { > itamar> + memset(hca, 0, malloc_size); > itamar> + spin_lock_init(&hca->lock); > itamar> + INIT_LIST_HEAD(&hca->ia_list); > itamar> + hca->name = (char *)hca + sizeof *hca; > itamar> + strcpy(hca->name, name); > itamar> + hca->ib_hca_handle = device; > itamar> + hca->port_num = port; > itamar> + } > itamar> + return hca; > itamar> +} > itamar> + > itamar> static void dapl_add_port(struct ib_device *device, u8 port) > itamar> { > itamar> struct dat_provider_info provider_info; > itamar> @@ -305,7 +322,7 @@ error: > itamar> (void)dapl_provider_list_remove(provider_info.ia_name); > itamar> > itamar> if (NULL != hca) > itamar> - dapl_hca_free(hca); > itamar> + kfree(hca); > itamar> } > itamar> } > itamar> > itamar> @@ -338,7 +355,7 @@ static void dapl_remove_port(struct ib_d > itamar> provider_info.ia_name); > itamar> } > itamar> > itamar> - dapl_hca_free(provider->extension); > itamar> + kfree(provider->extension); > itamar> > itamar> dapl_provider_list_remove(provider_info.ia_name); > itamar> } > itamar> -- > itamar> Itamar > itamar> > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From itamar at mellanox.co.il Thu Jun 23 14:02:29 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Fri, 24 Jun 2005 00:02:29 +0300 Subject: [openib-general] RE: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to dapl _provider.c Message-ID: <91DB792C7985D411BEC300B40080D29CC35D0C@mtvex01.mtv.mtl.com> hi James, I think that you are worng. There is no functionalty to this struct. it is only a struct that hold some data together. if we look into the files dapl_hca_util.[h.c] we will find 4 functions: 1) dapl_hca_alloc -> which is kmalloc and set to 4 data members of this struct 2) dapl_hca_free -> only 1 line == kfree 3) dapl_hca_link_ia -> which is only a call to list_add 4) dapl_hca_unlink_ia -> which is only a call to list_del you have found out that functions 3 & 4 are not needed so we need to delete them. so i dont see why we should have 2 files for kmalloc and kfree HCA is not like ia,evd,ep, ... the others are dapl spec structs so they should be in separete files , hca is just a name that come from IB it is part of the provider code. I believe that in other protocols implementaions they will not use "HCA". Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Thursday, June 23, 2005 11:27 PM > To: Itamar Rabenstein > Cc: openib-general > Subject: Re: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to > dapl_provider.c > > > > For consistency, I think we should keep the HCA object in its own > file. > > However, I'm not sure we need an HCA object. Is there a better way to > organize the data being stored in the dapl_hca structure? I have this > feeling that we should merge all the structures that we create on a > per-hca basis (the dapl_provider_entry, dat_provider table, and > dapl_hca). > > james > > On Mon, 20 Jun 2005, Itamar Rabenstein wrote: > > itamar> ntegrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c > itamar> (no need for 2 files just for 2 simple function that > kmalloc and kfree. > itamar> There is not any special logic in this functions > that need to separate > itamar> them into different files) > itamar> > itamar> Signed-off-by: Itamar Rabenstein > itamar> > itamar> diff -Nurp -X dontdiff dat-provider_link_ia/Makefile > dat-provider/Makefile > itamar> --- dat-provider_link_ia/Makefile Sun Jun 19 18:43:16 2005 > itamar> +++ dat-provider/Makefile Sun Jun 19 19:45:15 2005 > itamar> @@ -20,7 +20,6 @@ PROVIDER_MODULES := \ > itamar> dapl_cr \ > itamar> dapl_ep \ > itamar> dapl_evd \ > itamar> - dapl_hca_util \ > itamar> dapl_ia \ > itamar> dapl_lmr \ > itamar> dapl_provider \ > itamar> diff -Nurp -X dontdiff > dat-provider_link_ia/dapl_hca_util.c dat-provider/dapl_hca_util.c > itamar> --- dat-provider_link_ia/dapl_hca_util.c Sun Jun > 19 18:43:16 2005 > itamar> +++ dat-provider/dapl_hca_util.c Thu Jan 1 02:00:00 1970 > itamar> @@ -1,90 +0,0 @@ > itamar> -/* > itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. > All rights reserved. > itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All > rights reserved. > itamar> - * > itamar> - * This Software is licensed under one of the > following licenses: > itamar> - * > itamar> - * 1) under the terms of the "Common Public License > 1.0" a copy of which is > itamar> - * available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/cpl.php. > itamar> - * > itamar> - * 2) under the terms of the "The BSD License" a > copy of which is > itamar> - * available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/bsd-license.php. > itamar> - * > itamar> - * 3) under the terms of the "GNU General Public > License (GPL) Version 2" a > itamar> - * copy of which is available from the Open > Source Initiative, see > itamar> - * http://www.opensource.org/licenses/gpl-license.php. > itamar> - * > itamar> - * Licensee has the right to choose one of the above > licenses. > itamar> - * > itamar> - * Redistributions of source code must retain the > above copyright > itamar> - * notice and one of the license notices. > itamar> - * > itamar> - * Redistributions in binary form must reproduce > both the above copyright > itamar> - * notice, one of the license notices in the documentation > itamar> - * and/or other materials provided with the distribution. > itamar> - */ > itamar> - > itamar> -/* > itamar> - * $Id: dapl_hca_util.c 2640 2005-06-16 16:22:46Z jlentini $ > itamar> - */ > itamar> - > itamar> -#include "dapl.h" > itamar> -#include "dapl_openib_util.h" > itamar> -#include "dapl_provider.h" > itamar> -#include "dapl_hca_util.h" > itamar> - > itamar> -/* > itamar> - * dapl_hca_alloc > itamar> - * > itamar> - * alloc and initialize an HCA struct > itamar> - * > itamar> - * Input: > itamar> - * name > itamar> - * port > itamar> - * > itamar> - * Output: > itamar> - * hca > itamar> - * > itamar> - * Returns: > itamar> - * none > itamar> - * > itamar> - */ > itamar> -struct dapl_hca *dapl_hca_alloc(char *name, struct > ib_device *device, u8 port) > itamar> -{ > itamar> - struct dapl_hca *hca; > itamar> - int malloc_size = sizeof *hca + strlen(name) + 1; > itamar> - > itamar> - hca = kmalloc(malloc_size, GFP_ATOMIC); > itamar> - if (hca) { > itamar> - memset(hca, 0, malloc_size); > itamar> - spin_lock_init(&hca->lock); > itamar> - INIT_LIST_HEAD(&hca->ia_list); > itamar> - hca->name = (char *)hca + sizeof *hca; > itamar> - strcpy(hca->name, name); > itamar> - hca->ib_hca_handle = device; > itamar> - hca->port_num = port; > itamar> - } > itamar> - return hca; > itamar> -} > itamar> - > itamar> -/* > itamar> - * dapl_hca_free > itamar> - * > itamar> - * free an IA INFO struct > itamar> - * > itamar> - * Input: > itamar> - * hca > itamar> - * > itamar> - * Output: > itamar> - * none > itamar> - * > itamar> - * Returns: > itamar> - * none > itamar> - * > itamar> - */ > itamar> -void dapl_hca_free(struct dapl_hca *hca) > itamar> -{ > itamar> - kfree(hca); > itamar> -} > itamar> diff -Nurp -X dontdiff > dat-provider_link_ia/dapl_hca_util.h dat-provider/dapl_hca_util.h > itamar> --- dat-provider_link_ia/dapl_hca_util.h Sun Jun > 19 18:43:16 2005 > itamar> +++ dat-provider/dapl_hca_util.h Thu Jan 1 02:00:00 1970 > itamar> @@ -1,42 +0,0 @@ > itamar> -/* > itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. > All rights reserved. > itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All > rights reserved. > itamar> - * > itamar> - * This Software is licensed under one of the > following licenses: > itamar> - * > itamar> - * 1) under the terms of the "Common Public License > 1.0" a copy of which is > itamar> - * available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/cpl.php. > itamar> - * > itamar> - * 2) under the terms of the "The BSD License" a > copy of which is > itamar> - * available from the Open Source Initiative, see > itamar> - * http://www.opensource.org/licenses/bsd-license.php. > itamar> - * > itamar> - * 3) under the terms of the "GNU General Public > License (GPL) Version 2" a > itamar> - * copy of which is available from the Open > Source Initiative, see > itamar> - * http://www.opensource.org/licenses/gpl-license.php. > itamar> - * > itamar> - * Licensee has the right to choose one of the above > licenses. > itamar> - * > itamar> - * Redistributions of source code must retain the > above copyright > itamar> - * notice and one of the license notices. > itamar> - * > itamar> - * Redistributions in binary form must reproduce > both the above copyright > itamar> - * notice, one of the license notices in the documentation > itamar> - * and/or other materials provided with the distribution. > itamar> - */ > itamar> - > itamar> -/* > itamar> - * $Id: dapl_hca_util.h 2640 2005-06-16 16:22:46Z jlentini $ > itamar> - */ > itamar> - > itamar> -#ifndef DAPL_HCA_UTIL_H > itamar> -#define DAPL_HCA_UTIL_H > itamar> - > itamar> -#include "dapl.h" > itamar> - > itamar> -struct dapl_hca *dapl_hca_alloc(char *name, struct > ib_device *device, u8 port); > itamar> - > itamar> -void dapl_hca_free(struct dapl_hca *hca); > itamar> - > itamar> -#endif > itamar> diff -Nurp -X dontdiff > dat-provider_link_ia/dapl_provider.c dat-provider/dapl_provider.c > itamar> --- dat-provider_link_ia/dapl_provider.c Sun Jun > 19 18:43:16 2005 > itamar> +++ dat-provider/dapl_provider.c Sun Jun 19 19:47:16 2005 > itamar> @@ -35,7 +35,6 @@ > itamar> #include > itamar> > itamar> #include "dapl.h" > itamar> -#include "dapl_hca_util.h" > itamar> #include "dapl_provider.h" > itamar> #include "dapl_util.h" > itamar> #include "dapl_openib_util.h" > itamar> @@ -246,6 +245,24 @@ static void dapl_provider_info_init(stru > itamar> provider_info->ia_name[i+1] = '\0'; > itamar> } > itamar> > itamar> +static struct dapl_hca *dapl_hca_alloc(char *name, > struct ib_device *device, u8 port) > itamar> +{ > itamar> + struct dapl_hca *hca; > itamar> + int malloc_size = sizeof *hca + strlen(name) + 1; > itamar> + > itamar> + hca = kmalloc(malloc_size, GFP_ATOMIC); > itamar> + if (hca) { > itamar> + memset(hca, 0, malloc_size); > itamar> + spin_lock_init(&hca->lock); > itamar> + INIT_LIST_HEAD(&hca->ia_list); > itamar> + hca->name = (char *)hca + sizeof *hca; > itamar> + strcpy(hca->name, name); > itamar> + hca->ib_hca_handle = device; > itamar> + hca->port_num = port; > itamar> + } > itamar> + return hca; > itamar> +} > itamar> + > itamar> static void dapl_add_port(struct ib_device *device, u8 port) > itamar> { > itamar> struct dat_provider_info provider_info; > itamar> @@ -305,7 +322,7 @@ error: > itamar> > (void)dapl_provider_list_remove(provider_info.ia_name); > itamar> > itamar> if (NULL != hca) > itamar> - dapl_hca_free(hca); > itamar> + kfree(hca); > itamar> } > itamar> } > itamar> > itamar> @@ -338,7 +355,7 @@ static void dapl_remove_port(struct ib_d > itamar> provider_info.ia_name); > itamar> } > itamar> > itamar> - dapl_hca_free(provider->extension); > itamar> + kfree(provider->extension); > itamar> > itamar> dapl_provider_list_remove(provider_info.ia_name); > itamar> } > itamar> -- > itamar> Itamar > itamar> > From Thomas.Talpey at netapp.com Thu Jun 23 13:48:27 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Thu, 23 Jun 2005 16:48:27 -0400 Subject: [openib-general] NFS/RDMA/kDAPL In-Reply-To: <9d3b7de705062313111bf32bef@mail.gmail.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <6.2.1.2.2.20050623124730.05454140@exnane01.nane.netapp.com> <42BAED53.80206@linuxmachines.com> <6.2.1.2.2.20050623132913.03e18480@exnane01.nane.netapp.com> <1119553876.15654.1.camel@duffman> <6.2.1.2.2.20050623151939.03d80900@exnane01.nane.netapp.com> <9d3b7de705062313111bf32bef@mail.gmail.com> Message-ID: <6.2.1.2.2.20050623164538.04296eb0@exnane01.nane.netapp.com> At 04:11 PM 6/23/2005, Tom Duffy wrote: >I am new to NFSoRDMA. Where do I start? Is there a good doc about >how to set it up? Consult your OpenSolaris documentation? :-) Seriously, there isn't a cookbook for Linux, because we're not there yet. But it's not too hard. I will plan to roll up the updated client, then put together a cookbook. Chuck's new RPC transport patch has some switch API changes which Trond requested, which I have to study first. Let's aim for next week. >Tom, would you be so kind as to start a tiki page about it? >Just click on NFSoRDMA on this page and go: > >https://openib.org/tiki/tiki-index.php OK. After the above. Tom. From mst at mellanox.co.il Thu Jun 23 13:48:36 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 23 Jun 2005 23:48:36 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <52k6klm1kc.fsf@topspin.com> References: <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> <52k6klm1kc.fsf@topspin.com> Message-ID: <20050623204836.GA14000@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] rdma_lat-09 and results > > Michael> At least for this usage, I think it would be best to > Michael> rename mthca.a (and mthca.so) to libmthca.a (and > Michael> libmthca.so), respectively, and put them in libpath, then > Michael> softlink libmthca.so to infiniband directory. > > Michael> This avoids the need for such utilities. Would you > Michael> accept such a change? > > That makes sense, although I don't see why we want libmthca.so in the > regular library directory. I can imagine a cluster system where loading shared libraries work, but reading a directory doesnt. Ronald Minnich should be able to enlighten us on whether bproc really has such a limitation. Roland - do I describe brpoc correctly? I know bproc supports dlopen from remote applications, but what about glob() on a remote directory? Does that work from a blade? > It seems like the best would be to have > libmthca.a in $(libdir) and mthca.so in $(libdir)/infiniband. That > way we make it easy to do static linking and avoid having two names > for the same object. OK, but I dont know how to do that in autotools. Do you? > Michael> Hmm, this would be a security problem if someone sets > Michael> suid bit on programs using uverbs. > > True, good point. To me being able to set the search path is much > more useful than being able to have secure suid verbs programs, Actually, whats the importance of it? It seems the user can compile libibverbs as readily as the plugin ... > so I > don't want to get rid of the OPENIB_DRIVER_PATH. > Does something like the following seem good enough? Solves this particular problem, but maybe, additionally, there should be a configuration option to disable OPENIB_DRIVER_PATH lookup? I also wander about the user of strdupa - can it cause stack overflow? -- MST From jlentini at netapp.com Thu Jun 23 13:55:38 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 16:55:38 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove dapl_os_assert() In-Reply-To: <1119558325.21702.1.camel@duffman> References: <1119558325.21702.1.camel@duffman> Message-ID: Why did you remove the asserts in dapl_ib_post_recv() and dapl_ib_post_send()? My argument in favor of retaining them is that dapl_evd_wc_to_event() will crash if the cookie NULL. A BUG_ON will detect this situation sooner rather than later and therefore make the problem easier to diagnose. Did I miss something? On Thu, 23 Jun 2005, Tom Duffy wrote: tduffy> This patch removes dapl_os_assert(). In most cases, replacing with tduffy> BUG_ON(!). Some cases, I just called panic() where others I removed the tduffy> assert all together because the next line dereferences the pointer tduffy> anyways. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_cookie.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cookie.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) tduffy> @@ -184,7 +184,7 @@ u32 dapl_cb_get(struct dapl_cookie_buffe tduffy> u32 dat_status; tduffy> int new_head; tduffy> tduffy> - dapl_os_assert(NULL != cookie_ptr); tduffy> + BUG_ON(cookie_ptr == NULL); tduffy> tduffy> new_head = (atomic_read(&buffer->head) + 1) % buffer->pool_size; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -783,7 +783,7 @@ u32 dapl_ib_get_gid(struct ib_device *hc tduffy> { tduffy> int status; tduffy> tduffy> - dapl_os_assert(hca); tduffy> + BUG_ON(!hca); tduffy> tduffy> if (gid) { tduffy> status = ib_query_gid(hca, port, 0, gid); tduffy> Index: linux-kernel/dat-provider/dapl_openib_qp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) tduffy> @@ -76,7 +76,6 @@ u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ tduffy> tduffy> attr = &ep_ptr->param.ep_attr; tduffy> tduffy> - dapl_os_assert(ep_ptr->param.pz != NULL); tduffy> ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz)->pd; tduffy> ib_hca_handle = ia_ptr->hca->ib_hca_handle; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_openib_dto.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_dto.h (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_openib_dto.h (working copy) tduffy> @@ -43,8 +43,6 @@ static inline u32 dapl_ib_post_recv(stru tduffy> struct ib_sge *sg_list; tduffy> int status, i, total_len = 0; tduffy> tduffy> - dapl_os_assert(NULL != cookie); tduffy> - tduffy> sg_list = ep->recv_iov; tduffy> for (i = 0; i < num_segments; i++, sg_list++) { tduffy> sg_list->addr = local_iov[i].virtual_address; tduffy> @@ -84,8 +82,6 @@ static inline u32 dapl_ib_post_send(stru tduffy> struct ib_sge *sg_list; tduffy> int status, i, total_len = 0; tduffy> tduffy> - dapl_os_assert(NULL != cookie); tduffy> - tduffy> sg_list = ep->send_iov; tduffy> for (i = 0; i < num_segments; i++, sg_list++) { tduffy> sg_list->addr = local_iov[i].virtual_address; tduffy> Index: linux-kernel/dat-provider/dapl_ia.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_ia.c (working copy) tduffy> @@ -371,13 +371,13 @@ bail: tduffy> */ tduffy> void dapl_ia_free(struct dapl_ia *ia) tduffy> { tduffy> - dapl_os_assert(ia->async_error_evd == NULL); tduffy> - dapl_os_assert(list_empty(&ia->lmr_list)); tduffy> - dapl_os_assert(list_empty(&ia->rmr_list)); tduffy> - dapl_os_assert(list_empty(&ia->ep_list)); tduffy> - dapl_os_assert(list_empty(&ia->evd_list)); tduffy> - dapl_os_assert(list_empty(&ia->psp_list)); tduffy> - dapl_os_assert(list_empty(&ia->rsp_list)); tduffy> + BUG_ON(ia->async_error_evd != NULL); tduffy> + BUG_ON(!list_empty(&ia->lmr_list)); tduffy> + BUG_ON(!list_empty(&ia->rmr_list)); tduffy> + BUG_ON(!list_empty(&ia->ep_list)); tduffy> + BUG_ON(!list_empty(&ia->evd_list)); tduffy> + BUG_ON(!list_empty(&ia->psp_list)); tduffy> + BUG_ON(!list_empty(&ia->rsp_list)); tduffy> tduffy> dapl_hca_unlink_ia(ia->hca, ia); tduffy> /* no need to destroy ia->common.lock */ tduffy> Index: linux-kernel/dat-provider/dapl_rmr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_rmr.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) tduffy> @@ -126,8 +126,7 @@ static u64 dapl_rmr_get_address(DAT_REGI tduffy> * DAT_MEM_TYPE_IA tduffy> * DAT_MEM_TYPE_BYPASS tduffy> */ tduffy> - dapl_os_assert(0); tduffy> - return 0; tduffy> + panic("unimplemented or unknown memory type\n"); tduffy> } tduffy> } tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ep.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_ep.c (working copy) tduffy> @@ -675,8 +675,8 @@ u32 dapl_ep_free(struct dat_ep *ep) tduffy> */ tduffy> (void)dapl_ep_disconnect((struct dat_ep *)ep_ptr, tduffy> DAT_CLOSE_ABRUPT_FLAG); tduffy> - dapl_os_assert(ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || tduffy> - ep_ptr->param.ep_state == DAT_EP_STATE_UNCONNECTED); tduffy> + BUG_ON(ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECTED && tduffy> + ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED); tduffy> tduffy> /* tduffy> * Do verification of parameters and the state change atomically. tduffy> @@ -1538,11 +1538,11 @@ u32 dapl_ep_modify(struct dat_ep *ep, en tduffy> * occurred. But they're important to the logic of this routine, tduffy> * so we check. tduffy> */ tduffy> - dapl_os_assert(ep1 == ep2); tduffy> - dapl_os_assert(ep_attr2.max_recv_dtos == ep_attr1.max_recv_dtos); tduffy> - dapl_os_assert(ep_attr2.max_request_dtos == ep_attr1.max_request_dtos); tduffy> - dapl_os_assert(ep_attr2.max_recv_iov == ep_attr1.max_recv_iov); tduffy> - dapl_os_assert(ep_attr2.max_request_iov == ep_attr1.max_request_iov); tduffy> + BUG_ON(ep1 != ep2); tduffy> + BUG_ON(ep_attr2.max_recv_dtos != ep_attr1.max_recv_dtos); tduffy> + BUG_ON(ep_attr2.max_request_dtos != ep_attr1.max_request_dtos); tduffy> + BUG_ON(ep_attr2.max_recv_iov != ep_attr1.max_recv_iov); tduffy> + BUG_ON(ep_attr2.max_request_iov != ep_attr1.max_request_iov); tduffy> tduffy> copy_of_old_ep = *ep2; tduffy> tduffy> @@ -1607,10 +1607,10 @@ u32 dapl_ep_modify(struct dat_ep *ep, en tduffy> * because the parameter validate routine should protect us, tduffy> * but it's an important enough point that we assert it. tduffy> */ tduffy> - dapl_os_assert((ep2->param.ep_state tduffy> - != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING) tduffy> - && (ep2->param.ep_state tduffy> - != DAT_EP_STATE_ACTIVE_CONNECTION_PENDING)); tduffy> + BUG_ON(ep2->param.ep_state == tduffy> + DAT_EP_STATE_PASSIVE_CONNECTION_PENDING || tduffy> + ep2->param.ep_state == tduffy> + DAT_EP_STATE_ACTIVE_CONNECTION_PENDING); tduffy> tduffy> new_ep.qp = alloc_ep.qp; tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_cr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cr.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_cr.c (working copy) tduffy> @@ -271,7 +271,7 @@ static u32 dapl_connection_request(struc tduffy> DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING; tduffy> } else { tduffy> /* RSP */ tduffy> - dapl_os_assert(sp->sp.type == DAT_SP_TYPE_RSP); tduffy> + BUG_ON(sp->sp.type != DAT_SP_TYPE_RSP); tduffy> ep->param.ep_state = tduffy> DAT_EP_STATE_PASSIVE_CONNECTION_PENDING; tduffy> } tduffy> @@ -516,8 +516,7 @@ void dapl_cr_callback(struct dapl_cm_ctx tduffy> break; tduffy> default: tduffy> evd = NULL; tduffy> - dapl_os_assert(0); /* shouldn't happen */ tduffy> - break; tduffy> + panic("unknown callback event\n"); tduffy> } tduffy> tduffy> if (evd != NULL) tduffy> Index: linux-kernel/dat-provider/dapl_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_evd.c (working copy) tduffy> @@ -243,7 +243,7 @@ static u32 dapl_evd_dealloc(struct dapl_ tduffy> u32 status = DAT_SUCCESS; tduffy> struct dapl_ia *ia; tduffy> tduffy> - dapl_os_assert(atomic_read(&evd->evd_ref_count) == 0); tduffy> + BUG_ON(atomic_read(&evd->evd_ref_count) != 0); tduffy> tduffy> /* tduffy> * Destroy the CQ first, to keep any more callbacks from coming tduffy> @@ -330,9 +330,9 @@ static void dapl_evd_post_event(struct d tduffy> event->event_number); tduffy> tduffy> status = dapl_rbuf_add(&evd->pending_event_queue, event); tduffy> - dapl_os_assert(status == DAT_SUCCESS); tduffy> + BUG_ON(status != DAT_SUCCESS); tduffy> tduffy> - dapl_os_assert(evd->evd_state == DAPL_EVD_STATE_OPEN); tduffy> + BUG_ON(evd->evd_state != DAPL_EVD_STATE_OPEN); tduffy> tduffy> if (evd->evd_producer_locking_needed) tduffy> spin_unlock_irqrestore(&evd->common.lock, tduffy> @@ -522,10 +522,9 @@ static void dapl_evd_wc_to_event(struct tduffy> dto_status = dapl_ib_get_dto_status(wc); tduffy> tduffy> cookie = (struct dapl_cookie *) (unsigned long) wc->wr_id; tduffy> - dapl_os_assert(NULL != cookie); tduffy> tduffy> ep = cookie->ep; tduffy> - dapl_os_assert(NULL != ep); tduffy> + BUG_ON(ep == NULL); tduffy> tduffy> event->evd = (struct dat_evd *)evd; tduffy> tduffy> @@ -550,26 +549,6 @@ static void dapl_evd_wc_to_event(struct tduffy> event_data->user_cookie = cookie->val.dto.cookie; tduffy> event_data->status = dto_status; tduffy> tduffy> -#if 0 tduffy> - /* Currently mthca is not setting the opcode in */ tduffy> - /* succesful wc. The opcode will be IB_WC_SEND or */ tduffy> - /* IB_WC_RECV according the is_send bit */ tduffy> - /* We can not check the following assert for now */ tduffy> - if (dto_status == DAT_DTO_SUCCESS) { tduffy> - enum ib_wc_opcode ib_opcode = wc->opcode; tduffy> - enum dapl_dto_type dto_type = cookie->val.dto.type; tduffy> - dapl_os_assert((ib_opcode == IB_WC_SEND && tduffy> - dto_type == DAPL_DTO_TYPE_SEND) || tduffy> - (ib_opcode == IB_WC_RECV && tduffy> - dto_type == DAPL_DTO_TYPE_RECV) || tduffy> - (ib_opcide == IB_WC_RDMA_WRITE && tduffy> - dto_type == DAPL_DTO_TYPE_RDMA_WRITE) || tduffy> - (ib_opcode == IB_WC_RDMA_READ && tduffy> - dto_type == DAPL_DTO_TYPE_RDMA_READ)); tduffy> - tduffy> - } tduffy> -#endif tduffy> - tduffy> if (cookie->val.dto.type == DAPL_DTO_TYPE_SEND || tduffy> cookie->val.dto.type == DAPL_DTO_TYPE_RDMA_WRITE) { tduffy> /* Get size from DTO; CQE value may be off. */ tduffy> @@ -593,7 +572,7 @@ static void dapl_evd_wc_to_event(struct tduffy> event_data->user_cookie = cookie->val.rmr.cookie; tduffy> tduffy> if (dto_status == DAT_DTO_SUCCESS) { tduffy> - dapl_os_assert(wc->opcode == IB_WC_BIND_MW); tduffy> + BUG_ON(wc->opcode != IB_WC_BIND_MW); tduffy> event_data->status = DAT_RMR_BIND_SUCCESS; tduffy> } else { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, tduffy> @@ -607,10 +586,7 @@ static void dapl_evd_wc_to_event(struct tduffy> break; tduffy> } tduffy> default: tduffy> - { tduffy> - dapl_os_assert(!"Invalid Operation type"); tduffy> - break; tduffy> - } tduffy> + panic("Invalid Operation type\n"); tduffy> } /* end switch */ tduffy> } tduffy> tduffy> @@ -652,7 +628,7 @@ void dapl_evd_qp_async_error_callback(st tduffy> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> } tduffy> tduffy> - dapl_os_assert(async_evd != NULL); tduffy> + BUG_ON(async_evd == NULL); tduffy> tduffy> status = dapl_ib_get_async_event(cause, &async_event); tduffy> if (status == DAT_SUCCESS) { tduffy> @@ -792,9 +768,7 @@ void dapl_evd_connection_callback(struct tduffy> default: tduffy> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); tduffy> evd = NULL; tduffy> - tduffy> - dapl_os_assert(0); /* shouldn't happen */ tduffy> - break; tduffy> + panic("bad event\n"); tduffy> } tduffy> tduffy> /* tduffy> @@ -838,7 +812,7 @@ static void dapl_evd_dto_callback(struct tduffy> tduffy> evd = (struct dapl_evd *)user_context; tduffy> tduffy> - dapl_os_assert(evd->cq == cq); tduffy> + BUG_ON(evd->cq != cq); tduffy> tduffy> /* Read once. */ tduffy> state = *(volatile enum dapl_evd_state *)&evd->evd_state; tduffy> Index: linux-kernel/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.h (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_util.h (working copy) tduffy> @@ -48,15 +48,6 @@ tduffy> #include tduffy> #endif tduffy> tduffy> -#define dapl_os_assert(expression) \ tduffy> - do { \ tduffy> - if (!(expression)) { \ tduffy> - panic("ASSERTION fail in %s:%i:%s\n", \ tduffy> - __FILE__, __LINE__, __func__); \ tduffy> - } \ tduffy> - } while (0) tduffy> - tduffy> - tduffy> /* dapl_os_atomic_assign tduffy> * tduffy> * assign 'new_value' to '*v' if the current value tduffy> Index: linux-kernel/dat-provider/dapl_sp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_sp.c (working copy) tduffy> @@ -64,7 +64,7 @@ static struct dapl_sp *dapl_sp_alloc(str tduffy> tduffy> void dapl_sp_dealloc(struct dapl_sp *sp) tduffy> { tduffy> - dapl_os_assert(list_empty(&sp->cr_list)); tduffy> + BUG_ON(!list_empty(&sp->cr_list)); tduffy> tduffy> kfree(sp); tduffy> } tduffy> tduffy> From iod00d at hp.com Thu Jun 23 14:09:17 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 23 Jun 2005 14:09:17 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: remove dapl_os_assert() In-Reply-To: References: <1119558325.21702.1.camel@duffman> Message-ID: <20050623210917.GQ22934@esmail.cup.hp.com> On Thu, Jun 23, 2005 at 04:55:38PM -0400, James Lentini wrote: > My argument in favor of retaining them is that dapl_evd_wc_to_event() > will crash if the cookie NULL. A BUG_ON will detect this situation > sooner rather than later and therefore make the problem easier to > diagnose. Did I miss something? The tombstone from the data page fault panic should make this nearly as obvious as the BUG_ON(). Yes, I agree a BUG_ON() is completely obvious. But it's also burning CPU cycles for something that should never happen and debugging is straight forware without the BUG_ON(). (Trust me, I've done more than a few of these on three different architectures). grant From rminnich at lanl.gov Thu Jun 23 14:28:42 2005 From: rminnich at lanl.gov (Ronald G. Minnich) Date: Thu, 23 Jun 2005 15:28:42 -0600 (MDT) Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050623204836.GA14000@mellanox.co.il> References: <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> <52k6klm1kc.fsf@topspin.com> <20050623204836.GA14000@mellanox.co.il> Message-ID: On Thu, 23 Jun 2005, Michael S. Tsirkin wrote: > I know bproc supports dlopen from remote applications, but > what about glob() on a remote directory? Does that work from a blade? I'm not totally sure what you're saying here but: if you have a .so and you need it on a bproc slave node, you can configure bproc so that the .so will be on that node once the node boots. If you only want the .so on there occasionally, or not at all, you can use a remote file system of some sort to make it available. Does that answer the question or did I miss something? thanks ron From tduffy at sun.com Thu Jun 23 14:31:06 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 23 Jun 2005 14:31:06 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: remove dapl_os_assert() In-Reply-To: References: <1119558325.21702.1.camel@duffman> Message-ID: <1119562266.21702.5.camel@duffman> On Thu, 2005-06-23 at 16:55 -0400, James Lentini wrote: > Why did you remove the asserts in dapl_ib_post_recv() and > dapl_ib_post_send()? The code continues 6 lines down: < assert if cookie is null here > sg_list = ep->recv_iov; for (i = 0; i < num_segments; i++, sg_list++) { sg_list->addr = local_iov[i].virtual_address; sg_list->length = (u32) local_iov[i].segment_length; sg_list->lkey = local_iov[i].lmr_context; total_len += sg_list->length; } cookie->val.dto.size = total_len; ^^^^^^^^ would crash here anyways. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mst at mellanox.co.il Thu Jun 23 14:42:24 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 24 Jun 2005 00:42:24 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: References: <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> <52k6klm1kc.fsf@topspin.com> <20050623204836.GA14000@mellanox.co.il> Message-ID: <20050623214224.GB14461@mellanox.co.il> Quoting r. Ronald G. Minnich : > Subject: Re: [PATCH] rdma_lat-09 and results > > > > On Thu, 23 Jun 2005, Michael S. Tsirkin wrote: > > > I know bproc supports dlopen from remote applications, but > > what about glob() on a remote directory? Does that work from a blade? > > I'm not totally sure what you're saying here but: > > if you have a .so and you need it on a bproc slave node, you can configure > bproc so that the .so will be on that node once the node boots. > > If you only want the .so on there occasionally, or not at all, you can use > a remote file system of some sort to make it available. > > Does that answer the question or did I miss something? > > thanks > > ron > I had this impression that I can have a .so not being present on the slave at boot, and then dlopen could pull it across the network with some custom protocol without going over NFS. And I was asking, if so, what other calls can do this besides dlopen? -- MST From jlentini at netapp.com Thu Jun 23 14:43:22 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 17:43:22 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove dapl_os_assert() In-Reply-To: <1119562266.21702.5.camel@duffman> References: <1119558325.21702.1.camel@duffman> <1119562266.21702.5.camel@duffman> Message-ID: Gotcha. I missed that. On Thu, 23 Jun 2005, Tom Duffy wrote: > On Thu, 2005-06-23 at 16:55 -0400, James Lentini wrote: >> Why did you remove the asserts in dapl_ib_post_recv() and >> dapl_ib_post_send()? > > The code continues 6 lines down: > > < assert if cookie is null here > > > sg_list = ep->recv_iov; > for (i = 0; i < num_segments; i++, sg_list++) { > sg_list->addr = local_iov[i].virtual_address; > sg_list->length = (u32) local_iov[i].segment_length; > sg_list->lkey = local_iov[i].lmr_context; > total_len += sg_list->length; > } > cookie->val.dto.size = total_len; > ^^^^^^^^ > would crash here anyways. > > -tduffy > From rminnich at lanl.gov Thu Jun 23 14:44:28 2005 From: rminnich at lanl.gov (Ronald G. Minnich) Date: Thu, 23 Jun 2005 15:44:28 -0600 (MDT) Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050623214224.GB14461@mellanox.co.il> References: <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> <52k6klm1kc.fsf@topspin.com> <20050623204836.GA14000@mellanox.co.il> <20050623214224.GB14461@mellanox.co.il> Message-ID: On Fri, 24 Jun 2005, Michael S. Tsirkin wrote: > I had this impression that I can have a .so not being present on the > slave at boot, and then dlopen could pull it across the network with > some custom protocol without going over NFS. not at present, at least on bproc. dlopen needs a path name. thanks ron From rminnich at lanl.gov Thu Jun 23 14:45:07 2005 From: rminnich at lanl.gov (Ronald G. Minnich) Date: Thu, 23 Jun 2005 15:45:07 -0600 (MDT) Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050623214224.GB14461@mellanox.co.il> References: <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> <52k6klm1kc.fsf@topspin.com> <20050623204836.GA14000@mellanox.co.il> <20050623214224.GB14461@mellanox.co.il> Message-ID: On Fri, 24 Jun 2005, Michael S. Tsirkin wrote: > I had this impression that I can have a .so not being present on the slave > at boot, and then dlopen could pull it across the network with some > custom protocol without going over NFS. > > And I was asking, if so, what other calls can do this besides dlopen? but: we did have v9fs support integrated into bproc so that, e.g., you could say something like addmount /home/rminnich and then when your process started up on the slave node, that file system would be there in your private name space. ron From jlentini at netapp.com Thu Jun 23 14:46:15 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 17:46:15 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove dapl_os_assert() In-Reply-To: <1119558325.21702.1.camel@duffman> References: <1119558325.21702.1.camel@duffman> Message-ID: Committed in revision 2704. On Thu, 23 Jun 2005, Tom Duffy wrote: tduffy> This patch removes dapl_os_assert(). In most cases, replacing with tduffy> BUG_ON(!). Some cases, I just called panic() where others I removed the tduffy> assert all together because the next line dereferences the pointer tduffy> anyways. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_cookie.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cookie.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) tduffy> @@ -184,7 +184,7 @@ u32 dapl_cb_get(struct dapl_cookie_buffe tduffy> u32 dat_status; tduffy> int new_head; tduffy> tduffy> - dapl_os_assert(NULL != cookie_ptr); tduffy> + BUG_ON(cookie_ptr == NULL); tduffy> tduffy> new_head = (atomic_read(&buffer->head) + 1) % buffer->pool_size; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -783,7 +783,7 @@ u32 dapl_ib_get_gid(struct ib_device *hc tduffy> { tduffy> int status; tduffy> tduffy> - dapl_os_assert(hca); tduffy> + BUG_ON(!hca); tduffy> tduffy> if (gid) { tduffy> status = ib_query_gid(hca, port, 0, gid); tduffy> Index: linux-kernel/dat-provider/dapl_openib_qp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) tduffy> @@ -76,7 +76,6 @@ u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ tduffy> tduffy> attr = &ep_ptr->param.ep_attr; tduffy> tduffy> - dapl_os_assert(ep_ptr->param.pz != NULL); tduffy> ib_pd_handle = ((struct dapl_pz *)ep_ptr->param.pz)->pd; tduffy> ib_hca_handle = ia_ptr->hca->ib_hca_handle; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_openib_dto.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_dto.h (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_openib_dto.h (working copy) tduffy> @@ -43,8 +43,6 @@ static inline u32 dapl_ib_post_recv(stru tduffy> struct ib_sge *sg_list; tduffy> int status, i, total_len = 0; tduffy> tduffy> - dapl_os_assert(NULL != cookie); tduffy> - tduffy> sg_list = ep->recv_iov; tduffy> for (i = 0; i < num_segments; i++, sg_list++) { tduffy> sg_list->addr = local_iov[i].virtual_address; tduffy> @@ -84,8 +82,6 @@ static inline u32 dapl_ib_post_send(stru tduffy> struct ib_sge *sg_list; tduffy> int status, i, total_len = 0; tduffy> tduffy> - dapl_os_assert(NULL != cookie); tduffy> - tduffy> sg_list = ep->send_iov; tduffy> for (i = 0; i < num_segments; i++, sg_list++) { tduffy> sg_list->addr = local_iov[i].virtual_address; tduffy> Index: linux-kernel/dat-provider/dapl_ia.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_ia.c (working copy) tduffy> @@ -371,13 +371,13 @@ bail: tduffy> */ tduffy> void dapl_ia_free(struct dapl_ia *ia) tduffy> { tduffy> - dapl_os_assert(ia->async_error_evd == NULL); tduffy> - dapl_os_assert(list_empty(&ia->lmr_list)); tduffy> - dapl_os_assert(list_empty(&ia->rmr_list)); tduffy> - dapl_os_assert(list_empty(&ia->ep_list)); tduffy> - dapl_os_assert(list_empty(&ia->evd_list)); tduffy> - dapl_os_assert(list_empty(&ia->psp_list)); tduffy> - dapl_os_assert(list_empty(&ia->rsp_list)); tduffy> + BUG_ON(ia->async_error_evd != NULL); tduffy> + BUG_ON(!list_empty(&ia->lmr_list)); tduffy> + BUG_ON(!list_empty(&ia->rmr_list)); tduffy> + BUG_ON(!list_empty(&ia->ep_list)); tduffy> + BUG_ON(!list_empty(&ia->evd_list)); tduffy> + BUG_ON(!list_empty(&ia->psp_list)); tduffy> + BUG_ON(!list_empty(&ia->rsp_list)); tduffy> tduffy> dapl_hca_unlink_ia(ia->hca, ia); tduffy> /* no need to destroy ia->common.lock */ tduffy> Index: linux-kernel/dat-provider/dapl_rmr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_rmr.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_rmr.c (working copy) tduffy> @@ -126,8 +126,7 @@ static u64 dapl_rmr_get_address(DAT_REGI tduffy> * DAT_MEM_TYPE_IA tduffy> * DAT_MEM_TYPE_BYPASS tduffy> */ tduffy> - dapl_os_assert(0); tduffy> - return 0; tduffy> + panic("unimplemented or unknown memory type\n"); tduffy> } tduffy> } tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ep.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_ep.c (working copy) tduffy> @@ -675,8 +675,8 @@ u32 dapl_ep_free(struct dat_ep *ep) tduffy> */ tduffy> (void)dapl_ep_disconnect((struct dat_ep *)ep_ptr, tduffy> DAT_CLOSE_ABRUPT_FLAG); tduffy> - dapl_os_assert(ep_ptr->param.ep_state == DAT_EP_STATE_DISCONNECTED || tduffy> - ep_ptr->param.ep_state == DAT_EP_STATE_UNCONNECTED); tduffy> + BUG_ON(ep_ptr->param.ep_state != DAT_EP_STATE_DISCONNECTED && tduffy> + ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED); tduffy> tduffy> /* tduffy> * Do verification of parameters and the state change atomically. tduffy> @@ -1538,11 +1538,11 @@ u32 dapl_ep_modify(struct dat_ep *ep, en tduffy> * occurred. But they're important to the logic of this routine, tduffy> * so we check. tduffy> */ tduffy> - dapl_os_assert(ep1 == ep2); tduffy> - dapl_os_assert(ep_attr2.max_recv_dtos == ep_attr1.max_recv_dtos); tduffy> - dapl_os_assert(ep_attr2.max_request_dtos == ep_attr1.max_request_dtos); tduffy> - dapl_os_assert(ep_attr2.max_recv_iov == ep_attr1.max_recv_iov); tduffy> - dapl_os_assert(ep_attr2.max_request_iov == ep_attr1.max_request_iov); tduffy> + BUG_ON(ep1 != ep2); tduffy> + BUG_ON(ep_attr2.max_recv_dtos != ep_attr1.max_recv_dtos); tduffy> + BUG_ON(ep_attr2.max_request_dtos != ep_attr1.max_request_dtos); tduffy> + BUG_ON(ep_attr2.max_recv_iov != ep_attr1.max_recv_iov); tduffy> + BUG_ON(ep_attr2.max_request_iov != ep_attr1.max_request_iov); tduffy> tduffy> copy_of_old_ep = *ep2; tduffy> tduffy> @@ -1607,10 +1607,10 @@ u32 dapl_ep_modify(struct dat_ep *ep, en tduffy> * because the parameter validate routine should protect us, tduffy> * but it's an important enough point that we assert it. tduffy> */ tduffy> - dapl_os_assert((ep2->param.ep_state tduffy> - != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING) tduffy> - && (ep2->param.ep_state tduffy> - != DAT_EP_STATE_ACTIVE_CONNECTION_PENDING)); tduffy> + BUG_ON(ep2->param.ep_state == tduffy> + DAT_EP_STATE_PASSIVE_CONNECTION_PENDING || tduffy> + ep2->param.ep_state == tduffy> + DAT_EP_STATE_ACTIVE_CONNECTION_PENDING); tduffy> tduffy> new_ep.qp = alloc_ep.qp; tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_cr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cr.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_cr.c (working copy) tduffy> @@ -271,7 +271,7 @@ static u32 dapl_connection_request(struc tduffy> DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING; tduffy> } else { tduffy> /* RSP */ tduffy> - dapl_os_assert(sp->sp.type == DAT_SP_TYPE_RSP); tduffy> + BUG_ON(sp->sp.type != DAT_SP_TYPE_RSP); tduffy> ep->param.ep_state = tduffy> DAT_EP_STATE_PASSIVE_CONNECTION_PENDING; tduffy> } tduffy> @@ -516,8 +516,7 @@ void dapl_cr_callback(struct dapl_cm_ctx tduffy> break; tduffy> default: tduffy> evd = NULL; tduffy> - dapl_os_assert(0); /* shouldn't happen */ tduffy> - break; tduffy> + panic("unknown callback event\n"); tduffy> } tduffy> tduffy> if (evd != NULL) tduffy> Index: linux-kernel/dat-provider/dapl_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_evd.c (working copy) tduffy> @@ -243,7 +243,7 @@ static u32 dapl_evd_dealloc(struct dapl_ tduffy> u32 status = DAT_SUCCESS; tduffy> struct dapl_ia *ia; tduffy> tduffy> - dapl_os_assert(atomic_read(&evd->evd_ref_count) == 0); tduffy> + BUG_ON(atomic_read(&evd->evd_ref_count) != 0); tduffy> tduffy> /* tduffy> * Destroy the CQ first, to keep any more callbacks from coming tduffy> @@ -330,9 +330,9 @@ static void dapl_evd_post_event(struct d tduffy> event->event_number); tduffy> tduffy> status = dapl_rbuf_add(&evd->pending_event_queue, event); tduffy> - dapl_os_assert(status == DAT_SUCCESS); tduffy> + BUG_ON(status != DAT_SUCCESS); tduffy> tduffy> - dapl_os_assert(evd->evd_state == DAPL_EVD_STATE_OPEN); tduffy> + BUG_ON(evd->evd_state != DAPL_EVD_STATE_OPEN); tduffy> tduffy> if (evd->evd_producer_locking_needed) tduffy> spin_unlock_irqrestore(&evd->common.lock, tduffy> @@ -522,10 +522,9 @@ static void dapl_evd_wc_to_event(struct tduffy> dto_status = dapl_ib_get_dto_status(wc); tduffy> tduffy> cookie = (struct dapl_cookie *) (unsigned long) wc->wr_id; tduffy> - dapl_os_assert(NULL != cookie); tduffy> tduffy> ep = cookie->ep; tduffy> - dapl_os_assert(NULL != ep); tduffy> + BUG_ON(ep == NULL); tduffy> tduffy> event->evd = (struct dat_evd *)evd; tduffy> tduffy> @@ -550,26 +549,6 @@ static void dapl_evd_wc_to_event(struct tduffy> event_data->user_cookie = cookie->val.dto.cookie; tduffy> event_data->status = dto_status; tduffy> tduffy> -#if 0 tduffy> - /* Currently mthca is not setting the opcode in */ tduffy> - /* succesful wc. The opcode will be IB_WC_SEND or */ tduffy> - /* IB_WC_RECV according the is_send bit */ tduffy> - /* We can not check the following assert for now */ tduffy> - if (dto_status == DAT_DTO_SUCCESS) { tduffy> - enum ib_wc_opcode ib_opcode = wc->opcode; tduffy> - enum dapl_dto_type dto_type = cookie->val.dto.type; tduffy> - dapl_os_assert((ib_opcode == IB_WC_SEND && tduffy> - dto_type == DAPL_DTO_TYPE_SEND) || tduffy> - (ib_opcode == IB_WC_RECV && tduffy> - dto_type == DAPL_DTO_TYPE_RECV) || tduffy> - (ib_opcide == IB_WC_RDMA_WRITE && tduffy> - dto_type == DAPL_DTO_TYPE_RDMA_WRITE) || tduffy> - (ib_opcode == IB_WC_RDMA_READ && tduffy> - dto_type == DAPL_DTO_TYPE_RDMA_READ)); tduffy> - tduffy> - } tduffy> -#endif tduffy> - tduffy> if (cookie->val.dto.type == DAPL_DTO_TYPE_SEND || tduffy> cookie->val.dto.type == DAPL_DTO_TYPE_RDMA_WRITE) { tduffy> /* Get size from DTO; CQE value may be off. */ tduffy> @@ -593,7 +572,7 @@ static void dapl_evd_wc_to_event(struct tduffy> event_data->user_cookie = cookie->val.rmr.cookie; tduffy> tduffy> if (dto_status == DAT_DTO_SUCCESS) { tduffy> - dapl_os_assert(wc->opcode == IB_WC_BIND_MW); tduffy> + BUG_ON(wc->opcode != IB_WC_BIND_MW); tduffy> event_data->status = DAT_RMR_BIND_SUCCESS; tduffy> } else { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, tduffy> @@ -607,10 +586,7 @@ static void dapl_evd_wc_to_event(struct tduffy> break; tduffy> } tduffy> default: tduffy> - { tduffy> - dapl_os_assert(!"Invalid Operation type"); tduffy> - break; tduffy> - } tduffy> + panic("Invalid Operation type\n"); tduffy> } /* end switch */ tduffy> } tduffy> tduffy> @@ -652,7 +628,7 @@ void dapl_evd_qp_async_error_callback(st tduffy> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> } tduffy> tduffy> - dapl_os_assert(async_evd != NULL); tduffy> + BUG_ON(async_evd == NULL); tduffy> tduffy> status = dapl_ib_get_async_event(cause, &async_event); tduffy> if (status == DAT_SUCCESS) { tduffy> @@ -792,9 +768,7 @@ void dapl_evd_connection_callback(struct tduffy> default: tduffy> spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); tduffy> evd = NULL; tduffy> - tduffy> - dapl_os_assert(0); /* shouldn't happen */ tduffy> - break; tduffy> + panic("bad event\n"); tduffy> } tduffy> tduffy> /* tduffy> @@ -838,7 +812,7 @@ static void dapl_evd_dto_callback(struct tduffy> tduffy> evd = (struct dapl_evd *)user_context; tduffy> tduffy> - dapl_os_assert(evd->cq == cq); tduffy> + BUG_ON(evd->cq != cq); tduffy> tduffy> /* Read once. */ tduffy> state = *(volatile enum dapl_evd_state *)&evd->evd_state; tduffy> Index: linux-kernel/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.h (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_util.h (working copy) tduffy> @@ -48,15 +48,6 @@ tduffy> #include tduffy> #endif tduffy> tduffy> -#define dapl_os_assert(expression) \ tduffy> - do { \ tduffy> - if (!(expression)) { \ tduffy> - panic("ASSERTION fail in %s:%i:%s\n", \ tduffy> - __FILE__, __LINE__, __func__); \ tduffy> - } \ tduffy> - } while (0) tduffy> - tduffy> - tduffy> /* dapl_os_atomic_assign tduffy> * tduffy> * assign 'new_value' to '*v' if the current value tduffy> Index: linux-kernel/dat-provider/dapl_sp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp.c (revision 2701) tduffy> +++ linux-kernel/dat-provider/dapl_sp.c (working copy) tduffy> @@ -64,7 +64,7 @@ static struct dapl_sp *dapl_sp_alloc(str tduffy> tduffy> void dapl_sp_dealloc(struct dapl_sp *sp) tduffy> { tduffy> - dapl_os_assert(list_empty(&sp->cr_list)); tduffy> + BUG_ON(!list_empty(&sp->cr_list)); tduffy> tduffy> kfree(sp); tduffy> } tduffy> tduffy> From mst at mellanox.co.il Thu Jun 23 15:01:54 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 24 Jun 2005 01:01:54 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: References: <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> <52k6klm1kc.fsf@topspin.com> <20050623204836.GA14000@mellanox.co.il> <20050623214224.GB14461@mellanox.co.il> Message-ID: <20050623220154.GE14461@mellanox.co.il> Quoting r. Ronald G. Minnich : > Subject: Re: [PATCH] rdma_lat-09 and results > > > > On Fri, 24 Jun 2005, Michael S. Tsirkin wrote: > > > I had this impression that I can have a .so not being present on the > > slave at boot, and then dlopen could pull it across the network with > > some custom protocol without going over NFS. > > not at present, at least on bproc. dlopen needs a path name. So, if you want to run without nfs (or such), you basically need to link the applications statically, is that right? -- MST From rminnich at lanl.gov Thu Jun 23 15:04:37 2005 From: rminnich at lanl.gov (Ronald G. Minnich) Date: Thu, 23 Jun 2005 16:04:37 -0600 (MDT) Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050623220154.GE14461@mellanox.co.il> References: <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> <52k6klm1kc.fsf@topspin.com> <20050623204836.GA14000@mellanox.co.il> <20050623214224.GB14461@mellanox.co.il> <20050623220154.GE14461@mellanox.co.il> Message-ID: On Fri, 24 Jun 2005, Michael S. Tsirkin wrote: > So, if you want to run without nfs (or such), you basically need to link > the applications statically, is that right? the .so files you want have to be in /lib on the node, e.g. rminnich at hyrax ~]$ bpsh 0 ls /lib ld-2.3.3.so ld-linux.so.2 libdl-2.3.3.so libdl.so.2 libgcc_s-3.4.2-20041018.so.1 libgcc_s.so.1 libnsl-2.3.3.so libnsl.so.1 libnss_bproc.so.2 libresolv-2.3.3.so libresolv.so.2 tls OR you have to have them mounted via nfs,v9fs, or whatever, OR you have to copy the files out as part of running the app: bpcp blah.so 0:/tmp bpsh 0 someprogram OR you link statically There's no one answer. thanks ron From Thomas.Talpey at netapp.com Thu Jun 23 15:37:35 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Thu, 23 Jun 2005 18:37:35 -0400 Subject: [openib-general] Re: [PATCH] kDAPL: remove dapl_os_assert() In-Reply-To: <20050623210917.GQ22934@esmail.cup.hp.com> References: <1119558325.21702.1.camel@duffman> <20050623210917.GQ22934@esmail.cup.hp.com> Message-ID: <6.2.1.2.2.20050623183442.056a5eb0@exnane01.nane.netapp.com> At 05:09 PM 6/23/2005, Grant Grundler wrote: >On Thu, Jun 23, 2005 at 04:55:38PM -0400, James Lentini wrote: >> My argument in favor of retaining them is that dapl_evd_wc_to_event() >> will crash if the cookie NULL. A BUG_ON will detect this situation ... >The tombstone from the data page fault panic should make this nearly >as obvious as the BUG_ON(). Yes, I agree a BUG_ON() is completely >obvious. But it's also burning CPU cycles for something that Not to argue one way or the other, but if this cookie is NULL, whose fault would that be? I think that should govern whether it's common enough to warrant BUG_ON or rare enough to warrant a straight crash. I would suggest BUG_ON only if it were possible to trigger this from a loadable module, etc. Tom. From tduffy at sun.com Thu Jun 23 16:11:06 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 23 Jun 2005 16:11:06 -0700 Subject: [openib-general] [PATCH] kDAPL: cstyle dat-provider/dapl_ring_buffer_util.c Message-ID: <1119568266.21702.11.camel@duffman> Signed-off-by: Tom Duffy Index: linux-kernel-work/dat-provider/dapl_ring_buffer_util.c =================================================================== --- linux-kernel-work/dat-provider/dapl_ring_buffer_util.c (revision 2704) +++ linux-kernel-work/dat-provider/dapl_ring_buffer_util.c (working copy) @@ -63,9 +63,8 @@ u32 dapl_rbuf_alloc(struct dapl_ring_buf /* Put size on a power of 2 boundary */ rsize = 1; - while ((int) rsize < size) { + while ((int) rsize < size) rsize <<= 1; - } rbuf->base = kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); if (rbuf->base) { @@ -100,42 +99,37 @@ u32 dapl_rbuf_alloc(struct dapl_ring_buf */ u32 dapl_rbuf_realloc(struct dapl_ring_buffer *rbuf, int size) { - struct dapl_ring_buffer new_rbuf; - void *entry; - u32 dat_status = DAT_SUCCESS; - - /* decreasing the size or retaining the old size is not allowed */ - if (size <= rbuf->lim + 1) - { - dat_status = DAT_ERROR (DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - /* - * !This is NOT ATOMIC! - * Simple algorithm: Allocate a new ring buffer, take everything - * out of the old one and put it in the new one, and release the - * old base buffer. - */ - dat_status = dapl_rbuf_alloc (&new_rbuf, size); - if (dat_status != DAT_SUCCESS) - { - goto bail; - } - - while ( (entry = dapl_rbuf_remove(rbuf)) != NULL) - { - /* We know entries will fit so ignore the return code */ - (void)dapl_rbuf_add (&new_rbuf, entry); - } + struct dapl_ring_buffer new_rbuf; + void *entry; + u32 status = DAT_SUCCESS; + + /* decreasing the size or retaining the old size is not allowed */ + if (size <= rbuf->lim + 1) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + goto bail; + } + + /* + * !This is NOT ATOMIC! + * Simple algorithm: Allocate a new ring buffer, take everything + * out of the old one and put it in the new one, and release the + * old base buffer. + */ + status = dapl_rbuf_alloc(&new_rbuf, size); + if (status != DAT_SUCCESS) + goto bail; + + while ((entry = dapl_rbuf_remove(rbuf)) != NULL) + /* We know entries will fit so ignore the return code */ + (void)dapl_rbuf_add(&new_rbuf, entry); - /* release the old base buffer */ - kfree(rbuf->base); + /* release the old base buffer */ + kfree(rbuf->base); - *rbuf = new_rbuf; + *rbuf = new_rbuf; - bail: - return dat_status; +bail: + return status; } /* @@ -155,9 +149,8 @@ u32 dapl_rbuf_realloc(struct dapl_ring_b */ void dapl_rbuf_destroy(struct dapl_ring_buffer *rbuf) { - if ((NULL == rbuf) || (NULL == rbuf->base)) { + if ((NULL == rbuf) || (NULL == rbuf->base)) return; - } kfree(rbuf->base); rbuf->base = NULL; @@ -221,22 +214,18 @@ u32 dapl_rbuf_add(struct dapl_ring_buffe */ void *dapl_rbuf_remove(struct dapl_ring_buffer *rbuf) { - int pos; - int val; + int pos, val; - while (atomic_read(&rbuf->head) != - atomic_read(&rbuf->tail)) { + while (atomic_read(&rbuf->head) != atomic_read(&rbuf->tail)) { pos = atomic_read(&rbuf->tail); val = dapl_os_atomic_assign(&rbuf->tail, pos, pos + 1); if (val == pos) { pos = (pos + 1) & rbuf->lim; /* verify in range */ - return rbuf->base[pos]; } } return NULL; - } /* @@ -261,12 +250,10 @@ int dapl_rbuf_count(struct dapl_ring_buf head = atomic_read(&rbuf->head) & rbuf->lim; tail = atomic_read(&rbuf->tail) & rbuf->lim; - if (head > tail) { + if (head > tail) count = head - tail; - } else { - /* add 1 to lim as it is a mask, number of entries - 1 */ + else /* add 1 to lim as it is a mask, number of entries - 1 */ count = (rbuf->lim + 1 - tail + head) & rbuf->lim; - } return count; } @@ -292,12 +279,11 @@ int dapl_rbuf_count(struct dapl_ring_buf */ void dapl_rbuf_adjust(struct dapl_ring_buffer *rbuf, unsigned long offset) { - int pos; + int pos; - pos = atomic_read(&rbuf->head); - while ( pos != atomic_read(&rbuf->tail) ) - { - rbuf->base[pos] = rbuf->base[pos] + offset; - pos = (pos + 1) & rbuf->lim; /* verify in range */ - } + pos = atomic_read(&rbuf->head); + while (pos != atomic_read(&rbuf->tail)) { + rbuf->base[pos] = rbuf->base[pos] + offset; + pos = (pos + 1) & rbuf->lim; /* verify in range */ + } } From tduffy at sun.com Thu Jun 23 16:14:36 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 23 Jun 2005 16:14:36 -0700 Subject: [openib-general] [PATCH] kDAPL: remove dat_os_realloc() Message-ID: <1119568476.21702.13.camel@duffman> There was only one user of the dat_os_realloc() function. Roll it into the base function. Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2704) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -191,13 +191,15 @@ bail: static u32 dapl_evd_event_realloc(struct dapl_evd *evd, int qlen) { struct dat_event *events; - int i, old_qlen, diff; + int i, old_qlen, diff, size; u32 status; - /* Allocate EVENTs */ - events = (struct dat_event *)dapl_os_realloc(evd->events, - qlen * sizeof *events); - if (!events) { + size = qlen * sizeof *events; + events = kmalloc(size, GFP_ATOMIC); + if (events) { + memcpy(events, evd->events, size); + kfree(evd->events); + } else { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; Index: linux-kernel/dat-provider/dapl_util.h =================================================================== --- linux-kernel/dat-provider/dapl_util.h (revision 2704) +++ linux-kernel/dat-provider/dapl_util.h (working copy) @@ -89,23 +89,6 @@ static inline int dapl_os_atomic_assign( } /* - * Memory Functions - */ - -static inline void *dapl_os_realloc(void *ptr, int size) -{ - void *newptr; - - newptr = kmalloc(size, GFP_ATOMIC); - if (newptr) { - /* copy the mem array to the new */ - memcpy(newptr, ptr, size); - kfree(ptr); - } - return newptr; -} - -/* * *printf format helper. We use the C string constant concatenation * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. From tomduffy at gmail.com Thu Jun 23 16:16:26 2005 From: tomduffy at gmail.com (Tom Duffy) Date: Thu, 23 Jun 2005 16:16:26 -0700 Subject: [openib-general] [PATCH] kDAPL: remove dapl_os_realloc() In-Reply-To: <1119568476.21702.13.camel@duffman> References: <1119568476.21702.13.camel@duffman> Message-ID: <9d3b7de705062316167afa18aa@mail.gmail.com> On 6/23/05, Tom Duffy wrote: > There was only one user of the dat_os_realloc() function. Oops. That should of course read *dapl*_os_realloc(). -tduffy From jlentini at netapp.com Thu Jun 23 19:27:40 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 22:27:40 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: cstyle dat-provider/dapl_ring_buffer_util.c In-Reply-To: <1119568266.21702.11.camel@duffman> References: <1119568266.21702.11.camel@duffman> Message-ID: Committed in revision 2705. On Thu, 23 Jun 2005, Tom Duffy wrote: tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-work/dat-provider/dapl_ring_buffer_util.c tduffy> =================================================================== tduffy> --- linux-kernel-work/dat-provider/dapl_ring_buffer_util.c (revision 2704) tduffy> +++ linux-kernel-work/dat-provider/dapl_ring_buffer_util.c (working copy) tduffy> @@ -63,9 +63,8 @@ u32 dapl_rbuf_alloc(struct dapl_ring_buf tduffy> tduffy> /* Put size on a power of 2 boundary */ tduffy> rsize = 1; tduffy> - while ((int) rsize < size) { tduffy> + while ((int) rsize < size) tduffy> rsize <<= 1; tduffy> - } tduffy> tduffy> rbuf->base = kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); tduffy> if (rbuf->base) { tduffy> @@ -100,42 +99,37 @@ u32 dapl_rbuf_alloc(struct dapl_ring_buf tduffy> */ tduffy> u32 dapl_rbuf_realloc(struct dapl_ring_buffer *rbuf, int size) tduffy> { tduffy> - struct dapl_ring_buffer new_rbuf; tduffy> - void *entry; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - /* decreasing the size or retaining the old size is not allowed */ tduffy> - if (size <= rbuf->lim + 1) tduffy> - { tduffy> - dat_status = DAT_ERROR (DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * !This is NOT ATOMIC! tduffy> - * Simple algorithm: Allocate a new ring buffer, take everything tduffy> - * out of the old one and put it in the new one, and release the tduffy> - * old base buffer. tduffy> - */ tduffy> - dat_status = dapl_rbuf_alloc (&new_rbuf, size); tduffy> - if (dat_status != DAT_SUCCESS) tduffy> - { tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - while ( (entry = dapl_rbuf_remove(rbuf)) != NULL) tduffy> - { tduffy> - /* We know entries will fit so ignore the return code */ tduffy> - (void)dapl_rbuf_add (&new_rbuf, entry); tduffy> - } tduffy> + struct dapl_ring_buffer new_rbuf; tduffy> + void *entry; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + /* decreasing the size or retaining the old size is not allowed */ tduffy> + if (size <= rbuf->lim + 1) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* tduffy> + * !This is NOT ATOMIC! tduffy> + * Simple algorithm: Allocate a new ring buffer, take everything tduffy> + * out of the old one and put it in the new one, and release the tduffy> + * old base buffer. tduffy> + */ tduffy> + status = dapl_rbuf_alloc(&new_rbuf, size); tduffy> + if (status != DAT_SUCCESS) tduffy> + goto bail; tduffy> + tduffy> + while ((entry = dapl_rbuf_remove(rbuf)) != NULL) tduffy> + /* We know entries will fit so ignore the return code */ tduffy> + (void)dapl_rbuf_add(&new_rbuf, entry); tduffy> tduffy> - /* release the old base buffer */ tduffy> - kfree(rbuf->base); tduffy> + /* release the old base buffer */ tduffy> + kfree(rbuf->base); tduffy> tduffy> - *rbuf = new_rbuf; tduffy> + *rbuf = new_rbuf; tduffy> tduffy> - bail: tduffy> - return dat_status; tduffy> +bail: tduffy> + return status; tduffy> } tduffy> tduffy> /* tduffy> @@ -155,9 +149,8 @@ u32 dapl_rbuf_realloc(struct dapl_ring_b tduffy> */ tduffy> void dapl_rbuf_destroy(struct dapl_ring_buffer *rbuf) tduffy> { tduffy> - if ((NULL == rbuf) || (NULL == rbuf->base)) { tduffy> + if ((NULL == rbuf) || (NULL == rbuf->base)) tduffy> return; tduffy> - } tduffy> tduffy> kfree(rbuf->base); tduffy> rbuf->base = NULL; tduffy> @@ -221,22 +214,18 @@ u32 dapl_rbuf_add(struct dapl_ring_buffe tduffy> */ tduffy> void *dapl_rbuf_remove(struct dapl_ring_buffer *rbuf) tduffy> { tduffy> - int pos; tduffy> - int val; tduffy> + int pos, val; tduffy> tduffy> - while (atomic_read(&rbuf->head) != tduffy> - atomic_read(&rbuf->tail)) { tduffy> + while (atomic_read(&rbuf->head) != atomic_read(&rbuf->tail)) { tduffy> pos = atomic_read(&rbuf->tail); tduffy> val = dapl_os_atomic_assign(&rbuf->tail, pos, pos + 1); tduffy> if (val == pos) { tduffy> pos = (pos + 1) & rbuf->lim; /* verify in range */ tduffy> - tduffy> return rbuf->base[pos]; tduffy> } tduffy> } tduffy> tduffy> return NULL; tduffy> - tduffy> } tduffy> tduffy> /* tduffy> @@ -261,12 +250,10 @@ int dapl_rbuf_count(struct dapl_ring_buf tduffy> tduffy> head = atomic_read(&rbuf->head) & rbuf->lim; tduffy> tail = atomic_read(&rbuf->tail) & rbuf->lim; tduffy> - if (head > tail) { tduffy> + if (head > tail) tduffy> count = head - tail; tduffy> - } else { tduffy> - /* add 1 to lim as it is a mask, number of entries - 1 */ tduffy> + else /* add 1 to lim as it is a mask, number of entries - 1 */ tduffy> count = (rbuf->lim + 1 - tail + head) & rbuf->lim; tduffy> - } tduffy> tduffy> return count; tduffy> } tduffy> @@ -292,12 +279,11 @@ int dapl_rbuf_count(struct dapl_ring_buf tduffy> */ tduffy> void dapl_rbuf_adjust(struct dapl_ring_buffer *rbuf, unsigned long offset) tduffy> { tduffy> - int pos; tduffy> + int pos; tduffy> tduffy> - pos = atomic_read(&rbuf->head); tduffy> - while ( pos != atomic_read(&rbuf->tail) ) tduffy> - { tduffy> - rbuf->base[pos] = rbuf->base[pos] + offset; tduffy> - pos = (pos + 1) & rbuf->lim; /* verify in range */ tduffy> - } tduffy> + pos = atomic_read(&rbuf->head); tduffy> + while (pos != atomic_read(&rbuf->tail)) { tduffy> + rbuf->base[pos] = rbuf->base[pos] + offset; tduffy> + pos = (pos + 1) & rbuf->lim; /* verify in range */ tduffy> + } tduffy> } tduffy> From kjreilly at us.ibm.com Thu Jun 23 19:34:54 2005 From: kjreilly at us.ibm.com (Kevin Reilly) Date: Thu, 23 Jun 2005 22:34:54 -0400 Subject: [openib-general] Asynchronous Event notification and registration Message-ID: People need to excuse all my annoying startup questions. I just notices that the user space ib verbs lib doesn't have any functions to register completion or event handlers. Is the intent that the API will only support polling or are the asynchronous event handling stuff in somebodies branch someplace? Kevin J. Reilly STSM, HPC Architecture -Federation/HPS Chief Engineer -HPC interconnect architect (office) 845-433-7976 (tieline) 8-293-7976 From jlentini at netapp.com Thu Jun 23 19:37:00 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 23 Jun 2005 22:37:00 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove dat_os_realloc() In-Reply-To: <1119568476.21702.13.camel@duffman> References: <1119568476.21702.13.camel@duffman> Message-ID: What do you think about doing it this way? (see attached) On Thu, 23 Jun 2005, Tom Duffy wrote: tduffy> There was only one user of the dat_os_realloc() function. Roll it into tduffy> the base function. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd.c (revision 2704) tduffy> +++ linux-kernel/dat-provider/dapl_evd.c (working copy) tduffy> @@ -191,13 +191,15 @@ bail: tduffy> static u32 dapl_evd_event_realloc(struct dapl_evd *evd, int qlen) tduffy> { tduffy> struct dat_event *events; tduffy> - int i, old_qlen, diff; tduffy> + int i, old_qlen, diff, size; tduffy> u32 status; tduffy> tduffy> - /* Allocate EVENTs */ tduffy> - events = (struct dat_event *)dapl_os_realloc(evd->events, tduffy> - qlen * sizeof *events); tduffy> - if (!events) { tduffy> + size = qlen * sizeof *events; tduffy> + events = kmalloc(size, GFP_ATOMIC); tduffy> + if (events) { tduffy> + memcpy(events, evd->events, size); tduffy> + kfree(evd->events); tduffy> + } else { tduffy> status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> Index: linux-kernel/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.h (revision 2704) tduffy> +++ linux-kernel/dat-provider/dapl_util.h (working copy) tduffy> @@ -89,23 +89,6 @@ static inline int dapl_os_atomic_assign( tduffy> } tduffy> tduffy> /* tduffy> - * Memory Functions tduffy> - */ tduffy> - tduffy> -static inline void *dapl_os_realloc(void *ptr, int size) tduffy> -{ tduffy> - void *newptr; tduffy> - tduffy> - newptr = kmalloc(size, GFP_ATOMIC); tduffy> - if (newptr) { tduffy> - /* copy the mem array to the new */ tduffy> - memcpy(newptr, ptr, size); tduffy> - kfree(ptr); tduffy> - } tduffy> - return newptr; tduffy> -} tduffy> - tduffy> -/* tduffy> * *printf format helper. We use the C string constant concatenation tduffy> * ability to define 64 bit formats, which unfortunatly are non standard tduffy> * in the C compiler world. tduffy> -------------- next part -------------- Index: dat-provider/dapl_evd.c =================================================================== --- dat-provider/dapl_evd.c (revision 2704) +++ dat-provider/dapl_evd.c (working copy) @@ -191,19 +191,20 @@ bail: static u32 dapl_evd_event_realloc(struct dapl_evd *evd, int qlen) { struct dat_event *events; - int i, old_qlen, diff; + int i, old_qlen, diff, size; u32 status; - /* Allocate EVENTs */ - events = (struct dat_event *)dapl_os_realloc(evd->events, - qlen * sizeof *events); + size = qlen * sizeof *events; + events = kmalloc(size, GFP_ATOMIC); if (!events) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } + memcpy(events, evd->events, size); diff = events - evd->events; + kfree(evd->events); evd->events = events; old_qlen = evd->qlen; Index: dat-provider/dapl_util.h =================================================================== --- dat-provider/dapl_util.h (revision 2704) +++ dat-provider/dapl_util.h (working copy) @@ -89,23 +89,6 @@ static inline int dapl_os_atomic_assign( } /* - * Memory Functions - */ - -static inline void *dapl_os_realloc(void *ptr, int size) -{ - void *newptr; - - newptr = kmalloc(size, GFP_ATOMIC); - if (newptr) { - /* copy the mem array to the new */ - memcpy(newptr, ptr, size); - kfree(ptr); - } - return newptr; -} - -/* * *printf format helper. We use the C string constant concatenation * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. From roland at topspin.com Thu Jun 23 20:23:17 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 20:23:17 -0700 Subject: [openib-general] Asynchronous Event notification and registration In-Reply-To: (Kevin Reilly's message of "Thu, 23 Jun 2005 22:34:54 -0400") References: Message-ID: <52y890jv4q.fsf@topspin.com> Kevin> People need to excuse all my annoying startup questions. I Kevin> just notices that the user space ib verbs lib doesn't have Kevin> any functions to register completion or event handlers. Is Kevin> the intent that the API will only support polling or are Kevin> the asynchronous event handling stuff in somebodies branch Kevin> someplace? Everything you need should already be present. Completion and asynchronous events are delivered by reading special file descriptors that are created as part of the device context returned from ibv_open_device(). You don't need to do the read() yourself -- ibv_get_cq_event() and ibv_get_async_event() return the next available completion or asynchronous event, respectively. It is true that there is not a mechanism for receiving a callback when when an event is ready. However, this is by design because there is not really a single, efficient mechanism for kernel-to-user function calls, which works for all application architectures. You are free to detect that context->cq_fd[0] and/or context->async_fd have become readable in whatever way fits your application best. (Perhaps an API should be wrapped around these file descriptor members of the context structure so that applications don't have to look around directly in the struct) Some possibilities for detecting events are: - Use select()/poll()/epoll_wait() to monitor the FDs, along with any other FDs of interest to your event loop. - Have threads that block in ibv_get_cq_event() and/or ibv_get_async_event(). - Use SIGIO/POSIX realtime signals to detect events. may offer some inspiration. - R. From roland at topspin.com Thu Jun 23 20:31:16 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 20:31:16 -0700 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050623204836.GA14000@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 23 Jun 2005 23:48:36 +0300") References: <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622120743.GZ20041@mellanox.co.il> <52zmtis7ns.fsf@topspin.com> <20050622201706.GA2488@mellanox.co.il> <52wtomoyna.fsf@topspin.com> <20050623055835.GA10456@mellanox.co.il> <52k6klnjhz.fsf@topspin.com> <20050623164002.GA12420@mellanox.co.il> <52k6klm1kc.fsf@topspin.com> <20050623204836.GA14000@mellanox.co.il> Message-ID: <52u0jojurf.fsf@topspin.com> Roland> It seems like the best would be to have libmthca.a in Roland> $(libdir) and mthca.so in $(libdir)/infiniband. That way Roland> we make it easy to do static linking and avoid having two Roland> names for the same object. Michael> OK, but I dont know how to do that in autotools. Do you? Not sure, I'll poke around a little. Michael> Actually, whats the importance of it? It seems the user Michael> can compile libibverbs as readily as the plugin ... I think it's mostly useful once distributions start shipping libibverbs. It means someone can take binaries supplied by their distros and just install a new plugin driver in ~user/whatever, and then use whatever fancy new IB hardware they have. It's similar to how I can put libjavaplugin_oji.so in ~/.mozilla/plugins if I'm feeling impure. Michael> Solves this particular problem, but maybe, additionally, Michael> there should be a configuration option to disable Michael> OPENIB_DRIVER_PATH lookup? Would anyone really use this? How does it help? For processes not running SUID, a hostile user can already generate any conceivable binary, link in any and all libraries, and set LD_PRELOAD to whatever. Michael> I also wander about the user of strdupa - can it cause Michael> stack overflow? I doubt an environment variable can be made long enough to overflow the stack, but even if it can, who cares? We can only hit that strdupa() if the processes EUID == real UID, and in that case the user could already just run a program like while (1) { alloca(1000000); } - R. From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 02/14] IB/mthca: Clean up error messages In-Reply-To: <2005623214.8M2FOQ4JBL5cpK2n@topspin.com> Message-ID: <2005623214.mXu9RTJT2MIk7KOo@topspin.com> From: Bernhard Fischer - Fix incorrect cut-n-paste in error messages. - Add missing newlines in error messages. - Use DRV_NAME instead of "ib_mthca" in a couple of places. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_eq.c | 9 +++------ linux.git/drivers/infiniband/hw/mthca/mthca_main.c | 6 +++--- 2 files changed, 6 insertions(+), 9 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_eq.c 2005-06-23 13:03:02.247630576 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_eq.c 2005-06-23 13:03:03.703315530 -0700 @@ -615,8 +615,7 @@ static void mthca_free_eq(struct mthca_d if (err) mthca_warn(dev, "HW2SW_EQ failed (%d)\n", err); if (status) - mthca_warn(dev, "HW2SW_EQ returned status 0x%02x\n", - status); + mthca_warn(dev, "HW2SW_EQ returned status 0x%02x\n", status); dev->eq_table.arm_mask &= ~eq->eqn_mask; @@ -709,8 +708,7 @@ static int __devinit mthca_map_eq_regs(s if (mthca_map_reg(dev, ((pci_resource_len(dev->pdev, 0) - 1) & dev->fw.arbel.eq_arm_base) + 4, 4, &dev->eq_regs.arbel.eq_arm)) { - mthca_err(dev, "Couldn't map interrupt clear register, " - "aborting.\n"); + mthca_err(dev, "Couldn't map EQ arm register, aborting.\n"); mthca_unmap_reg(dev, (pci_resource_len(dev->pdev, 0) - 1) & dev->fw.arbel.clr_int_base, MTHCA_CLR_INT_SIZE, dev->clr_base); @@ -721,8 +719,7 @@ static int __devinit mthca_map_eq_regs(s dev->fw.arbel.eq_set_ci_base, MTHCA_EQ_SET_CI_SIZE, &dev->eq_regs.arbel.eq_set_ci_base)) { - mthca_err(dev, "Couldn't map interrupt clear register, " - "aborting.\n"); + mthca_err(dev, "Couldn't map EQ CI register, aborting.\n"); mthca_unmap_reg(dev, ((pci_resource_len(dev->pdev, 0) - 1) & dev->fw.arbel.eq_arm_base) + 4, 4, dev->eq_regs.arbel.eq_arm); --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_main.c 2005-06-23 13:03:02.630547703 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_main.c 2005-06-23 13:03:03.703315530 -0700 @@ -70,7 +70,7 @@ MODULE_PARM_DESC(msi, "attempt to use MS #endif /* CONFIG_PCI_MSI */ static const char mthca_version[] __devinitdata = - "ib_mthca: Mellanox InfiniBand HCA driver v" + DRV_NAME ": Mellanox InfiniBand HCA driver v" DRV_VERSION " (" DRV_RELDATE ")\n"; static struct mthca_profile default_profile = { @@ -928,13 +928,13 @@ static int __devinit mthca_init_one(stru */ if (!(pci_resource_flags(pdev, 0) & IORESOURCE_MEM) || pci_resource_len(pdev, 0) != 1 << 20) { - dev_err(&pdev->dev, "Missing DCS, aborting."); + dev_err(&pdev->dev, "Missing DCS, aborting.\n"); err = -ENODEV; goto err_disable_pdev; } if (!(pci_resource_flags(pdev, 2) & IORESOURCE_MEM) || pci_resource_len(pdev, 2) != 1 << 23) { - dev_err(&pdev->dev, "Missing UAR, aborting."); + dev_err(&pdev->dev, "Missing UAR, aborting.\n"); err = -ENODEV; goto err_disable_pdev; } @@ -1164,7 +1164,7 @@ static struct pci_device_id mthca_pci_ta MODULE_DEVICE_TABLE(pci, mthca_pci_table); static struct pci_driver mthca_driver = { - .name = "ib_mthca", + .name = DRV_NAME, .id_table = mthca_pci_table, .probe = mthca_init_one, .remove = __devexit_p(mthca_remove_one) From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 03/14] IB/mthca: Clean up CQ debug In-Reply-To: <2005623214.mXu9RTJT2MIk7KOo@topspin.com> Message-ID: <2005623214.Zy3c64TeILEv3G5E@topspin.com> Clean up CQ debugging code: make dump_cqe print on one line, and only dump error CQ entries for local operation errors. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_cq.c | 39 +++++++++++++++------------------ 1 files changed, 18 insertions(+), 21 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:02.629547920 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:04.326180727 -0700 @@ -172,6 +172,17 @@ static inline void set_cqe_hw(struct mth cqe->owner = MTHCA_CQ_ENTRY_OWNER_HW; } +static void dump_cqe(struct mthca_dev *dev, void *cqe_ptr) +{ + __be32 *cqe = cqe_ptr; + + (void) cqe; /* avoid warning if mthca_dbg compiled away... */ + mthca_dbg(dev, "CQE contents %08x %08x %08x %08x %08x %08x %08x %08x\n", + be32_to_cpu(cqe[0]), be32_to_cpu(cqe[1]), be32_to_cpu(cqe[2]), + be32_to_cpu(cqe[3]), be32_to_cpu(cqe[4]), be32_to_cpu(cqe[5]), + be32_to_cpu(cqe[6]), be32_to_cpu(cqe[7])); +} + /* * incr is ignored in native Arbel (mem-free) mode, so cq->cons_index * should be correct before calling update_cons_index(). @@ -281,16 +292,12 @@ static int handle_error_cqe(struct mthca int dbd; u32 new_wqe; - if (1 && cqe->syndrome != SYNDROME_WR_FLUSH_ERR) { - int j; - - mthca_dbg(dev, "%x/%d: error CQE -> QPN %06x, WQE @ %08x\n", - cq->cqn, cq->cons_index, be32_to_cpu(cqe->my_qpn), - be32_to_cpu(cqe->wqe)); - - for (j = 0; j < 8; ++j) - printk(KERN_DEBUG " [%2x] %08x\n", - j * 4, be32_to_cpu(((u32 *) cqe)[j])); + if (cqe->syndrome == SYNDROME_LOCAL_QP_OP_ERR) { + mthca_dbg(dev, "local QP operation err " + "(QPN %06x, WQE @ %08x, CQN %06x, index %d)\n", + be32_to_cpu(cqe->my_qpn), be32_to_cpu(cqe->wqe), + cq->cqn, cq->cons_index); + dump_cqe(dev, cqe); } /* @@ -378,15 +385,6 @@ static int handle_error_cqe(struct mthca return 0; } -static void dump_cqe(struct mthca_cqe *cqe) -{ - int j; - - for (j = 0; j < 8; ++j) - printk(KERN_DEBUG " [%2x] %08x\n", - j * 4, be32_to_cpu(((u32 *) cqe)[j])); -} - static inline int mthca_poll_one(struct mthca_dev *dev, struct mthca_cq *cq, struct mthca_qp **cur_qp, @@ -415,8 +413,7 @@ static inline int mthca_poll_one(struct mthca_dbg(dev, "%x/%d: CQE -> QPN %06x, WQE @ %08x\n", cq->cqn, cq->cons_index, be32_to_cpu(cqe->my_qpn), be32_to_cpu(cqe->wqe)); - - dump_cqe(cqe); + dump_cqe(dev, cqe); } is_error = (cqe->opcode & MTHCA_ERROR_CQE_OPCODE_MASK) == From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 00/14] IB/mthca: merge Message-ID: <2005623214.eJuSN72mmScT1do9@topspin.com> Here is a series of patches to the mthca Mellanox HCA driver. Some of the patches are largish but there shouldn't be anything too major here. This is mostly to bring my tree closer to the main kernel and make the merge of direct userspace access smaller, since it will require plenty of review. Thanks, Roland From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 01/14] IB/mthca: Add Sun copyright notice In-Reply-To: <2005623214.eJuSN72mmScT1do9@topspin.com> Message-ID: <2005623214.8M2FOQ4JBL5cpK2n@topspin.com> From: Tom Duffy Add Sun copyright to files modified by Tom Duffy. Signed-off-by: Tom Duffy Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_av.c | 1 + linux.git/drivers/infiniband/hw/mthca/mthca_cq.c | 1 + linux.git/drivers/infiniband/hw/mthca/mthca_dev.h | 1 + linux.git/drivers/infiniband/hw/mthca/mthca_doorbell.h | 1 + linux.git/drivers/infiniband/hw/mthca/mthca_main.c | 1 + linux.git/drivers/infiniband/hw/mthca/mthca_provider.c | 1 + 6 files changed, 6 insertions(+) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_av.c 2005-06-23 13:03:02.393598985 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_av.c 2005-06-23 13:03:02.629547920 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:02.393598985 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:02.629547920 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-23 13:03:02.393598985 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-23 13:03:02.630547703 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_doorbell.h 2005-06-23 13:03:02.393598985 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_doorbell.h 2005-06-23 13:03:02.630547703 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_main.c 2005-06-23 13:03:02.393598985 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_main.c 2005-06-23 13:03:02.630547703 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-23 13:03:02.393598985 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-23 13:03:02.629547920 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 05/14] IB/mthca: Set QP static rate correctly In-Reply-To: <2005623214.8EmcOxrdtkDFLmDq@topspin.com> Message-ID: <2005623214.ZulWzVKl5lpV91f5@topspin.com> Fix offset of static_rate in QP context. Pointed out by Dror Goldenberg. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_qp.c | 2 +- 1 files changed, 1 insertion(+), 1 deletion(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:04.751088766 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:05.234984039 -0700 @@ -683,7 +683,7 @@ int mthca_modify_qp(struct ib_qp *ibqp, if (attr_mask & IB_QP_AV) { qp_context->pri_path.g_mylmc = attr->ah_attr.src_path_bits & 0x7f; qp_context->pri_path.rlid = cpu_to_be16(attr->ah_attr.dlid); - qp_context->pri_path.static_rate = (!!attr->ah_attr.static_rate) << 3; + qp_context->pri_path.static_rate = !!attr->ah_attr.static_rate; if (attr->ah_attr.ah_flags & IB_AH_GRH) { qp_context->pri_path.g_mylmc |= 1 << 7; qp_context->pri_path.mgid_index = attr->ah_attr.grh.sgid_index; From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 04/14] IB/mthca: Use dma_alloc_coherent instead of pci_alloc_consistent In-Reply-To: <2005623214.Zy3c64TeILEv3G5E@topspin.com> Message-ID: <2005623214.8EmcOxrdtkDFLmDq@topspin.com> Switch all allocations of coherent memory from pci_alloc_consistent() to dma_alloc_coherent(), so that we can pass GFP_KERNEL. This should help when the system is low on memory. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_cq.c | 25 +++++++++++++------------ linux.git/drivers/infiniband/hw/mthca/mthca_eq.c | 12 ++++++------ linux.git/drivers/infiniband/hw/mthca/mthca_qp.c | 19 ++++++++++--------- 3 files changed, 29 insertions(+), 27 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:04.326180727 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:04.752088550 -0700 @@ -636,19 +636,19 @@ static void mthca_free_cq_buf(struct mth int size; if (cq->is_direct) - pci_free_consistent(dev->pdev, - (cq->ibcq.cqe + 1) * MTHCA_CQ_ENTRY_SIZE, - cq->queue.direct.buf, - pci_unmap_addr(&cq->queue.direct, - mapping)); + dma_free_coherent(&dev->pdev->dev, + (cq->ibcq.cqe + 1) * MTHCA_CQ_ENTRY_SIZE, + cq->queue.direct.buf, + pci_unmap_addr(&cq->queue.direct, + mapping)); else { size = (cq->ibcq.cqe + 1) * MTHCA_CQ_ENTRY_SIZE; for (i = 0; i < (size + PAGE_SIZE - 1) / PAGE_SIZE; ++i) if (cq->queue.page_list[i].buf) - pci_free_consistent(dev->pdev, PAGE_SIZE, - cq->queue.page_list[i].buf, - pci_unmap_addr(&cq->queue.page_list[i], - mapping)); + dma_free_coherent(&dev->pdev->dev, PAGE_SIZE, + cq->queue.page_list[i].buf, + pci_unmap_addr(&cq->queue.page_list[i], + mapping)); kfree(cq->queue.page_list); } @@ -668,8 +668,8 @@ static int mthca_alloc_cq_buf(struct mth npages = 1; shift = get_order(size) + PAGE_SHIFT; - cq->queue.direct.buf = pci_alloc_consistent(dev->pdev, - size, &t); + cq->queue.direct.buf = dma_alloc_coherent(&dev->pdev->dev, + size, &t, GFP_KERNEL); if (!cq->queue.direct.buf) return -ENOMEM; @@ -707,7 +707,8 @@ static int mthca_alloc_cq_buf(struct mth for (i = 0; i < npages; ++i) { cq->queue.page_list[i].buf = - pci_alloc_consistent(dev->pdev, PAGE_SIZE, &t); + dma_alloc_coherent(&dev->pdev->dev, PAGE_SIZE, + &t, GFP_KERNEL); if (!cq->queue.page_list[i].buf) goto err_free; --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_eq.c 2005-06-23 13:03:03.703315530 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_eq.c 2005-06-23 13:03:04.752088550 -0700 @@ -501,8 +501,8 @@ static int __devinit mthca_create_eq(str eq_context = MAILBOX_ALIGN(mailbox); for (i = 0; i < npages; ++i) { - eq->page_list[i].buf = pci_alloc_consistent(dev->pdev, - PAGE_SIZE, &t); + eq->page_list[i].buf = dma_alloc_coherent(&dev->pdev->dev, + PAGE_SIZE, &t, GFP_KERNEL); if (!eq->page_list[i].buf) goto err_out_free; @@ -582,10 +582,10 @@ static int __devinit mthca_create_eq(str err_out_free: for (i = 0; i < npages; ++i) if (eq->page_list[i].buf) - pci_free_consistent(dev->pdev, PAGE_SIZE, - eq->page_list[i].buf, - pci_unmap_addr(&eq->page_list[i], - mapping)); + dma_free_coherent(&dev->pdev->dev, PAGE_SIZE, + eq->page_list[i].buf, + pci_unmap_addr(&eq->page_list[i], + mapping)); kfree(eq->page_list); kfree(dma_list); --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:02.079666927 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:04.751088766 -0700 @@ -934,7 +934,8 @@ static int mthca_alloc_wqe_buf(struct mt mthca_dbg(dev, "Creating direct QP of size %d (shift %d)\n", size, shift); - qp->queue.direct.buf = pci_alloc_consistent(dev->pdev, size, &t); + qp->queue.direct.buf = dma_alloc_coherent(&dev->pdev->dev, size, + &t, GFP_KERNEL); if (!qp->queue.direct.buf) goto err_out; @@ -973,7 +974,8 @@ static int mthca_alloc_wqe_buf(struct mt for (i = 0; i < npages; ++i) { qp->queue.page_list[i].buf = - pci_alloc_consistent(dev->pdev, PAGE_SIZE, &t); + dma_alloc_coherent(&dev->pdev->dev, PAGE_SIZE, + &t, GFP_KERNEL); if (!qp->queue.page_list[i].buf) goto err_out_free; @@ -996,16 +998,15 @@ static int mthca_alloc_wqe_buf(struct mt err_out_free: if (qp->is_direct) { - pci_free_consistent(dev->pdev, size, - qp->queue.direct.buf, - pci_unmap_addr(&qp->queue.direct, mapping)); + dma_free_coherent(&dev->pdev->dev, size, qp->queue.direct.buf, + pci_unmap_addr(&qp->queue.direct, mapping)); } else for (i = 0; i < npages; ++i) { if (qp->queue.page_list[i].buf) - pci_free_consistent(dev->pdev, PAGE_SIZE, - qp->queue.page_list[i].buf, - pci_unmap_addr(&qp->queue.page_list[i], - mapping)); + dma_free_coherent(&dev->pdev->dev, PAGE_SIZE, + qp->queue.page_list[i].buf, + pci_unmap_addr(&qp->queue.page_list[i], + mapping)); } From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 07/14] IB/mthca: Enable unreliable connected transport In-Reply-To: <2005623214.rtUVEh14lfm8dUC3@topspin.com> Message-ID: <2005623214.1YZfsHaBvXkxQMAb@topspin.com> Add support for unreliable connected (UC) transport to mthca driver: - Add attributes for UC to modify QP table. - Add support for posting UC work requests. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_qp.c | 79 ++++++++++++++++++++++++++++++++- 1 files changed, 78 insertions(+), 1 deletion(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:05.636897055 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:06.027812451 -0700 @@ -357,6 +357,9 @@ static const struct { [UD] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_QKEY), + [UC] = (IB_QP_PKEY_INDEX | + IB_QP_PORT | + IB_QP_ACCESS_FLAGS), [RC] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_ACCESS_FLAGS), @@ -378,6 +381,9 @@ static const struct { [UD] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_QKEY), + [UC] = (IB_QP_PKEY_INDEX | + IB_QP_PORT | + IB_QP_ACCESS_FLAGS), [RC] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_ACCESS_FLAGS), @@ -388,6 +394,11 @@ static const struct { [IB_QPS_RTR] = { .trans = MTHCA_TRANS_INIT2RTR, .req_param = { + [UC] = (IB_QP_AV | + IB_QP_PATH_MTU | + IB_QP_DEST_QPN | + IB_QP_RQ_PSN | + IB_QP_MAX_DEST_RD_ATOMIC), [RC] = (IB_QP_AV | IB_QP_PATH_MTU | IB_QP_DEST_QPN | @@ -398,6 +409,9 @@ static const struct { .opt_param = { [UD] = (IB_QP_PKEY_INDEX | IB_QP_QKEY), + [UC] = (IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PKEY_INDEX), [RC] = (IB_QP_ALT_PATH | IB_QP_ACCESS_FLAGS | IB_QP_PKEY_INDEX), @@ -413,6 +427,8 @@ static const struct { .trans = MTHCA_TRANS_RTR2RTS, .req_param = { [UD] = IB_QP_SQ_PSN, + [UC] = (IB_QP_SQ_PSN | + IB_QP_MAX_QP_RD_ATOMIC), [RC] = (IB_QP_TIMEOUT | IB_QP_RETRY_CNT | IB_QP_RNR_RETRY | @@ -423,6 +439,11 @@ static const struct { .opt_param = { [UD] = (IB_QP_CUR_STATE | IB_QP_QKEY), + [UC] = (IB_QP_CUR_STATE | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PKEY_INDEX | + IB_QP_PATH_MIG_STATE), [RC] = (IB_QP_CUR_STATE | IB_QP_ALT_PATH | IB_QP_ACCESS_FLAGS | @@ -442,6 +463,9 @@ static const struct { .opt_param = { [UD] = (IB_QP_CUR_STATE | IB_QP_QKEY), + [UC] = (IB_QP_ACCESS_FLAGS | + IB_QP_ALT_PATH | + IB_QP_PATH_MIG_STATE), [RC] = (IB_QP_ACCESS_FLAGS | IB_QP_ALT_PATH | IB_QP_PATH_MIG_STATE | @@ -462,6 +486,10 @@ static const struct { .opt_param = { [UD] = (IB_QP_CUR_STATE | IB_QP_QKEY), + [UC] = (IB_QP_CUR_STATE | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PATH_MIG_STATE), [RC] = (IB_QP_CUR_STATE | IB_QP_ALT_PATH | IB_QP_ACCESS_FLAGS | @@ -476,6 +504,14 @@ static const struct { .opt_param = { [UD] = (IB_QP_PKEY_INDEX | IB_QP_QKEY), + [UC] = (IB_QP_AV | + IB_QP_MAX_QP_RD_ATOMIC | + IB_QP_MAX_DEST_RD_ATOMIC | + IB_QP_CUR_STATE | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PKEY_INDEX | + IB_QP_PATH_MIG_STATE), [RC] = (IB_QP_AV | IB_QP_TIMEOUT | IB_QP_RETRY_CNT | @@ -501,6 +537,7 @@ static const struct { .opt_param = { [UD] = (IB_QP_CUR_STATE | IB_QP_QKEY), + [UC] = (IB_QP_CUR_STATE), [RC] = (IB_QP_CUR_STATE | IB_QP_MIN_RNR_TIMER), [MLX] = (IB_QP_CUR_STATE | @@ -1530,6 +1567,26 @@ int mthca_tavor_post_send(struct ib_qp * break; + case UC: + switch (wr->opcode) { + case IB_WR_RDMA_WRITE: + case IB_WR_RDMA_WRITE_WITH_IMM: + ((struct mthca_raddr_seg *) wqe)->raddr = + cpu_to_be64(wr->wr.rdma.remote_addr); + ((struct mthca_raddr_seg *) wqe)->rkey = + cpu_to_be32(wr->wr.rdma.rkey); + ((struct mthca_raddr_seg *) wqe)->reserved = 0; + wqe += sizeof (struct mthca_raddr_seg); + size += sizeof (struct mthca_raddr_seg) / 16; + break; + + default: + /* No extra segments required for sends */ + break; + } + + break; + case UD: ((struct mthca_tavor_ud_seg *) wqe)->lkey = cpu_to_be32(to_mah(wr->wr.ud.ah)->key); @@ -1815,9 +1872,29 @@ int mthca_arbel_post_send(struct ib_qp * sizeof (struct mthca_atomic_seg); break; + case IB_WR_RDMA_READ: + case IB_WR_RDMA_WRITE: + case IB_WR_RDMA_WRITE_WITH_IMM: + ((struct mthca_raddr_seg *) wqe)->raddr = + cpu_to_be64(wr->wr.rdma.remote_addr); + ((struct mthca_raddr_seg *) wqe)->rkey = + cpu_to_be32(wr->wr.rdma.rkey); + ((struct mthca_raddr_seg *) wqe)->reserved = 0; + wqe += sizeof (struct mthca_raddr_seg); + size += sizeof (struct mthca_raddr_seg) / 16; + break; + + default: + /* No extra segments required for sends */ + break; + } + + break; + + case UC: + switch (wr->opcode) { case IB_WR_RDMA_WRITE: case IB_WR_RDMA_WRITE_WITH_IMM: - case IB_WR_RDMA_READ: ((struct mthca_raddr_seg *) wqe)->raddr = cpu_to_be64(wr->wr.rdma.remote_addr); ((struct mthca_raddr_seg *) wqe)->rkey = From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 06/14] IB/mthca: Set RDMA/atomic capabilities correctly In-Reply-To: <2005623214.ZulWzVKl5lpV91f5@topspin.com> Message-ID: <2005623214.rtUVEh14lfm8dUC3@topspin.com> mthca apparently had the meanings of the max_rd_atomic and max_dest_rd_atomic QP attributes backwards. max_rd_atomic limits the maximum number of outstanding RDMA/atomic requests as an initiator (on a send queue), and max_dest_rd_atomic specifies the resources allocated to handle RMDA/atomic requests from the remote end of the connection. We were programming our QP context with these values swapped. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_qp.c | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:05.234984039 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:05.636897055 -0700 @@ -724,9 +724,9 @@ int mthca_modify_qp(struct ib_qp *ibqp, qp_param->opt_param_mask |= cpu_to_be32(MTHCA_QP_OPTPAR_RETRY_COUNT); } - if (attr_mask & IB_QP_MAX_DEST_RD_ATOMIC) { - qp_context->params1 |= cpu_to_be32(min(attr->max_dest_rd_atomic ? - ffs(attr->max_dest_rd_atomic) - 1 : 0, + if (attr_mask & IB_QP_MAX_QP_RD_ATOMIC) { + qp_context->params1 |= cpu_to_be32(min(attr->max_rd_atomic ? + ffs(attr->max_rd_atomic) - 1 : 0, 7) << 21); qp_param->opt_param_mask |= cpu_to_be32(MTHCA_QP_OPTPAR_SRA_MAX); } @@ -764,10 +764,10 @@ int mthca_modify_qp(struct ib_qp *ibqp, qp->atomic_rd_en = attr->qp_access_flags; } - if (attr_mask & IB_QP_MAX_QP_RD_ATOMIC) { + if (attr_mask & IB_QP_MAX_DEST_RD_ATOMIC) { u8 rra_max; - if (qp->resp_depth && !attr->max_rd_atomic) { + if (qp->resp_depth && !attr->max_dest_rd_atomic) { /* * Lowering our responder resources to zero. * Turn off RDMA/atomics as responder. @@ -778,7 +778,7 @@ int mthca_modify_qp(struct ib_qp *ibqp, MTHCA_QP_OPTPAR_RAE); } - if (!qp->resp_depth && attr->max_rd_atomic) { + if (!qp->resp_depth && attr->max_dest_rd_atomic) { /* * Increasing our responder resources from * zero. Turn on RDMA/atomics as appropriate. @@ -799,7 +799,7 @@ int mthca_modify_qp(struct ib_qp *ibqp, } for (rra_max = 0; - 1 << rra_max < attr->max_rd_atomic && + 1 << rra_max < attr->max_dest_rd_atomic && rra_max < dev->qp_table.rdb_shift; ++rra_max) ; /* nothing */ @@ -807,7 +807,7 @@ int mthca_modify_qp(struct ib_qp *ibqp, qp_context->params2 |= cpu_to_be32(rra_max << 21); qp_param->opt_param_mask |= cpu_to_be32(MTHCA_QP_OPTPAR_RRA_MAX); - qp->resp_depth = attr->max_rd_atomic; + qp->resp_depth = attr->max_dest_rd_atomic; } qp_context->params2 |= cpu_to_be32(MTHCA_QP_BIT_RSC); From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 08/14] IB/mthca: Fix memset size In-Reply-To: <2005623214.1YZfsHaBvXkxQMAb@topspin.com> Message-ID: <2005623214.G1df9mxuEjBWFXr4@topspin.com> Fix memset to use sizeof *props instead of just sizeof props. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_provider.c | 2 +- 1 files changed, 1 insertion(+), 1 deletion(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-23 13:03:02.629547920 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-23 13:03:06.413728929 -0700 @@ -53,7 +53,7 @@ static int mthca_query_device(struct ib_ if (!in_mad || !out_mad) goto out; - memset(props, 0, sizeof props); + memset(props, 0, sizeof *props); props->fw_ver = mdev->fw_ver; From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 09/14] IB/mthca: Move mthca_is_memfree checks In-Reply-To: <2005623214.G1df9mxuEjBWFXr4@topspin.com> Message-ID: <2005623214.PWhRmdueOgH2vhWW@topspin.com> Make mthca_table_put() and mthca_table_put_range() NOPs if the device is not mem-free, so that we don't have to have "if (mthca_is_memfree())" tests in the callers of these functions. This makes our code more readable and maintainable, and saves a couple dozen bytes of text in ib_mthca.ko as well. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_cq.c | 2 +- linux.git/drivers/infiniband/hw/mthca/mthca_memfree.c | 10 +++++++++- linux.git/drivers/infiniband/hw/mthca/mthca_mr.c | 20 +++++++------------- linux.git/drivers/infiniband/hw/mthca/mthca_qp.c | 9 +++++---- 4 files changed, 22 insertions(+), 19 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:04.752088550 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:06.793646706 -0700 @@ -918,9 +918,9 @@ void mthca_free_cq(struct mthca_dev *dev if (mthca_is_memfree(dev)) { mthca_free_db(dev, MTHCA_DB_TYPE_CQ_ARM, cq->arm_db_index); mthca_free_db(dev, MTHCA_DB_TYPE_CQ_SET_CI, cq->set_ci_db_index); - mthca_table_put(dev, dev->cq_table.table, cq->cqn); } + mthca_table_put(dev, dev->cq_table.table, cq->cqn); mthca_free(&dev->cq_table.alloc, cq->cqn); kfree(mailbox); } --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_memfree.c 2005-06-23 13:03:01.610768408 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_memfree.c 2005-06-23 13:03:06.791647139 -0700 @@ -179,9 +179,14 @@ out: void mthca_table_put(struct mthca_dev *dev, struct mthca_icm_table *table, int obj) { - int i = (obj & (table->num_obj - 1)) * table->obj_size / MTHCA_TABLE_CHUNK_SIZE; + int i; u8 status; + if (!mthca_is_memfree(dev)) + return; + + i = (obj & (table->num_obj - 1)) * table->obj_size / MTHCA_TABLE_CHUNK_SIZE; + down(&table->mutex); if (--table->icm[i]->refcount == 0) { @@ -256,6 +261,9 @@ void mthca_table_put_range(struct mthca_ { int i; + if (!mthca_is_memfree(dev)) + return; + for (i = start; i <= end; i += MTHCA_TABLE_CHUNK_SIZE / table->obj_size) mthca_table_put(dev, table, i); } --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_mr.c 2005-06-23 13:03:01.610768408 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_mr.c 2005-06-23 13:03:06.792646922 -0700 @@ -195,10 +195,8 @@ static void mthca_free_mtt(struct mthca_ struct mthca_buddy* buddy) { mthca_buddy_free(buddy, seg, order); - - if (mthca_is_memfree(dev)) - mthca_table_put_range(dev, dev->mr_table.mtt_table, seg, - seg + (1 << order) - 1); + mthca_table_put_range(dev, dev->mr_table.mtt_table, seg, + seg + (1 << order) - 1); } static inline u32 tavor_hw_index_to_key(u32 ind) @@ -299,8 +297,7 @@ int mthca_mr_alloc_notrans(struct mthca_ return err; err_out_table: - if (mthca_is_memfree(dev)) - mthca_table_put(dev, dev->mr_table.mpt_table, key); + mthca_table_put(dev, dev->mr_table.mpt_table, key); err_out_mpt_free: mthca_free(&dev->mr_table.mpt_alloc, key); @@ -437,8 +434,7 @@ err_out_free_mtt: mthca_free_mtt(dev, mr->first_seg, mr->order, &dev->mr_table.mtt_buddy); err_out_table: - if (mthca_is_memfree(dev)) - mthca_table_put(dev, dev->mr_table.mpt_table, key); + mthca_table_put(dev, dev->mr_table.mpt_table, key); err_out_mpt_free: mthca_free(&dev->mr_table.mpt_alloc, key); @@ -452,9 +448,8 @@ static void mthca_free_region(struct mth if (order >= 0) mthca_free_mtt(dev, first_seg, order, buddy); - if (mthca_is_memfree(dev)) - mthca_table_put(dev, dev->mr_table.mpt_table, - arbel_key_to_hw_index(lkey)); + mthca_table_put(dev, dev->mr_table.mpt_table, + arbel_key_to_hw_index(lkey)); mthca_free(&dev->mr_table.mpt_alloc, key_to_hw_index(dev, lkey)); } @@ -596,8 +591,7 @@ err_out_free_mtt: dev->mr_table.fmr_mtt_buddy); err_out_table: - if (mthca_is_memfree(dev)) - mthca_table_put(dev, dev->mr_table.mpt_table, key); + mthca_table_put(dev, dev->mr_table.mpt_table, key); err_out_mpt_free: mthca_free(&dev->mr_table.mpt_alloc, mr->ibmr.lkey); --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:06.027812451 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:06.792646922 -0700 @@ -1111,11 +1111,12 @@ static void mthca_free_memfree(struct mt if (mthca_is_memfree(dev)) { mthca_free_db(dev, MTHCA_DB_TYPE_SQ, qp->sq.db_index); mthca_free_db(dev, MTHCA_DB_TYPE_RQ, qp->rq.db_index); - mthca_table_put(dev, dev->qp_table.rdb_table, - qp->qpn << dev->qp_table.rdb_shift); - mthca_table_put(dev, dev->qp_table.eqp_table, qp->qpn); - mthca_table_put(dev, dev->qp_table.qp_table, qp->qpn); } + + mthca_table_put(dev, dev->qp_table.rdb_table, + qp->qpn << dev->qp_table.rdb_shift); + mthca_table_put(dev, dev->qp_table.eqp_table, qp->qpn); + mthca_table_put(dev, dev->qp_table.qp_table, qp->qpn); } static void mthca_wq_init(struct mthca_wq* wq) From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 10/14] IB/mthca: Split off MTT allocation In-Reply-To: <2005623214.PWhRmdueOgH2vhWW@topspin.com> Message-ID: <2005623214.fUJVvXKwjz2o8abB@topspin.com> Split allocation of MTT range from creation of MR. This will be useful for implementing shared memory regions and userspace verbs. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_dev.h | 6 linux.git/drivers/infiniband/hw/mthca/mthca_mr.c | 325 +++++++++++++-------------- linux.git/drivers/infiniband/hw/mthca/mthca_provider.h | 14 - 3 files changed, 177 insertions(+), 168 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-23 13:03:02.630547703 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-23 13:03:07.308535271 -0700 @@ -380,6 +380,12 @@ void mthca_uar_free(struct mthca_dev *de int mthca_pd_alloc(struct mthca_dev *dev, struct mthca_pd *pd); void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd); +struct mthca_mtt *mthca_alloc_mtt(struct mthca_dev *dev, int size); +void mthca_free_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt); +int mthca_write_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt, + int start_index, u64 *buffer_list, int list_len); +int mthca_mr_alloc(struct mthca_dev *dev, u32 pd, int buffer_size_shift, + u64 iova, u64 total_size, u32 access, struct mthca_mr *mr); int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, u32 access, struct mthca_mr *mr); int mthca_mr_alloc_phys(struct mthca_dev *dev, u32 pd, --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_mr.c 2005-06-23 13:03:06.792646922 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_mr.c 2005-06-23 13:03:07.308535271 -0700 @@ -40,6 +40,12 @@ #include "mthca_cmd.h" #include "mthca_memfree.h" +struct mthca_mtt { + struct mthca_buddy *buddy; + int order; + u32 first_seg; +}; + /* * Must be packed because mtt_seg is 64 bits but only aligned to 32 bits. */ @@ -173,8 +179,8 @@ static void __devexit mthca_buddy_cleanu kfree(buddy->bits); } -static u32 mthca_alloc_mtt(struct mthca_dev *dev, int order, - struct mthca_buddy *buddy) +static u32 mthca_alloc_mtt_range(struct mthca_dev *dev, int order, + struct mthca_buddy *buddy) { u32 seg = mthca_buddy_alloc(buddy, order); @@ -191,12 +197,100 @@ static u32 mthca_alloc_mtt(struct mthca_ return seg; } -static void mthca_free_mtt(struct mthca_dev *dev, u32 seg, int order, - struct mthca_buddy* buddy) +static struct mthca_mtt *__mthca_alloc_mtt(struct mthca_dev *dev, int size, + struct mthca_buddy *buddy) +{ + struct mthca_mtt *mtt; + int i; + + if (size <= 0) + return ERR_PTR(-EINVAL); + + mtt = kmalloc(sizeof *mtt, GFP_KERNEL); + if (!mtt) + return ERR_PTR(-ENOMEM); + + mtt->buddy = buddy; + mtt->order = 0; + for (i = MTHCA_MTT_SEG_SIZE / 8; i < size; i <<= 1) + ++mtt->order; + + mtt->first_seg = mthca_alloc_mtt_range(dev, mtt->order, buddy); + if (mtt->first_seg == -1) { + kfree(mtt); + return ERR_PTR(-ENOMEM); + } + + return mtt; +} + +struct mthca_mtt *mthca_alloc_mtt(struct mthca_dev *dev, int size) +{ + return __mthca_alloc_mtt(dev, size, &dev->mr_table.mtt_buddy); +} + +void mthca_free_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt) +{ + if (!mtt) + return; + + mthca_buddy_free(mtt->buddy, mtt->first_seg, mtt->order); + + mthca_table_put_range(dev, dev->mr_table.mtt_table, + mtt->first_seg, + mtt->first_seg + (1 << mtt->order) - 1); + + kfree(mtt); +} + +int mthca_write_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt, + int start_index, u64 *buffer_list, int list_len) { - mthca_buddy_free(buddy, seg, order); - mthca_table_put_range(dev, dev->mr_table.mtt_table, seg, - seg + (1 << order) - 1); + u64 *mtt_entry; + int err = 0; + u8 status; + int i; + + mtt_entry = (u64 *) __get_free_page(GFP_KERNEL); + if (!mtt_entry) + return -ENOMEM; + + while (list_len > 0) { + mtt_entry[0] = cpu_to_be64(dev->mr_table.mtt_base + + mtt->first_seg * MTHCA_MTT_SEG_SIZE + + start_index * 8); + mtt_entry[1] = 0; + for (i = 0; i < list_len && i < PAGE_SIZE / 8 - 2; ++i) + mtt_entry[i + 2] = cpu_to_be64(buffer_list[i] | + MTHCA_MTT_FLAG_PRESENT); + + /* + * If we have an odd number of entries to write, add + * one more dummy entry for firmware efficiency. + */ + if (i & 1) + mtt_entry[i + 2] = 0; + + err = mthca_WRITE_MTT(dev, mtt_entry, (i + 1) & ~1, &status); + if (err) { + mthca_warn(dev, "WRITE_MTT failed (%d)\n", err); + goto out; + } + if (status) { + mthca_warn(dev, "WRITE_MTT returned status 0x%02x\n", + status); + err = -EINVAL; + goto out; + } + + list_len -= i; + start_index += i; + buffer_list += i; + } + +out: + free_page((unsigned long) mtt_entry); + return err; } static inline u32 tavor_hw_index_to_key(u32 ind) @@ -235,18 +329,20 @@ static inline u32 key_to_hw_index(struct return tavor_key_to_hw_index(key); } -int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, - u32 access, struct mthca_mr *mr) +int mthca_mr_alloc(struct mthca_dev *dev, u32 pd, int buffer_size_shift, + u64 iova, u64 total_size, u32 access, struct mthca_mr *mr) { - void *mailbox = NULL; + void *mailbox; struct mthca_mpt_entry *mpt_entry; u32 key; + int i; int err; u8 status; might_sleep(); - mr->order = -1; + WARN_ON(buffer_size_shift >= 32); + key = mthca_alloc(&dev->mr_table.mpt_alloc); if (key == -1) return -ENOMEM; @@ -268,186 +364,98 @@ int mthca_mr_alloc_notrans(struct mthca_ mpt_entry->flags = cpu_to_be32(MTHCA_MPT_FLAG_SW_OWNS | MTHCA_MPT_FLAG_MIO | - MTHCA_MPT_FLAG_PHYSICAL | MTHCA_MPT_FLAG_REGION | access); - mpt_entry->page_size = 0; + if (!mr->mtt) + mpt_entry->flags |= cpu_to_be32(MTHCA_MPT_FLAG_PHYSICAL); + + mpt_entry->page_size = cpu_to_be32(buffer_size_shift - 12); mpt_entry->key = cpu_to_be32(key); mpt_entry->pd = cpu_to_be32(pd); - mpt_entry->start = 0; - mpt_entry->length = ~0ULL; + mpt_entry->start = cpu_to_be64(iova); + mpt_entry->length = cpu_to_be64(total_size); memset(&mpt_entry->lkey, 0, sizeof *mpt_entry - offsetof(struct mthca_mpt_entry, lkey)); + if (mr->mtt) + mpt_entry->mtt_seg = + cpu_to_be64(dev->mr_table.mtt_base + + mr->mtt->first_seg * MTHCA_MTT_SEG_SIZE); + + if (0) { + mthca_dbg(dev, "Dumping MPT entry %08x:\n", mr->ibmr.lkey); + for (i = 0; i < sizeof (struct mthca_mpt_entry) / 4; ++i) { + if (i % 4 == 0) + printk("[%02x] ", i * 4); + printk(" %08x", be32_to_cpu(((u32 *) mpt_entry)[i])); + if ((i + 1) % 4 == 0) + printk("\n"); + } + } + err = mthca_SW2HW_MPT(dev, mpt_entry, key & (dev->limits.num_mpts - 1), &status); if (err) { mthca_warn(dev, "SW2HW_MPT failed (%d)\n", err); - goto err_out_table; + goto err_out_mailbox; } else if (status) { mthca_warn(dev, "SW2HW_MPT returned status 0x%02x\n", status); err = -EINVAL; - goto err_out_table; + goto err_out_mailbox; } kfree(mailbox); return err; +err_out_mailbox: + kfree(mailbox); + err_out_table: mthca_table_put(dev, dev->mr_table.mpt_table, key); err_out_mpt_free: mthca_free(&dev->mr_table.mpt_alloc, key); - kfree(mailbox); return err; } +int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, + u32 access, struct mthca_mr *mr) +{ + mr->mtt = NULL; + return mthca_mr_alloc(dev, pd, 12, 0, ~0ULL, access, mr); +} + int mthca_mr_alloc_phys(struct mthca_dev *dev, u32 pd, u64 *buffer_list, int buffer_size_shift, int list_len, u64 iova, u64 total_size, u32 access, struct mthca_mr *mr) { - void *mailbox; - u64 *mtt_entry; - struct mthca_mpt_entry *mpt_entry; - u32 key; - int err = -ENOMEM; - u8 status; - int i; - - might_sleep(); - WARN_ON(buffer_size_shift >= 32); - - key = mthca_alloc(&dev->mr_table.mpt_alloc); - if (key == -1) - return -ENOMEM; - mr->ibmr.rkey = mr->ibmr.lkey = hw_index_to_key(dev, key); - - if (mthca_is_memfree(dev)) { - err = mthca_table_get(dev, dev->mr_table.mpt_table, key); - if (err) - goto err_out_mpt_free; - } - - for (i = MTHCA_MTT_SEG_SIZE / 8, mr->order = 0; - i < list_len; - i <<= 1, ++mr->order) - ; /* nothing */ - - mr->first_seg = mthca_alloc_mtt(dev, mr->order, - &dev->mr_table.mtt_buddy); - if (mr->first_seg == -1) - goto err_out_table; - - /* - * If list_len is odd, we add one more dummy entry for - * firmware efficiency. - */ - mailbox = kmalloc(max(sizeof *mpt_entry, - (size_t) 8 * (list_len + (list_len & 1) + 2)) + - MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) - goto err_out_free_mtt; - - mtt_entry = MAILBOX_ALIGN(mailbox); + int err; - mtt_entry[0] = cpu_to_be64(dev->mr_table.mtt_base + - mr->first_seg * MTHCA_MTT_SEG_SIZE); - mtt_entry[1] = 0; - for (i = 0; i < list_len; ++i) - mtt_entry[i + 2] = cpu_to_be64(buffer_list[i] | - MTHCA_MTT_FLAG_PRESENT); - if (list_len & 1) { - mtt_entry[i + 2] = 0; - ++list_len; - } + mr->mtt = mthca_alloc_mtt(dev, list_len); + if (IS_ERR(mr->mtt)) + return PTR_ERR(mr->mtt); - if (0) { - mthca_dbg(dev, "Dumping MPT entry\n"); - for (i = 0; i < list_len + 2; ++i) - printk(KERN_ERR "[%2d] %016llx\n", - i, (unsigned long long) be64_to_cpu(mtt_entry[i])); - } - - err = mthca_WRITE_MTT(dev, mtt_entry, list_len, &status); + err = mthca_write_mtt(dev, mr->mtt, 0, buffer_list, list_len); if (err) { - mthca_warn(dev, "WRITE_MTT failed (%d)\n", err); - goto err_out_mailbox_free; - } - if (status) { - mthca_warn(dev, "WRITE_MTT returned status 0x%02x\n", - status); - err = -EINVAL; - goto err_out_mailbox_free; - } - - mpt_entry = MAILBOX_ALIGN(mailbox); - - mpt_entry->flags = cpu_to_be32(MTHCA_MPT_FLAG_SW_OWNS | - MTHCA_MPT_FLAG_MIO | - MTHCA_MPT_FLAG_REGION | - access); - - mpt_entry->page_size = cpu_to_be32(buffer_size_shift - 12); - mpt_entry->key = cpu_to_be32(key); - mpt_entry->pd = cpu_to_be32(pd); - mpt_entry->start = cpu_to_be64(iova); - mpt_entry->length = cpu_to_be64(total_size); - memset(&mpt_entry->lkey, 0, - sizeof *mpt_entry - offsetof(struct mthca_mpt_entry, lkey)); - mpt_entry->mtt_seg = cpu_to_be64(dev->mr_table.mtt_base + - mr->first_seg * MTHCA_MTT_SEG_SIZE); - - if (0) { - mthca_dbg(dev, "Dumping MPT entry %08x:\n", mr->ibmr.lkey); - for (i = 0; i < sizeof (struct mthca_mpt_entry) / 4; ++i) { - if (i % 4 == 0) - printk("[%02x] ", i * 4); - printk(" %08x", be32_to_cpu(((u32 *) mpt_entry)[i])); - if ((i + 1) % 4 == 0) - printk("\n"); - } + mthca_free_mtt(dev, mr->mtt); + return err; } - err = mthca_SW2HW_MPT(dev, mpt_entry, - key & (dev->limits.num_mpts - 1), - &status); + err = mthca_mr_alloc(dev, pd, buffer_size_shift, iova, + total_size, access, mr); if (err) - mthca_warn(dev, "SW2HW_MPT failed (%d)\n", err); - else if (status) { - mthca_warn(dev, "SW2HW_MPT returned status 0x%02x\n", - status); - err = -EINVAL; - } + mthca_free_mtt(dev, mr->mtt); - kfree(mailbox); - return err; - -err_out_mailbox_free: - kfree(mailbox); - -err_out_free_mtt: - mthca_free_mtt(dev, mr->first_seg, mr->order, &dev->mr_table.mtt_buddy); - -err_out_table: - mthca_table_put(dev, dev->mr_table.mpt_table, key); - -err_out_mpt_free: - mthca_free(&dev->mr_table.mpt_alloc, key); return err; } /* Free mr or fmr */ -static void mthca_free_region(struct mthca_dev *dev, u32 lkey, int order, - u32 first_seg, struct mthca_buddy *buddy) +static void mthca_free_region(struct mthca_dev *dev, u32 lkey) { - if (order >= 0) - mthca_free_mtt(dev, first_seg, order, buddy); - mthca_table_put(dev, dev->mr_table.mpt_table, arbel_key_to_hw_index(lkey)); @@ -471,8 +479,8 @@ void mthca_free_mr(struct mthca_dev *dev mthca_warn(dev, "HW2SW_MPT returned status 0x%02x\n", status); - mthca_free_region(dev, mr->ibmr.lkey, mr->order, mr->first_seg, - &dev->mr_table.mtt_buddy); + mthca_free_region(dev, mr->ibmr.lkey); + mthca_free_mtt(dev, mr->mtt); } int mthca_fmr_alloc(struct mthca_dev *dev, u32 pd, @@ -517,21 +525,15 @@ int mthca_fmr_alloc(struct mthca_dev *de mr->mem.tavor.mpt = dev->mr_table.tavor_fmr.mpt_base + sizeof *(mr->mem.tavor.mpt) * idx; - for (i = MTHCA_MTT_SEG_SIZE / 8, mr->order = 0; - i < list_len; - i <<= 1, ++mr->order) - ; /* nothing */ - - mr->first_seg = mthca_alloc_mtt(dev, mr->order, - dev->mr_table.fmr_mtt_buddy); - if (mr->first_seg == -1) + mr->mtt = __mthca_alloc_mtt(dev, list_len, dev->mr_table.fmr_mtt_buddy); + if (IS_ERR(mr->mtt)) goto err_out_table; - mtt_seg = mr->first_seg * MTHCA_MTT_SEG_SIZE; + mtt_seg = mr->mtt->first_seg * MTHCA_MTT_SEG_SIZE; if (mthca_is_memfree(dev)) { mr->mem.arbel.mtts = mthca_table_find(dev->mr_table.mtt_table, - mr->first_seg); + mr->mtt->first_seg); BUG_ON(!mr->mem.arbel.mtts); } else mr->mem.tavor.mtts = dev->mr_table.tavor_fmr.mtt_base + mtt_seg; @@ -587,8 +589,7 @@ err_out_mailbox_free: kfree(mailbox); err_out_free_mtt: - mthca_free_mtt(dev, mr->first_seg, mr->order, - dev->mr_table.fmr_mtt_buddy); + mthca_free_mtt(dev, mr->mtt); err_out_table: mthca_table_put(dev, dev->mr_table.mpt_table, key); @@ -603,8 +604,9 @@ int mthca_free_fmr(struct mthca_dev *dev if (fmr->maps) return -EBUSY; - mthca_free_region(dev, fmr->ibmr.lkey, fmr->order, fmr->first_seg, - dev->mr_table.fmr_mtt_buddy); + mthca_free_region(dev, fmr->ibmr.lkey); + mthca_free_mtt(dev, fmr->mtt); + return 0; } @@ -820,7 +822,8 @@ int __devinit mthca_init_mr_table(struct if (dev->limits.reserved_mtts) { i = fls(dev->limits.reserved_mtts - 1); - if (mthca_alloc_mtt(dev, i, dev->mr_table.fmr_mtt_buddy) == -1) { + if (mthca_alloc_mtt_range(dev, i, + dev->mr_table.fmr_mtt_buddy) == -1) { mthca_warn(dev, "MTT table of order %d is too small.\n", dev->mr_table.fmr_mtt_buddy->max_order); err = -ENOMEM; --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_provider.h 2005-06-23 13:03:01.490794374 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_provider.h 2005-06-23 13:03:07.308535271 -0700 @@ -54,18 +54,18 @@ struct mthca_uar { int index; }; +struct mthca_mtt; + struct mthca_mr { - struct ib_mr ibmr; - int order; - u32 first_seg; + struct ib_mr ibmr; + struct mthca_mtt *mtt; }; struct mthca_fmr { - struct ib_fmr ibmr; + struct ib_fmr ibmr; struct ib_fmr_attr attr; - int order; - u32 first_seg; - int maps; + struct mthca_mtt *mtt; + int maps; union { struct { struct mthca_mpt_entry __iomem *mpt; From roland at topspin.com Thu Jun 23 21:04:20 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:20 -0700 Subject: [openib-general] [PATCH 11/14] IB/mthca: Fix memory leak on error path In-Reply-To: <2005623214.fUJVvXKwjz2o8abB@topspin.com> Message-ID: <2005623214.gjur4MFr788QnYf6@topspin.com> Free page_list buffer on error path of mthca_reg_phys_mr(). Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_provider.c | 1 + 1 files changed, 1 insertion(+) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-23 13:03:06.413728929 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-23 13:03:07.987388350 -0700 @@ -559,6 +559,7 @@ static struct ib_mr *mthca_reg_phys_mr(s convert_access(acc), mr); if (err) { + kfree(page_list); kfree(mr); return ERR_PTR(err); } From roland at topspin.com Thu Jun 23 21:04:21 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:21 -0700 Subject: [openib-general] [PATCH 12/14] IB/mthca: Encapsulate command interface init In-Reply-To: <2005623214.gjur4MFr788QnYf6@topspin.com> Message-ID: <2005623214.Wl3n9twXcA60cNkd@topspin.com> Encapsulate mthca command interface initialization/cleanup. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_cmd.c | 21 +++++++++++++++++++++ linux.git/drivers/infiniband/hw/mthca/mthca_cmd.h | 2 ++ linux.git/drivers/infiniband/hw/mthca/mthca_main.c | 23 +++++++---------------- 3 files changed, 30 insertions(+), 16 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cmd.c 2005-06-23 13:03:01.180861451 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cmd.c 2005-06-23 13:03:08.794213733 -0700 @@ -431,6 +431,27 @@ static int mthca_cmd_imm(struct mthca_de timeout, status); } +int mthca_cmd_init(struct mthca_dev *dev) +{ + sema_init(&dev->cmd.hcr_sem, 1); + sema_init(&dev->cmd.poll_sem, 1); + dev->cmd.use_events = 0; + + dev->hcr = ioremap(pci_resource_start(dev->pdev, 0) + MTHCA_HCR_BASE, + MTHCA_HCR_SIZE); + if (!dev->hcr) { + mthca_err(dev, "Couldn't map command register."); + return -ENOMEM; + } + + return 0; +} + +void mthca_cmd_cleanup(struct mthca_dev *dev) +{ + iounmap(dev->hcr); +} + /* * Switch to using events to issue FW commands (should be called after * event queue to command events has been initialized). --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cmd.h 2005-06-23 13:03:01.180861451 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cmd.h 2005-06-23 13:03:08.795213517 -0700 @@ -235,6 +235,8 @@ struct mthca_set_ib_param { u32 cap_mask; }; +int mthca_cmd_init(struct mthca_dev *dev); +void mthca_cmd_cleanup(struct mthca_dev *dev); int mthca_cmd_use_events(struct mthca_dev *dev); void mthca_cmd_use_polling(struct mthca_dev *dev); void mthca_cmd_event(struct mthca_dev *dev, u16 token, --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_main.c 2005-06-23 13:03:03.703315530 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_main.c 2005-06-23 13:03:08.795213517 -0700 @@ -1005,25 +1005,18 @@ static int __devinit mthca_init_one(stru !pci_enable_msi(pdev)) mdev->mthca_flags |= MTHCA_FLAG_MSI; - sema_init(&mdev->cmd.hcr_sem, 1); - sema_init(&mdev->cmd.poll_sem, 1); - mdev->cmd.use_events = 0; - - mdev->hcr = ioremap(pci_resource_start(pdev, 0) + MTHCA_HCR_BASE, MTHCA_HCR_SIZE); - if (!mdev->hcr) { - mthca_err(mdev, "Couldn't map command register, " - "aborting.\n"); - err = -ENOMEM; + if (mthca_cmd_init(mdev)) { + mthca_err(mdev, "Failed to init command interface, aborting.\n"); goto err_free_dev; } err = mthca_tune_pci(mdev); if (err) - goto err_iounmap; + goto err_cmd; err = mthca_init_hca(mdev); if (err) - goto err_iounmap; + goto err_cmd; if (mdev->fw_ver < mthca_hca_table[id->driver_data].latest_fw) { mthca_warn(mdev, "HCA FW version %x.%x.%x is old (%x.%x.%x is current).\n", @@ -1071,8 +1064,8 @@ err_cleanup: err_close: mthca_close_hca(mdev); -err_iounmap: - iounmap(mdev->hcr); +err_cmd: + mthca_cmd_cleanup(mdev); err_free_dev: if (mdev->mthca_flags & MTHCA_FLAG_MSI_X) @@ -1119,10 +1112,8 @@ static void __devexit mthca_remove_one(s iounmap(mdev->kar); mthca_uar_free(mdev, &mdev->driver_uar); mthca_cleanup_uar_table(mdev); - mthca_close_hca(mdev); - - iounmap(mdev->hcr); + mthca_cmd_cleanup(mdev); if (mdev->mthca_flags & MTHCA_FLAG_MSI_X) pci_disable_msix(pdev); From roland at topspin.com Thu Jun 23 21:04:21 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:21 -0700 Subject: [openib-general] [PATCH 13/14] IB/mthca: Align FW command mailboxes to 4K In-Reply-To: <2005623214.Wl3n9twXcA60cNkd@topspin.com> Message-ID: <2005623214.1yVGvzp1yi0UQ3bC@topspin.com> Future versions of Mellanox HCA firmware will require command mailboxes to be aligned to 4K. Support this by using a pci_pool to allocate all mailboxes. This has the added benefit of shrinking the source and text of mthca. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_cmd.c | 512 ++++++++++++-------------------- linux.git/drivers/infiniband/hw/mthca/mthca_cmd.h | 46 +- linux.git/drivers/infiniband/hw/mthca/mthca_cq.c | 34 +- linux.git/drivers/infiniband/hw/mthca/mthca_dev.h | 1 linux.git/drivers/infiniband/hw/mthca/mthca_eq.c | 37 +- linux.git/drivers/infiniband/hw/mthca/mthca_mcg.c | 63 ++- linux.git/drivers/infiniband/hw/mthca/mthca_mr.c | 46 +- linux.git/drivers/infiniband/hw/mthca/mthca_qp.c | 14 8 files changed, 330 insertions(+), 423 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cmd.c 2005-06-23 13:03:08.794213733 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cmd.c 2005-06-23 13:03:09.282108141 -0700 @@ -444,11 +444,20 @@ int mthca_cmd_init(struct mthca_dev *dev return -ENOMEM; } + dev->cmd.pool = pci_pool_create("mthca_cmd", dev->pdev, + MTHCA_MAILBOX_SIZE, + MTHCA_MAILBOX_SIZE, 0); + if (!dev->cmd.pool) { + iounmap(dev->hcr); + return -ENOMEM; + } + return 0; } void mthca_cmd_cleanup(struct mthca_dev *dev) { + pci_pool_destroy(dev->cmd.pool); iounmap(dev->hcr); } @@ -510,6 +519,33 @@ void mthca_cmd_use_polling(struct mthca_ up(&dev->cmd.poll_sem); } +struct mthca_mailbox *mthca_alloc_mailbox(struct mthca_dev *dev, + unsigned int gfp_mask) +{ + struct mthca_mailbox *mailbox; + + mailbox = kmalloc(sizeof *mailbox, gfp_mask); + if (!mailbox) + return ERR_PTR(-ENOMEM); + + mailbox->buf = pci_pool_alloc(dev->cmd.pool, gfp_mask, &mailbox->dma); + if (!mailbox->buf) { + kfree(mailbox); + return ERR_PTR(-ENOMEM); + } + + return mailbox; +} + +void mthca_free_mailbox(struct mthca_dev *dev, struct mthca_mailbox *mailbox) +{ + if (!mailbox) + return; + + pci_pool_free(dev->cmd.pool, mailbox->buf, mailbox->dma); + kfree(mailbox); +} + int mthca_SYS_EN(struct mthca_dev *dev, u8 *status) { u64 out; @@ -534,20 +570,20 @@ int mthca_SYS_DIS(struct mthca_dev *dev, static int mthca_map_cmd(struct mthca_dev *dev, u16 op, struct mthca_icm *icm, u64 virt, u8 *status) { - u32 *inbox; - dma_addr_t indma; + struct mthca_mailbox *mailbox; struct mthca_icm_iter iter; + __be64 *pages; int lg; int nent = 0; int i; int err = 0; int ts = 0, tc = 0; - inbox = pci_alloc_consistent(dev->pdev, PAGE_SIZE, &indma); - if (!inbox) - return -ENOMEM; - - memset(inbox, 0, PAGE_SIZE); + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + memset(mailbox->buf, 0, MTHCA_MAILBOX_SIZE); + pages = mailbox->buf; for (mthca_icm_first(icm, &iter); !mthca_icm_last(&iter); @@ -567,19 +603,17 @@ static int mthca_map_cmd(struct mthca_de } for (i = 0; i < mthca_icm_size(&iter) / (1 << lg); ++i, ++nent) { if (virt != -1) { - *((__be64 *) (inbox + nent * 4)) = - cpu_to_be64(virt); + pages[nent * 2] = cpu_to_be64(virt); virt += 1 << lg; } - - *((__be64 *) (inbox + nent * 4 + 2)) = - cpu_to_be64((mthca_icm_addr(&iter) + - (i << lg)) | (lg - 12)); + + pages[nent * 2 + 1] = cpu_to_be64((mthca_icm_addr(&iter) + + (i << lg)) | (lg - 12)); ts += 1 << (lg - 10); ++tc; - if (nent == PAGE_SIZE / 16) { - err = mthca_cmd(dev, indma, nent, 0, op, + if (nent == MTHCA_MAILBOX_SIZE / 16) { + err = mthca_cmd(dev, mailbox->dma, nent, 0, op, CMD_TIME_CLASS_B, status); if (err || *status) goto out; @@ -589,7 +623,7 @@ static int mthca_map_cmd(struct mthca_de } if (nent) - err = mthca_cmd(dev, indma, nent, 0, op, + err = mthca_cmd(dev, mailbox->dma, nent, 0, op, CMD_TIME_CLASS_B, status); switch (op) { @@ -606,7 +640,7 @@ static int mthca_map_cmd(struct mthca_de } out: - pci_free_consistent(dev->pdev, PAGE_SIZE, inbox, indma); + mthca_free_mailbox(dev, mailbox); return err; } @@ -627,8 +661,8 @@ int mthca_RUN_FW(struct mthca_dev *dev, int mthca_QUERY_FW(struct mthca_dev *dev, u8 *status) { + struct mthca_mailbox *mailbox; u32 *outbox; - dma_addr_t outdma; int err = 0; u8 lg; @@ -646,12 +680,12 @@ int mthca_QUERY_FW(struct mthca_dev *dev #define QUERY_FW_EQ_ARM_BASE_OFFSET 0x40 #define QUERY_FW_EQ_SET_CI_BASE_OFFSET 0x48 - outbox = pci_alloc_consistent(dev->pdev, QUERY_FW_OUT_SIZE, &outdma); - if (!outbox) { - return -ENOMEM; - } + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + outbox = mailbox->buf; - err = mthca_cmd_box(dev, 0, outdma, 0, 0, CMD_QUERY_FW, + err = mthca_cmd_box(dev, 0, mailbox->dma, 0, 0, CMD_QUERY_FW, CMD_TIME_CLASS_A, status); if (err) @@ -702,15 +736,15 @@ int mthca_QUERY_FW(struct mthca_dev *dev } out: - pci_free_consistent(dev->pdev, QUERY_FW_OUT_SIZE, outbox, outdma); + mthca_free_mailbox(dev, mailbox); return err; } int mthca_ENABLE_LAM(struct mthca_dev *dev, u8 *status) { + struct mthca_mailbox *mailbox; u8 info; u32 *outbox; - dma_addr_t outdma; int err = 0; #define ENABLE_LAM_OUT_SIZE 0x100 @@ -721,11 +755,12 @@ int mthca_ENABLE_LAM(struct mthca_dev *d #define ENABLE_LAM_INFO_HIDDEN_FLAG (1 << 4) #define ENABLE_LAM_INFO_ECC_MASK 0x3 - outbox = pci_alloc_consistent(dev->pdev, ENABLE_LAM_OUT_SIZE, &outdma); - if (!outbox) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + outbox = mailbox->buf; - err = mthca_cmd_box(dev, 0, outdma, 0, 0, CMD_ENABLE_LAM, + err = mthca_cmd_box(dev, 0, mailbox->dma, 0, 0, CMD_ENABLE_LAM, CMD_TIME_CLASS_C, status); if (err) @@ -754,7 +789,7 @@ int mthca_ENABLE_LAM(struct mthca_dev *d (unsigned long long) dev->ddr_end); out: - pci_free_consistent(dev->pdev, ENABLE_LAM_OUT_SIZE, outbox, outdma); + mthca_free_mailbox(dev, mailbox); return err; } @@ -765,9 +800,9 @@ int mthca_DISABLE_LAM(struct mthca_dev * int mthca_QUERY_DDR(struct mthca_dev *dev, u8 *status) { + struct mthca_mailbox *mailbox; u8 info; u32 *outbox; - dma_addr_t outdma; int err = 0; #define QUERY_DDR_OUT_SIZE 0x100 @@ -778,11 +813,12 @@ int mthca_QUERY_DDR(struct mthca_dev *de #define QUERY_DDR_INFO_HIDDEN_FLAG (1 << 4) #define QUERY_DDR_INFO_ECC_MASK 0x3 - outbox = pci_alloc_consistent(dev->pdev, QUERY_DDR_OUT_SIZE, &outdma); - if (!outbox) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + outbox = mailbox->buf; - err = mthca_cmd_box(dev, 0, outdma, 0, 0, CMD_QUERY_DDR, + err = mthca_cmd_box(dev, 0, mailbox->dma, 0, 0, CMD_QUERY_DDR, CMD_TIME_CLASS_A, status); if (err) @@ -808,15 +844,15 @@ int mthca_QUERY_DDR(struct mthca_dev *de (unsigned long long) dev->ddr_end); out: - pci_free_consistent(dev->pdev, QUERY_DDR_OUT_SIZE, outbox, outdma); + mthca_free_mailbox(dev, mailbox); return err; } int mthca_QUERY_DEV_LIM(struct mthca_dev *dev, struct mthca_dev_lim *dev_lim, u8 *status) { + struct mthca_mailbox *mailbox; u32 *outbox; - dma_addr_t outdma; u8 field; u16 size; int err; @@ -881,11 +917,12 @@ int mthca_QUERY_DEV_LIM(struct mthca_dev #define QUERY_DEV_LIM_LAMR_OFFSET 0x9f #define QUERY_DEV_LIM_MAX_ICM_SZ_OFFSET 0xa0 - outbox = pci_alloc_consistent(dev->pdev, QUERY_DEV_LIM_OUT_SIZE, &outdma); - if (!outbox) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + outbox = mailbox->buf; - err = mthca_cmd_box(dev, 0, outdma, 0, 0, CMD_QUERY_DEV_LIM, + err = mthca_cmd_box(dev, 0, mailbox->dma, 0, 0, CMD_QUERY_DEV_LIM, CMD_TIME_CLASS_A, status); if (err) @@ -1041,15 +1078,15 @@ int mthca_QUERY_DEV_LIM(struct mthca_dev } out: - pci_free_consistent(dev->pdev, QUERY_DEV_LIM_OUT_SIZE, outbox, outdma); + mthca_free_mailbox(dev, mailbox); return err; } int mthca_QUERY_ADAPTER(struct mthca_dev *dev, struct mthca_adapter *adapter, u8 *status) { + struct mthca_mailbox *mailbox; u32 *outbox; - dma_addr_t outdma; int err; #define QUERY_ADAPTER_OUT_SIZE 0x100 @@ -1058,23 +1095,24 @@ int mthca_QUERY_ADAPTER(struct mthca_dev #define QUERY_ADAPTER_REVISION_ID_OFFSET 0x08 #define QUERY_ADAPTER_INTA_PIN_OFFSET 0x10 - outbox = pci_alloc_consistent(dev->pdev, QUERY_ADAPTER_OUT_SIZE, &outdma); - if (!outbox) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + outbox = mailbox->buf; - err = mthca_cmd_box(dev, 0, outdma, 0, 0, CMD_QUERY_ADAPTER, + err = mthca_cmd_box(dev, 0, mailbox->dma, 0, 0, CMD_QUERY_ADAPTER, CMD_TIME_CLASS_A, status); if (err) goto out; - MTHCA_GET(adapter->vendor_id, outbox, QUERY_ADAPTER_VENDOR_ID_OFFSET); - MTHCA_GET(adapter->device_id, outbox, QUERY_ADAPTER_DEVICE_ID_OFFSET); + MTHCA_GET(adapter->vendor_id, outbox, QUERY_ADAPTER_VENDOR_ID_OFFSET); + MTHCA_GET(adapter->device_id, outbox, QUERY_ADAPTER_DEVICE_ID_OFFSET); MTHCA_GET(adapter->revision_id, outbox, QUERY_ADAPTER_REVISION_ID_OFFSET); - MTHCA_GET(adapter->inta_pin, outbox, QUERY_ADAPTER_INTA_PIN_OFFSET); + MTHCA_GET(adapter->inta_pin, outbox, QUERY_ADAPTER_INTA_PIN_OFFSET); out: - pci_free_consistent(dev->pdev, QUERY_DEV_LIM_OUT_SIZE, outbox, outdma); + mthca_free_mailbox(dev, mailbox); return err; } @@ -1082,8 +1120,8 @@ int mthca_INIT_HCA(struct mthca_dev *dev struct mthca_init_hca_param *param, u8 *status) { + struct mthca_mailbox *mailbox; u32 *inbox; - dma_addr_t indma; int err; #define INIT_HCA_IN_SIZE 0x200 @@ -1123,9 +1161,10 @@ int mthca_INIT_HCA(struct mthca_dev *dev #define INIT_HCA_UAR_SCATCH_BASE_OFFSET (INIT_HCA_UAR_OFFSET + 0x10) #define INIT_HCA_UAR_CTX_BASE_OFFSET (INIT_HCA_UAR_OFFSET + 0x18) - inbox = pci_alloc_consistent(dev->pdev, INIT_HCA_IN_SIZE, &indma); - if (!inbox) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + inbox = mailbox->buf; memset(inbox, 0, INIT_HCA_IN_SIZE); @@ -1188,10 +1227,9 @@ int mthca_INIT_HCA(struct mthca_dev *dev MTHCA_PUT(inbox, param->uarc_base, INIT_HCA_UAR_CTX_BASE_OFFSET); } - err = mthca_cmd(dev, indma, 0, 0, CMD_INIT_HCA, - HZ, status); + err = mthca_cmd(dev, mailbox->dma, 0, 0, CMD_INIT_HCA, HZ, status); - pci_free_consistent(dev->pdev, INIT_HCA_IN_SIZE, inbox, indma); + mthca_free_mailbox(dev, mailbox); return err; } @@ -1199,8 +1237,8 @@ int mthca_INIT_IB(struct mthca_dev *dev, struct mthca_init_ib_param *param, int port, u8 *status) { + struct mthca_mailbox *mailbox; u32 *inbox; - dma_addr_t indma; int err; u32 flags; @@ -1220,9 +1258,10 @@ int mthca_INIT_IB(struct mthca_dev *dev, #define INIT_IB_NODE_GUID_OFFSET 0x18 #define INIT_IB_SI_GUID_OFFSET 0x20 - inbox = pci_alloc_consistent(dev->pdev, INIT_IB_IN_SIZE, &indma); - if (!inbox) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + inbox = mailbox->buf; memset(inbox, 0, INIT_IB_IN_SIZE); @@ -1242,10 +1281,10 @@ int mthca_INIT_IB(struct mthca_dev *dev, MTHCA_PUT(inbox, param->node_guid, INIT_IB_NODE_GUID_OFFSET); MTHCA_PUT(inbox, param->si_guid, INIT_IB_SI_GUID_OFFSET); - err = mthca_cmd(dev, indma, port, 0, CMD_INIT_IB, + err = mthca_cmd(dev, mailbox->dma, port, 0, CMD_INIT_IB, CMD_TIME_CLASS_A, status); - pci_free_consistent(dev->pdev, INIT_HCA_IN_SIZE, inbox, indma); + mthca_free_mailbox(dev, mailbox); return err; } @@ -1262,8 +1301,8 @@ int mthca_CLOSE_HCA(struct mthca_dev *de int mthca_SET_IB(struct mthca_dev *dev, struct mthca_set_ib_param *param, int port, u8 *status) { + struct mthca_mailbox *mailbox; u32 *inbox; - dma_addr_t indma; int err; u32 flags = 0; @@ -1274,9 +1313,10 @@ int mthca_SET_IB(struct mthca_dev *dev, #define SET_IB_CAP_MASK_OFFSET 0x04 #define SET_IB_SI_GUID_OFFSET 0x08 - inbox = pci_alloc_consistent(dev->pdev, SET_IB_IN_SIZE, &indma); - if (!inbox) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + inbox = mailbox->buf; memset(inbox, 0, SET_IB_IN_SIZE); @@ -1287,10 +1327,10 @@ int mthca_SET_IB(struct mthca_dev *dev, MTHCA_PUT(inbox, param->cap_mask, SET_IB_CAP_MASK_OFFSET); MTHCA_PUT(inbox, param->si_guid, SET_IB_SI_GUID_OFFSET); - err = mthca_cmd(dev, indma, port, 0, CMD_SET_IB, + err = mthca_cmd(dev, mailbox->dma, port, 0, CMD_SET_IB, CMD_TIME_CLASS_B, status); - pci_free_consistent(dev->pdev, INIT_HCA_IN_SIZE, inbox, indma); + mthca_free_mailbox(dev, mailbox); return err; } @@ -1301,20 +1341,22 @@ int mthca_MAP_ICM(struct mthca_dev *dev, int mthca_MAP_ICM_page(struct mthca_dev *dev, u64 dma_addr, u64 virt, u8 *status) { + struct mthca_mailbox *mailbox; u64 *inbox; - dma_addr_t indma; int err; - inbox = pci_alloc_consistent(dev->pdev, 16, &indma); - if (!inbox) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + inbox = mailbox->buf; inbox[0] = cpu_to_be64(virt); inbox[1] = cpu_to_be64(dma_addr); - err = mthca_cmd(dev, indma, 1, 0, CMD_MAP_ICM, CMD_TIME_CLASS_B, status); + err = mthca_cmd(dev, mailbox->dma, 1, 0, CMD_MAP_ICM, + CMD_TIME_CLASS_B, status); - pci_free_consistent(dev->pdev, 16, inbox, indma); + mthca_free_mailbox(dev, mailbox); if (!err) mthca_dbg(dev, "Mapped page at %llx to %llx for ICM.\n", @@ -1359,69 +1401,26 @@ int mthca_SET_ICM_SIZE(struct mthca_dev return 0; } -int mthca_SW2HW_MPT(struct mthca_dev *dev, void *mpt_entry, +int mthca_SW2HW_MPT(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int mpt_index, u8 *status) { - dma_addr_t indma; - int err; - - indma = pci_map_single(dev->pdev, mpt_entry, - MTHCA_MPT_ENTRY_SIZE, - PCI_DMA_TODEVICE); - if (pci_dma_mapping_error(indma)) - return -ENOMEM; - - err = mthca_cmd(dev, indma, mpt_index, 0, CMD_SW2HW_MPT, - CMD_TIME_CLASS_B, status); - - pci_unmap_single(dev->pdev, indma, - MTHCA_MPT_ENTRY_SIZE, PCI_DMA_TODEVICE); - return err; + return mthca_cmd(dev, mailbox->dma, mpt_index, 0, CMD_SW2HW_MPT, + CMD_TIME_CLASS_B, status); } -int mthca_HW2SW_MPT(struct mthca_dev *dev, void *mpt_entry, +int mthca_HW2SW_MPT(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int mpt_index, u8 *status) { - dma_addr_t outdma = 0; - int err; - - if (mpt_entry) { - outdma = pci_map_single(dev->pdev, mpt_entry, - MTHCA_MPT_ENTRY_SIZE, - PCI_DMA_FROMDEVICE); - if (pci_dma_mapping_error(outdma)) - return -ENOMEM; - } - - err = mthca_cmd_box(dev, 0, outdma, mpt_index, !mpt_entry, - CMD_HW2SW_MPT, - CMD_TIME_CLASS_B, status); - - if (mpt_entry) - pci_unmap_single(dev->pdev, outdma, - MTHCA_MPT_ENTRY_SIZE, - PCI_DMA_FROMDEVICE); - return err; + return mthca_cmd_box(dev, 0, mailbox ? mailbox->dma : 0, mpt_index, + !mailbox, CMD_HW2SW_MPT, + CMD_TIME_CLASS_B, status); } -int mthca_WRITE_MTT(struct mthca_dev *dev, u64 *mtt_entry, +int mthca_WRITE_MTT(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int num_mtt, u8 *status) { - dma_addr_t indma; - int err; - - indma = pci_map_single(dev->pdev, mtt_entry, - (num_mtt + 2) * 8, - PCI_DMA_TODEVICE); - if (pci_dma_mapping_error(indma)) - return -ENOMEM; - - err = mthca_cmd(dev, indma, num_mtt, 0, CMD_WRITE_MTT, - CMD_TIME_CLASS_B, status); - - pci_unmap_single(dev->pdev, indma, - (num_mtt + 2) * 8, PCI_DMA_TODEVICE); - return err; + return mthca_cmd(dev, mailbox->dma, num_mtt, 0, CMD_WRITE_MTT, + CMD_TIME_CLASS_B, status); } int mthca_SYNC_TPT(struct mthca_dev *dev, u8 *status) @@ -1439,92 +1438,38 @@ int mthca_MAP_EQ(struct mthca_dev *dev, 0, CMD_MAP_EQ, CMD_TIME_CLASS_B, status); } -int mthca_SW2HW_EQ(struct mthca_dev *dev, void *eq_context, +int mthca_SW2HW_EQ(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int eq_num, u8 *status) { - dma_addr_t indma; - int err; - - indma = pci_map_single(dev->pdev, eq_context, - MTHCA_EQ_CONTEXT_SIZE, - PCI_DMA_TODEVICE); - if (pci_dma_mapping_error(indma)) - return -ENOMEM; - - err = mthca_cmd(dev, indma, eq_num, 0, CMD_SW2HW_EQ, - CMD_TIME_CLASS_A, status); - - pci_unmap_single(dev->pdev, indma, - MTHCA_EQ_CONTEXT_SIZE, PCI_DMA_TODEVICE); - return err; + return mthca_cmd(dev, mailbox->dma, eq_num, 0, CMD_SW2HW_EQ, + CMD_TIME_CLASS_A, status); } -int mthca_HW2SW_EQ(struct mthca_dev *dev, void *eq_context, +int mthca_HW2SW_EQ(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int eq_num, u8 *status) { - dma_addr_t outdma = 0; - int err; - - outdma = pci_map_single(dev->pdev, eq_context, - MTHCA_EQ_CONTEXT_SIZE, - PCI_DMA_FROMDEVICE); - if (pci_dma_mapping_error(outdma)) - return -ENOMEM; - - err = mthca_cmd_box(dev, 0, outdma, eq_num, 0, - CMD_HW2SW_EQ, - CMD_TIME_CLASS_A, status); - - pci_unmap_single(dev->pdev, outdma, - MTHCA_EQ_CONTEXT_SIZE, - PCI_DMA_FROMDEVICE); - return err; + return mthca_cmd_box(dev, 0, mailbox->dma, eq_num, 0, + CMD_HW2SW_EQ, + CMD_TIME_CLASS_A, status); } -int mthca_SW2HW_CQ(struct mthca_dev *dev, void *cq_context, +int mthca_SW2HW_CQ(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int cq_num, u8 *status) { - dma_addr_t indma; - int err; - - indma = pci_map_single(dev->pdev, cq_context, - MTHCA_CQ_CONTEXT_SIZE, - PCI_DMA_TODEVICE); - if (pci_dma_mapping_error(indma)) - return -ENOMEM; - - err = mthca_cmd(dev, indma, cq_num, 0, CMD_SW2HW_CQ, + return mthca_cmd(dev, mailbox->dma, cq_num, 0, CMD_SW2HW_CQ, CMD_TIME_CLASS_A, status); - - pci_unmap_single(dev->pdev, indma, - MTHCA_CQ_CONTEXT_SIZE, PCI_DMA_TODEVICE); - return err; } -int mthca_HW2SW_CQ(struct mthca_dev *dev, void *cq_context, +int mthca_HW2SW_CQ(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int cq_num, u8 *status) { - dma_addr_t outdma = 0; - int err; - - outdma = pci_map_single(dev->pdev, cq_context, - MTHCA_CQ_CONTEXT_SIZE, - PCI_DMA_FROMDEVICE); - if (pci_dma_mapping_error(outdma)) - return -ENOMEM; - - err = mthca_cmd_box(dev, 0, outdma, cq_num, 0, - CMD_HW2SW_CQ, - CMD_TIME_CLASS_A, status); - - pci_unmap_single(dev->pdev, outdma, - MTHCA_CQ_CONTEXT_SIZE, - PCI_DMA_FROMDEVICE); - return err; + return mthca_cmd_box(dev, 0, mailbox->dma, cq_num, 0, + CMD_HW2SW_CQ, + CMD_TIME_CLASS_A, status); } int mthca_MODIFY_QP(struct mthca_dev *dev, int trans, u32 num, - int is_ee, void *qp_context, u32 optmask, + int is_ee, struct mthca_mailbox *mailbox, u32 optmask, u8 *status) { static const u16 op[] = { @@ -1541,36 +1486,34 @@ int mthca_MODIFY_QP(struct mthca_dev *de [MTHCA_TRANS_ANY2RST] = CMD_ERR2RST_QPEE }; u8 op_mod = 0; - - dma_addr_t indma; + int my_mailbox = 0; int err; if (trans < 0 || trans >= ARRAY_SIZE(op)) return -EINVAL; if (trans == MTHCA_TRANS_ANY2RST) { - indma = 0; op_mod = 3; /* don't write outbox, any->reset */ /* For debugging */ - qp_context = pci_alloc_consistent(dev->pdev, MTHCA_QP_CONTEXT_SIZE, - &indma); - op_mod = 2; /* write outbox, any->reset */ + if (!mailbox) { + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (!IS_ERR(mailbox)) { + my_mailbox = 1; + op_mod = 2; /* write outbox, any->reset */ + } else + mailbox = NULL; + } } else { - indma = pci_map_single(dev->pdev, qp_context, - MTHCA_QP_CONTEXT_SIZE, - PCI_DMA_TODEVICE); - if (pci_dma_mapping_error(indma)) - return -ENOMEM; - if (0) { int i; mthca_dbg(dev, "Dumping QP context:\n"); - printk(" opt param mask: %08x\n", be32_to_cpup(qp_context)); + printk(" opt param mask: %08x\n", be32_to_cpup(mailbox->buf)); for (i = 0; i < 0x100 / 4; ++i) { if (i % 8 == 0) printk(" [%02x] ", i * 4); - printk(" %08x", be32_to_cpu(((u32 *) qp_context)[i + 2])); + printk(" %08x", + be32_to_cpu(((u32 *) mailbox->buf)[i + 2])); if ((i + 1) % 8 == 0) printk("\n"); } @@ -1578,55 +1521,39 @@ int mthca_MODIFY_QP(struct mthca_dev *de } if (trans == MTHCA_TRANS_ANY2RST) { - err = mthca_cmd_box(dev, 0, indma, (!!is_ee << 24) | num, - op_mod, op[trans], CMD_TIME_CLASS_C, status); + err = mthca_cmd_box(dev, 0, mailbox ? mailbox->dma : 0, + (!!is_ee << 24) | num, op_mod, + op[trans], CMD_TIME_CLASS_C, status); - if (0) { + if (0 && mailbox) { int i; mthca_dbg(dev, "Dumping QP context:\n"); - printk(" %08x\n", be32_to_cpup(qp_context)); + printk(" %08x\n", be32_to_cpup(mailbox->buf)); for (i = 0; i < 0x100 / 4; ++i) { if (i % 8 == 0) printk("[%02x] ", i * 4); - printk(" %08x", be32_to_cpu(((u32 *) qp_context)[i + 2])); + printk(" %08x", + be32_to_cpu(((u32 *) mailbox->buf)[i + 2])); if ((i + 1) % 8 == 0) printk("\n"); } } } else - err = mthca_cmd(dev, indma, (!!is_ee << 24) | num, + err = mthca_cmd(dev, mailbox->dma, (!!is_ee << 24) | num, op_mod, op[trans], CMD_TIME_CLASS_C, status); - if (trans != MTHCA_TRANS_ANY2RST) - pci_unmap_single(dev->pdev, indma, - MTHCA_QP_CONTEXT_SIZE, PCI_DMA_TODEVICE); - else - pci_free_consistent(dev->pdev, MTHCA_QP_CONTEXT_SIZE, - qp_context, indma); + if (my_mailbox) + mthca_free_mailbox(dev, mailbox); + return err; } int mthca_QUERY_QP(struct mthca_dev *dev, u32 num, int is_ee, - void *qp_context, u8 *status) + struct mthca_mailbox *mailbox, u8 *status) { - dma_addr_t outdma = 0; - int err; - - outdma = pci_map_single(dev->pdev, qp_context, - MTHCA_QP_CONTEXT_SIZE, - PCI_DMA_FROMDEVICE); - if (pci_dma_mapping_error(outdma)) - return -ENOMEM; - - err = mthca_cmd_box(dev, 0, outdma, (!!is_ee << 24) | num, 0, - CMD_QUERY_QPEE, - CMD_TIME_CLASS_A, status); - - pci_unmap_single(dev->pdev, outdma, - MTHCA_QP_CONTEXT_SIZE, - PCI_DMA_FROMDEVICE); - return err; + return mthca_cmd_box(dev, 0, mailbox->dma, (!!is_ee << 24) | num, 0, + CMD_QUERY_QPEE, CMD_TIME_CLASS_A, status); } int mthca_CONF_SPECIAL_QP(struct mthca_dev *dev, int type, u32 qpn, @@ -1656,11 +1583,11 @@ int mthca_CONF_SPECIAL_QP(struct mthca_d } int mthca_MAD_IFC(struct mthca_dev *dev, int ignore_mkey, int ignore_bkey, - int port, struct ib_wc* in_wc, struct ib_grh* in_grh, + int port, struct ib_wc *in_wc, struct ib_grh *in_grh, void *in_mad, void *response_mad, u8 *status) { - void *box; - dma_addr_t dma; + struct mthca_mailbox *inmailbox, *outmailbox; + void *inbox; int err; u32 in_modifier = port; u8 op_modifier = 0; @@ -1674,11 +1601,18 @@ int mthca_MAD_IFC(struct mthca_dev *dev, #define MAD_IFC_PKEY_OFFSET 0x10e #define MAD_IFC_GRH_OFFSET 0x140 - box = pci_alloc_consistent(dev->pdev, MAD_IFC_BOX_SIZE, &dma); - if (!box) - return -ENOMEM; + inmailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(inmailbox)) + return PTR_ERR(inmailbox); + inbox = inmailbox->buf; + + outmailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(outmailbox)) { + mthca_free_mailbox(dev, inmailbox); + return PTR_ERR(outmailbox); + } - memcpy(box, in_mad, 256); + memcpy(inbox, in_mad, 256); /* * Key check traps can't be generated unless we have in_wc to @@ -1692,97 +1626,65 @@ int mthca_MAD_IFC(struct mthca_dev *dev, if (in_wc) { u8 val; - memset(box + 256, 0, 256); + memset(inbox + 256, 0, 256); - MTHCA_PUT(box, in_wc->qp_num, MAD_IFC_MY_QPN_OFFSET); - MTHCA_PUT(box, in_wc->src_qp, MAD_IFC_RQPN_OFFSET); + MTHCA_PUT(inbox, in_wc->qp_num, MAD_IFC_MY_QPN_OFFSET); + MTHCA_PUT(inbox, in_wc->src_qp, MAD_IFC_RQPN_OFFSET); val = in_wc->sl << 4; - MTHCA_PUT(box, val, MAD_IFC_SL_OFFSET); + MTHCA_PUT(inbox, val, MAD_IFC_SL_OFFSET); val = in_wc->dlid_path_bits | (in_wc->wc_flags & IB_WC_GRH ? 0x80 : 0); - MTHCA_PUT(box, val, MAD_IFC_GRH_OFFSET); + MTHCA_PUT(inbox, val, MAD_IFC_GRH_OFFSET); - MTHCA_PUT(box, in_wc->slid, MAD_IFC_RLID_OFFSET); - MTHCA_PUT(box, in_wc->pkey_index, MAD_IFC_PKEY_OFFSET); + MTHCA_PUT(inbox, in_wc->slid, MAD_IFC_RLID_OFFSET); + MTHCA_PUT(inbox, in_wc->pkey_index, MAD_IFC_PKEY_OFFSET); if (in_grh) - memcpy((u8 *) box + MAD_IFC_GRH_OFFSET, in_grh, 40); + memcpy(inbox + MAD_IFC_GRH_OFFSET, in_grh, 40); op_modifier |= 0x10; in_modifier |= in_wc->slid << 16; } - err = mthca_cmd_box(dev, dma, dma + 512, in_modifier, op_modifier, + err = mthca_cmd_box(dev, inmailbox->dma, outmailbox->dma, + in_modifier, op_modifier, CMD_MAD_IFC, CMD_TIME_CLASS_C, status); if (!err && !*status) - memcpy(response_mad, box + 512, 256); + memcpy(response_mad, outmailbox->buf, 256); - pci_free_consistent(dev->pdev, MAD_IFC_BOX_SIZE, box, dma); + mthca_free_mailbox(dev, inmailbox); + mthca_free_mailbox(dev, outmailbox); return err; } -int mthca_READ_MGM(struct mthca_dev *dev, int index, void *mgm, - u8 *status) +int mthca_READ_MGM(struct mthca_dev *dev, int index, + struct mthca_mailbox *mailbox, u8 *status) { - dma_addr_t outdma = 0; - int err; - - outdma = pci_map_single(dev->pdev, mgm, - MTHCA_MGM_ENTRY_SIZE, - PCI_DMA_FROMDEVICE); - if (pci_dma_mapping_error(outdma)) - return -ENOMEM; - - err = mthca_cmd_box(dev, 0, outdma, index, 0, - CMD_READ_MGM, - CMD_TIME_CLASS_A, status); - - pci_unmap_single(dev->pdev, outdma, - MTHCA_MGM_ENTRY_SIZE, - PCI_DMA_FROMDEVICE); - return err; + return mthca_cmd_box(dev, 0, mailbox->dma, index, 0, + CMD_READ_MGM, CMD_TIME_CLASS_A, status); } -int mthca_WRITE_MGM(struct mthca_dev *dev, int index, void *mgm, - u8 *status) +int mthca_WRITE_MGM(struct mthca_dev *dev, int index, + struct mthca_mailbox *mailbox, u8 *status) { - dma_addr_t indma; - int err; - - indma = pci_map_single(dev->pdev, mgm, - MTHCA_MGM_ENTRY_SIZE, - PCI_DMA_TODEVICE); - if (pci_dma_mapping_error(indma)) - return -ENOMEM; - - err = mthca_cmd(dev, indma, index, 0, CMD_WRITE_MGM, - CMD_TIME_CLASS_A, status); - - pci_unmap_single(dev->pdev, indma, - MTHCA_MGM_ENTRY_SIZE, PCI_DMA_TODEVICE); - return err; + return mthca_cmd(dev, mailbox->dma, index, 0, CMD_WRITE_MGM, + CMD_TIME_CLASS_A, status); } -int mthca_MGID_HASH(struct mthca_dev *dev, void *gid, u16 *hash, - u8 *status) +int mthca_MGID_HASH(struct mthca_dev *dev, struct mthca_mailbox *mailbox, + u16 *hash, u8 *status) { - dma_addr_t indma; u64 imm; int err; - indma = pci_map_single(dev->pdev, gid, 16, PCI_DMA_TODEVICE); - if (pci_dma_mapping_error(indma)) - return -ENOMEM; - - err = mthca_cmd_imm(dev, indma, &imm, 0, 0, CMD_MGID_HASH, + err = mthca_cmd_imm(dev, mailbox->dma, &imm, 0, 0, CMD_MGID_HASH, CMD_TIME_CLASS_A, status); - *hash = imm; - pci_unmap_single(dev->pdev, indma, 16, PCI_DMA_TODEVICE); + *hash = imm; return err; } --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cmd.h 2005-06-23 13:03:08.795213517 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cmd.h 2005-06-23 13:03:09.282108141 -0700 @@ -37,8 +37,7 @@ #include -#define MTHCA_CMD_MAILBOX_ALIGN 16UL -#define MTHCA_CMD_MAILBOX_EXTRA (MTHCA_CMD_MAILBOX_ALIGN - 1) +#define MTHCA_MAILBOX_SIZE 4096 enum { /* command completed successfully: */ @@ -112,6 +111,11 @@ enum { DEV_LIM_FLAG_UD_MULTI = 1 << 21, }; +struct mthca_mailbox { + dma_addr_t dma; + void *buf; +}; + struct mthca_dev_lim { int max_srq_sz; int max_qp_sz; @@ -242,6 +246,10 @@ void mthca_cmd_use_polling(struct mthca_ void mthca_cmd_event(struct mthca_dev *dev, u16 token, u8 status, u64 out_param); +struct mthca_mailbox *mthca_alloc_mailbox(struct mthca_dev *dev, + unsigned int gfp_mask); +void mthca_free_mailbox(struct mthca_dev *dev, struct mthca_mailbox *mailbox); + int mthca_SYS_EN(struct mthca_dev *dev, u8 *status); int mthca_SYS_DIS(struct mthca_dev *dev, u8 *status); int mthca_MAP_FA(struct mthca_dev *dev, struct mthca_icm *icm, u8 *status); @@ -272,41 +280,39 @@ int mthca_MAP_ICM_AUX(struct mthca_dev * int mthca_UNMAP_ICM_AUX(struct mthca_dev *dev, u8 *status); int mthca_SET_ICM_SIZE(struct mthca_dev *dev, u64 icm_size, u64 *aux_pages, u8 *status); -int mthca_SW2HW_MPT(struct mthca_dev *dev, void *mpt_entry, +int mthca_SW2HW_MPT(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int mpt_index, u8 *status); -int mthca_HW2SW_MPT(struct mthca_dev *dev, void *mpt_entry, +int mthca_HW2SW_MPT(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int mpt_index, u8 *status); -int mthca_WRITE_MTT(struct mthca_dev *dev, u64 *mtt_entry, +int mthca_WRITE_MTT(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int num_mtt, u8 *status); int mthca_SYNC_TPT(struct mthca_dev *dev, u8 *status); int mthca_MAP_EQ(struct mthca_dev *dev, u64 event_mask, int unmap, int eq_num, u8 *status); -int mthca_SW2HW_EQ(struct mthca_dev *dev, void *eq_context, +int mthca_SW2HW_EQ(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int eq_num, u8 *status); -int mthca_HW2SW_EQ(struct mthca_dev *dev, void *eq_context, +int mthca_HW2SW_EQ(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int eq_num, u8 *status); -int mthca_SW2HW_CQ(struct mthca_dev *dev, void *cq_context, +int mthca_SW2HW_CQ(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int cq_num, u8 *status); -int mthca_HW2SW_CQ(struct mthca_dev *dev, void *cq_context, +int mthca_HW2SW_CQ(struct mthca_dev *dev, struct mthca_mailbox *mailbox, int cq_num, u8 *status); int mthca_MODIFY_QP(struct mthca_dev *dev, int trans, u32 num, - int is_ee, void *qp_context, u32 optmask, + int is_ee, struct mthca_mailbox *mailbox, u32 optmask, u8 *status); int mthca_QUERY_QP(struct mthca_dev *dev, u32 num, int is_ee, - void *qp_context, u8 *status); + struct mthca_mailbox *mailbox, u8 *status); int mthca_CONF_SPECIAL_QP(struct mthca_dev *dev, int type, u32 qpn, u8 *status); int mthca_MAD_IFC(struct mthca_dev *dev, int ignore_mkey, int ignore_bkey, - int port, struct ib_wc* in_wc, struct ib_grh* in_grh, + int port, struct ib_wc *in_wc, struct ib_grh *in_grh, void *in_mad, void *response_mad, u8 *status); -int mthca_READ_MGM(struct mthca_dev *dev, int index, void *mgm, - u8 *status); -int mthca_WRITE_MGM(struct mthca_dev *dev, int index, void *mgm, - u8 *status); -int mthca_MGID_HASH(struct mthca_dev *dev, void *gid, u16 *hash, - u8 *status); +int mthca_READ_MGM(struct mthca_dev *dev, int index, + struct mthca_mailbox *mailbox, u8 *status); +int mthca_WRITE_MGM(struct mthca_dev *dev, int index, + struct mthca_mailbox *mailbox, u8 *status); +int mthca_MGID_HASH(struct mthca_dev *dev, struct mthca_mailbox *mailbox, + u16 *hash, u8 *status); int mthca_NOP(struct mthca_dev *dev, u8 *status); -#define MAILBOX_ALIGN(x) ((void *) ALIGN((unsigned long) (x), MTHCA_CMD_MAILBOX_ALIGN)) - #endif /* MTHCA_CMD_H */ --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:06.793646706 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-23 13:03:09.280108573 -0700 @@ -745,7 +745,7 @@ int mthca_init_cq(struct mthca_dev *dev, struct mthca_cq *cq) { int size = nent * MTHCA_CQ_ENTRY_SIZE; - void *mailbox = NULL; + struct mthca_mailbox *mailbox; struct mthca_cq_context *cq_context; int err = -ENOMEM; u8 status; @@ -779,12 +779,11 @@ int mthca_init_cq(struct mthca_dev *dev, goto err_out_ci; } - mailbox = kmalloc(sizeof (struct mthca_cq_context) + MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) - goto err_out_mailbox; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + goto err_out_arm; - cq_context = MAILBOX_ALIGN(mailbox); + cq_context = mailbox->buf; err = mthca_alloc_cq_buf(dev, size, cq); if (err) @@ -815,7 +814,7 @@ int mthca_init_cq(struct mthca_dev *dev, cq_context->state_db = cpu_to_be32(cq->arm_db_index); } - err = mthca_SW2HW_CQ(dev, cq_context, cq->cqn, &status); + err = mthca_SW2HW_CQ(dev, mailbox, cq->cqn, &status); if (err) { mthca_warn(dev, "SW2HW_CQ failed (%d)\n", err); goto err_out_free_mr; @@ -839,7 +838,7 @@ int mthca_init_cq(struct mthca_dev *dev, cq->cons_index = 0; - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); return 0; @@ -848,8 +847,9 @@ err_out_free_mr: mthca_free_cq_buf(dev, cq); err_out_mailbox: - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); +err_out_arm: if (mthca_is_memfree(dev)) mthca_free_db(dev, MTHCA_DB_TYPE_CQ_ARM, cq->arm_db_index); @@ -869,28 +869,26 @@ err_out: void mthca_free_cq(struct mthca_dev *dev, struct mthca_cq *cq) { - void *mailbox; + struct mthca_mailbox *mailbox; int err; u8 status; might_sleep(); - mailbox = kmalloc(sizeof (struct mthca_cq_context) + MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) { + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) { mthca_warn(dev, "No memory for mailbox to free CQ.\n"); return; } - err = mthca_HW2SW_CQ(dev, MAILBOX_ALIGN(mailbox), cq->cqn, &status); + err = mthca_HW2SW_CQ(dev, mailbox, cq->cqn, &status); if (err) mthca_warn(dev, "HW2SW_CQ failed (%d)\n", err); else if (status) - mthca_warn(dev, "HW2SW_CQ returned status 0x%02x\n", - status); + mthca_warn(dev, "HW2SW_CQ returned status 0x%02x\n", status); if (0) { - u32 *ctx = MAILBOX_ALIGN(mailbox); + u32 *ctx = mailbox->buf; int j; printk(KERN_ERR "context for CQN %x (cons index %x, next sw %d)\n", @@ -922,7 +920,7 @@ void mthca_free_cq(struct mthca_dev *dev mthca_table_put(dev, dev->cq_table.table, cq->cqn); mthca_free(&dev->cq_table.alloc, cq->cqn); - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); } int __devinit mthca_init_cq_table(struct mthca_dev *dev) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-23 13:03:07.308535271 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-23 13:03:09.283107924 -0700 @@ -99,6 +99,7 @@ enum { }; struct mthca_cmd { + struct pci_pool *pool; int use_events; struct semaphore hcr_sem; struct semaphore poll_sem; --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_eq.c 2005-06-23 13:03:04.752088550 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_eq.c 2005-06-23 13:03:09.281108357 -0700 @@ -469,7 +469,7 @@ static int __devinit mthca_create_eq(str PAGE_SIZE; u64 *dma_list = NULL; dma_addr_t t; - void *mailbox = NULL; + struct mthca_mailbox *mailbox; struct mthca_eq_context *eq_context; int err = -ENOMEM; int i; @@ -494,17 +494,16 @@ static int __devinit mthca_create_eq(str if (!dma_list) goto err_out_free; - mailbox = kmalloc(sizeof *eq_context + MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) goto err_out_free; - eq_context = MAILBOX_ALIGN(mailbox); + eq_context = mailbox->buf; for (i = 0; i < npages; ++i) { eq->page_list[i].buf = dma_alloc_coherent(&dev->pdev->dev, PAGE_SIZE, &t, GFP_KERNEL); if (!eq->page_list[i].buf) - goto err_out_free; + goto err_out_free_pages; dma_list[i] = t; pci_unmap_addr_set(&eq->page_list[i], mapping, t); @@ -517,7 +516,7 @@ static int __devinit mthca_create_eq(str eq->eqn = mthca_alloc(&dev->eq_table.alloc); if (eq->eqn == -1) - goto err_out_free; + goto err_out_free_pages; err = mthca_mr_alloc_phys(dev, dev->driver_pd.pd_num, dma_list, PAGE_SHIFT, npages, @@ -548,7 +547,7 @@ static int __devinit mthca_create_eq(str eq_context->intr = intr; eq_context->lkey = cpu_to_be32(eq->mr.ibmr.lkey); - err = mthca_SW2HW_EQ(dev, eq_context, eq->eqn, &status); + err = mthca_SW2HW_EQ(dev, mailbox, eq->eqn, &status); if (err) { mthca_warn(dev, "SW2HW_EQ failed (%d)\n", err); goto err_out_free_mr; @@ -561,7 +560,7 @@ static int __devinit mthca_create_eq(str } kfree(dma_list); - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); eq->eqn_mask = swab32(1 << eq->eqn); eq->cons_index = 0; @@ -579,7 +578,7 @@ static int __devinit mthca_create_eq(str err_out_free_eq: mthca_free(&dev->eq_table.alloc, eq->eqn); - err_out_free: + err_out_free_pages: for (i = 0; i < npages; ++i) if (eq->page_list[i].buf) dma_free_coherent(&dev->pdev->dev, PAGE_SIZE, @@ -587,9 +586,11 @@ static int __devinit mthca_create_eq(str pci_unmap_addr(&eq->page_list[i], mapping)); + mthca_free_mailbox(dev, mailbox); + + err_out_free: kfree(eq->page_list); kfree(dma_list); - kfree(mailbox); err_out: return err; @@ -598,20 +599,18 @@ static int __devinit mthca_create_eq(str static void mthca_free_eq(struct mthca_dev *dev, struct mthca_eq *eq) { - void *mailbox = NULL; + struct mthca_mailbox *mailbox; int err; u8 status; int npages = (eq->nent * MTHCA_EQ_ENTRY_SIZE + PAGE_SIZE - 1) / PAGE_SIZE; int i; - mailbox = kmalloc(sizeof (struct mthca_eq_context) + MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) return; - err = mthca_HW2SW_EQ(dev, MAILBOX_ALIGN(mailbox), - eq->eqn, &status); + err = mthca_HW2SW_EQ(dev, mailbox, eq->eqn, &status); if (err) mthca_warn(dev, "HW2SW_EQ failed (%d)\n", err); if (status) @@ -624,7 +623,7 @@ static void mthca_free_eq(struct mthca_d for (i = 0; i < sizeof (struct mthca_eq_context) / 4; ++i) { if (i % 4 == 0) printk("[%02x] ", i * 4); - printk(" %08x", be32_to_cpup(MAILBOX_ALIGN(mailbox) + i * 4)); + printk(" %08x", be32_to_cpup(mailbox->buf + i * 4)); if ((i + 1) % 4 == 0) printk("\n"); } @@ -637,7 +636,7 @@ static void mthca_free_eq(struct mthca_d pci_unmap_addr(&eq->page_list[i], mapping)); kfree(eq->page_list); - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); } static void mthca_free_irqs(struct mthca_dev *dev) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_mcg.c 2005-06-23 13:03:00.964908188 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_mcg.c 2005-06-23 13:03:09.281108357 -0700 @@ -66,22 +66,23 @@ static const u8 zero_gid[16]; /* automat * entry in hash chain and *mgm holds end of hash chain. */ static int find_mgm(struct mthca_dev *dev, - u8 *gid, struct mthca_mgm *mgm, + u8 *gid, struct mthca_mailbox *mgm_mailbox, u16 *hash, int *prev, int *index) { - void *mailbox; + struct mthca_mailbox *mailbox; + struct mthca_mgm *mgm = mgm_mailbox->buf; u8 *mgid; int err; u8 status; - mailbox = kmalloc(16 + MTHCA_CMD_MAILBOX_EXTRA, GFP_KERNEL); - if (!mailbox) + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) return -ENOMEM; - mgid = MAILBOX_ALIGN(mailbox); + mgid = mailbox->buf; memcpy(mgid, gid, 16); - err = mthca_MGID_HASH(dev, mgid, hash, &status); + err = mthca_MGID_HASH(dev, mailbox, hash, &status); if (err) goto out; if (status) { @@ -103,7 +104,7 @@ static int find_mgm(struct mthca_dev *de *prev = -1; do { - err = mthca_READ_MGM(dev, *index, mgm, &status); + err = mthca_READ_MGM(dev, *index, mgm_mailbox, &status); if (err) goto out; if (status) { @@ -129,14 +130,14 @@ static int find_mgm(struct mthca_dev *de *index = -1; out: - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); return err; } int mthca_multicast_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid) { struct mthca_dev *dev = to_mdev(ibqp->device); - void *mailbox; + struct mthca_mailbox *mailbox; struct mthca_mgm *mgm; u16 hash; int index, prev; @@ -145,15 +146,15 @@ int mthca_multicast_attach(struct ib_qp int err; u8 status; - mailbox = kmalloc(sizeof *mgm + MTHCA_CMD_MAILBOX_EXTRA, GFP_KERNEL); - if (!mailbox) - return -ENOMEM; - mgm = MAILBOX_ALIGN(mailbox); + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + mgm = mailbox->buf; if (down_interruptible(&dev->mcg_table.sem)) return -EINTR; - err = find_mgm(dev, gid->raw, mgm, &hash, &prev, &index); + err = find_mgm(dev, gid->raw, mailbox, &hash, &prev, &index); if (err) goto out; @@ -170,7 +171,7 @@ int mthca_multicast_attach(struct ib_qp goto out; } - err = mthca_READ_MGM(dev, index, mgm, &status); + err = mthca_READ_MGM(dev, index, mailbox, &status); if (err) goto out; if (status) { @@ -195,7 +196,7 @@ int mthca_multicast_attach(struct ib_qp goto out; } - err = mthca_WRITE_MGM(dev, index, mgm, &status); + err = mthca_WRITE_MGM(dev, index, mailbox, &status); if (err) goto out; if (status) { @@ -206,7 +207,7 @@ int mthca_multicast_attach(struct ib_qp if (!link) goto out; - err = mthca_READ_MGM(dev, prev, mgm, &status); + err = mthca_READ_MGM(dev, prev, mailbox, &status); if (err) goto out; if (status) { @@ -217,7 +218,7 @@ int mthca_multicast_attach(struct ib_qp mgm->next_gid_index = cpu_to_be32(index << 5); - err = mthca_WRITE_MGM(dev, prev, mgm, &status); + err = mthca_WRITE_MGM(dev, prev, mailbox, &status); if (err) goto out; if (status) { @@ -227,14 +228,14 @@ int mthca_multicast_attach(struct ib_qp out: up(&dev->mcg_table.sem); - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); return err; } int mthca_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid) { struct mthca_dev *dev = to_mdev(ibqp->device); - void *mailbox; + struct mthca_mailbox *mailbox; struct mthca_mgm *mgm; u16 hash; int prev, index; @@ -242,15 +243,15 @@ int mthca_multicast_detach(struct ib_qp int err; u8 status; - mailbox = kmalloc(sizeof *mgm + MTHCA_CMD_MAILBOX_EXTRA, GFP_KERNEL); - if (!mailbox) - return -ENOMEM; - mgm = MAILBOX_ALIGN(mailbox); + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + mgm = mailbox->buf; if (down_interruptible(&dev->mcg_table.sem)) return -EINTR; - err = find_mgm(dev, gid->raw, mgm, &hash, &prev, &index); + err = find_mgm(dev, gid->raw, mailbox, &hash, &prev, &index); if (err) goto out; @@ -285,7 +286,7 @@ int mthca_multicast_detach(struct ib_qp mgm->qp[loc] = mgm->qp[i - 1]; mgm->qp[i - 1] = 0; - err = mthca_WRITE_MGM(dev, index, mgm, &status); + err = mthca_WRITE_MGM(dev, index, mailbox, &status); if (err) goto out; if (status) { @@ -304,7 +305,7 @@ int mthca_multicast_detach(struct ib_qp if (be32_to_cpu(mgm->next_gid_index) >> 5) { err = mthca_READ_MGM(dev, be32_to_cpu(mgm->next_gid_index) >> 5, - mgm, &status); + mailbox, &status); if (err) goto out; if (status) { @@ -316,7 +317,7 @@ int mthca_multicast_detach(struct ib_qp } else memset(mgm->gid, 0, 16); - err = mthca_WRITE_MGM(dev, index, mgm, &status); + err = mthca_WRITE_MGM(dev, index, mailbox, &status); if (err) goto out; if (status) { @@ -327,7 +328,7 @@ int mthca_multicast_detach(struct ib_qp } else { /* Remove entry from AMGM */ index = be32_to_cpu(mgm->next_gid_index) >> 5; - err = mthca_READ_MGM(dev, prev, mgm, &status); + err = mthca_READ_MGM(dev, prev, mailbox, &status); if (err) goto out; if (status) { @@ -338,7 +339,7 @@ int mthca_multicast_detach(struct ib_qp mgm->next_gid_index = cpu_to_be32(index << 5); - err = mthca_WRITE_MGM(dev, prev, mgm, &status); + err = mthca_WRITE_MGM(dev, prev, mailbox, &status); if (err) goto out; if (status) { @@ -350,7 +351,7 @@ int mthca_multicast_detach(struct ib_qp out: up(&dev->mcg_table.sem); - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); return err; } --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_mr.c 2005-06-23 13:03:07.308535271 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_mr.c 2005-06-23 13:03:09.279108790 -0700 @@ -246,21 +246,23 @@ void mthca_free_mtt(struct mthca_dev *de int mthca_write_mtt(struct mthca_dev *dev, struct mthca_mtt *mtt, int start_index, u64 *buffer_list, int list_len) { + struct mthca_mailbox *mailbox; u64 *mtt_entry; int err = 0; u8 status; int i; - mtt_entry = (u64 *) __get_free_page(GFP_KERNEL); - if (!mtt_entry) - return -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + mtt_entry = mailbox->buf; while (list_len > 0) { mtt_entry[0] = cpu_to_be64(dev->mr_table.mtt_base + mtt->first_seg * MTHCA_MTT_SEG_SIZE + start_index * 8); mtt_entry[1] = 0; - for (i = 0; i < list_len && i < PAGE_SIZE / 8 - 2; ++i) + for (i = 0; i < list_len && i < MTHCA_MAILBOX_SIZE / 8 - 2; ++i) mtt_entry[i + 2] = cpu_to_be64(buffer_list[i] | MTHCA_MTT_FLAG_PRESENT); @@ -271,7 +273,7 @@ int mthca_write_mtt(struct mthca_dev *de if (i & 1) mtt_entry[i + 2] = 0; - err = mthca_WRITE_MTT(dev, mtt_entry, (i + 1) & ~1, &status); + err = mthca_WRITE_MTT(dev, mailbox, (i + 1) & ~1, &status); if (err) { mthca_warn(dev, "WRITE_MTT failed (%d)\n", err); goto out; @@ -289,7 +291,7 @@ int mthca_write_mtt(struct mthca_dev *de } out: - free_page((unsigned long) mtt_entry); + mthca_free_mailbox(dev, mailbox); return err; } @@ -332,7 +334,7 @@ static inline u32 key_to_hw_index(struct int mthca_mr_alloc(struct mthca_dev *dev, u32 pd, int buffer_size_shift, u64 iova, u64 total_size, u32 access, struct mthca_mr *mr) { - void *mailbox; + struct mthca_mailbox *mailbox; struct mthca_mpt_entry *mpt_entry; u32 key; int i; @@ -354,13 +356,12 @@ int mthca_mr_alloc(struct mthca_dev *dev goto err_out_mpt_free; } - mailbox = kmalloc(sizeof *mpt_entry + MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) { - err = -ENOMEM; + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) { + err = PTR_ERR(mailbox); goto err_out_table; } - mpt_entry = MAILBOX_ALIGN(mailbox); + mpt_entry = mailbox->buf; mpt_entry->flags = cpu_to_be32(MTHCA_MPT_FLAG_SW_OWNS | MTHCA_MPT_FLAG_MIO | @@ -394,7 +395,7 @@ int mthca_mr_alloc(struct mthca_dev *dev } } - err = mthca_SW2HW_MPT(dev, mpt_entry, + err = mthca_SW2HW_MPT(dev, mailbox, key & (dev->limits.num_mpts - 1), &status); if (err) { @@ -407,11 +408,11 @@ int mthca_mr_alloc(struct mthca_dev *dev goto err_out_mailbox; } - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); return err; err_out_mailbox: - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); err_out_table: mthca_table_put(dev, dev->mr_table.mpt_table, key); @@ -487,7 +488,7 @@ int mthca_fmr_alloc(struct mthca_dev *de u32 access, struct mthca_fmr *mr) { struct mthca_mpt_entry *mpt_entry; - void *mailbox; + struct mthca_mailbox *mailbox; u64 mtt_seg; u32 key, idx; u8 status; @@ -538,12 +539,11 @@ int mthca_fmr_alloc(struct mthca_dev *de } else mr->mem.tavor.mtts = dev->mr_table.tavor_fmr.mtt_base + mtt_seg; - mailbox = kmalloc(sizeof *mpt_entry + MTHCA_CMD_MAILBOX_EXTRA, - GFP_KERNEL); - if (!mailbox) + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) goto err_out_free_mtt; - mpt_entry = MAILBOX_ALIGN(mailbox); + mpt_entry = mailbox->buf; mpt_entry->flags = cpu_to_be32(MTHCA_MPT_FLAG_SW_OWNS | MTHCA_MPT_FLAG_MIO | @@ -568,7 +568,7 @@ int mthca_fmr_alloc(struct mthca_dev *de } } - err = mthca_SW2HW_MPT(dev, mpt_entry, + err = mthca_SW2HW_MPT(dev, mailbox, key & (dev->limits.num_mpts - 1), &status); if (err) { @@ -582,11 +582,11 @@ int mthca_fmr_alloc(struct mthca_dev *de goto err_out_mailbox_free; } - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); return 0; err_out_mailbox_free: - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); err_out_free_mtt: mthca_free_mtt(dev, mr->mtt); --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:06.792646922 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-23 13:03:09.280108573 -0700 @@ -589,7 +589,7 @@ int mthca_modify_qp(struct ib_qp *ibqp, struct mthca_dev *dev = to_mdev(ibqp->device); struct mthca_qp *qp = to_mqp(ibqp); enum ib_qp_state cur_state, new_state; - void *mailbox = NULL; + struct mthca_mailbox *mailbox; struct mthca_qp_param *qp_param; struct mthca_qp_context *qp_context; u32 req_param, opt_param; @@ -646,10 +646,10 @@ int mthca_modify_qp(struct ib_qp *ibqp, return -EINVAL; } - mailbox = kmalloc(sizeof (*qp_param) + MTHCA_CMD_MAILBOX_EXTRA, GFP_KERNEL); - if (!mailbox) - return -ENOMEM; - qp_param = MAILBOX_ALIGN(mailbox); + mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + qp_param = mailbox->buf; qp_context = &qp_param->context; memset(qp_param, 0, sizeof *qp_param); @@ -872,7 +872,7 @@ int mthca_modify_qp(struct ib_qp *ibqp, } err = mthca_MODIFY_QP(dev, state_table[cur_state][new_state].trans, - qp->qpn, 0, qp_param, 0, &status); + qp->qpn, 0, mailbox, 0, &status); if (status) { mthca_warn(dev, "modify QP %d returned status %02x.\n", state_table[cur_state][new_state].trans, status); @@ -882,7 +882,7 @@ int mthca_modify_qp(struct ib_qp *ibqp, if (!err) qp->state = new_state; - kfree(mailbox); + mthca_free_mailbox(dev, mailbox); if (is_sqp(dev, qp)) store_attrs(to_msqp(qp), attr, attr_mask); From roland at topspin.com Thu Jun 23 21:04:21 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 23 Jun 2005 21:04:21 -0700 Subject: [openib-general] [PATCH 14/14] IB/mthca: Bump version In-Reply-To: <2005623214.1yVGvzp1yi0UQ3bC@topspin.com> Message-ID: <2005623214.KAnnSVPywiOCFGvX@topspin.com> It's about time for a version bump. Signed-off-by: Roland Dreier --- linux.git/drivers/infiniband/hw/mthca/mthca_dev.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) --- linux.git.orig/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-23 13:03:09.283107924 -0700 +++ linux.git/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-23 13:03:10.111928547 -0700 @@ -47,8 +47,8 @@ #define DRV_NAME "ib_mthca" #define PFX DRV_NAME ": " -#define DRV_VERSION "0.06-pre" -#define DRV_RELDATE "November 8, 2004" +#define DRV_VERSION "0.06" +#define DRV_RELDATE "June 23, 2005" enum { MTHCA_FLAG_DDR_HIDDEN = 1 << 1, From halr at voltaire.com Fri Jun 24 06:03:01 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 09:03:01 -0400 Subject: [openib-general] Re: [openib-commits] r2695 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <20050623191815.CA02B2283E1@openib.ca.sandia.gov> References: <20050623191815.CA02B2283E1@openib.ca.sandia.gov> Message-ID: <1119618180.4477.3372.camel@localhost.localdomain> On Thu, 2005-06-23 at 15:18, jlentini at openib.org wrote: > Author: jlentini > Date: 2005-06-23 12:18:14 -0700 (Thu, 23 Jun 2005) > New Revision: 2695 > > Modified: > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c > Log: > Retry normal packets but not the CM messages. CM message retries are > linked to dat_ep_connect()'s timeout value > Signed-off-by: James Lentini > > > Modified: gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c > =================================================================== > -- gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c 2005-06-23 19:15:12 UTC (rev 2694) > +++ gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c 2005-06-23 19:18:14 UTC (rev 2695) > @@ -39,10 +39,10 @@ ... > #define DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ > -#define DAPL_IB_MAX_CM_RETRIES 4 > +#define DAPL_IB_MAX_CM_RETRIES 0 Until the algorithm previously discussed is implemented, changing DAPL_IB_MAX_CM_RETRIES to 0 makes this more frail (in the case of a lost REQ, etc.). -- Hal From halr at voltaire.com Fri Jun 24 06:40:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 09:40:02 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52ekatm13h.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> Message-ID: <1119620402.4480.26.camel@localhost.localdomain> On Thu, 2005-06-23 at 13:31, Roland Dreier wrote: > James> Perhaps a bit of motivation of how the GID->IP service can > James> be used is in order. > > James> kDAPL uses this feature to provide the passive side of a > James> connection with the IP address of the remote peer. kDAPL > James> consumers can use this information as a weak authentication > James> mechanism. > > This seems so weak as to be not useful, and rather expensive to boot. > To implement this, a system receiving a connection request would have > to perform an SA query to map the remote LID back to a GuidInfo > record, and then for each GID attached to the remote LID, I think this part is simpler than this. Aren't the primary/alternate GIDs in the CM REQ ? > somehow > retrieve the set of IP addresses configured for that GID (assuming > that is somehow even possible). > > James> Could SDP make use of this service to validate a connection > James> request's source IP address? > > No, SDP passes the remote peer's IP address directly as part of its > connection establishment. In fact, the SDP annex in the IBA spec > contains this rather enlightening passage: > > IP over InfiniBand does not define a mechanism to perform an > inverse lookup (from an InfiniBand address to an IP address). It > is also possible for a single InfiniBand address to have many IP > addresses, providing a one-to-many mapping when attempting to > perform an inverse lookup. To resolve these issues, the complete > source and destination IP address is provided during connection > setup to enable mapping the destination and source LID/GID to an > IP address at the accepting peer of the connection. DAPL/IBAT does this a different way. DAPL/IBAT uses SA ServiceRecords to handle the inverse mapping. It can handle the many IPs to single GID issue. The only limitation is that the passive side couldn't know unambiguously which of those IPs for that GID was used (when there are multiple IPs). -- Hal From jlentini at netapp.com Fri Jun 24 06:59:12 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 24 Jun 2005 09:59:12 -0400 (EDT) Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52ekatm13h.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> Message-ID: On Thu, 23 Jun 2005, Roland Dreier wrote: > James> Perhaps a bit of motivation of how the GID->IP service can > James> be used is in order. > > James> kDAPL uses this feature to provide the passive side of a > James> connection with the IP address of the remote peer. kDAPL > James> consumers can use this information as a weak authentication > James> mechanism. > > This seems so weak as to be not useful, My understanding is that a full featured NFS implementation needs this capability. My original email didn't state the requirement very well. We don't necessarily need to map the GID to an IP address (although that might be the best thing to do). What we really want is: Given an InfiniBand connection request, the ability to determine an IP address of the source node in an interoperable way. > and rather expensive to boot. Let's defer discussing how to implement it until we agree that it is required. > To implement this, a system receiving a connection request would have > to perform an SA query to map the remote LID back to a GuidInfo > record, and then for each GID attached to the remote LID, somehow > retrieve the set of IP addresses configured for that GID (assuming > that is somehow even possible). > > James> Could SDP make use of this service to validate a connection > James> request's source IP address? > > No, SDP passes the remote peer's IP address directly as part of its > connection establishment. In fact, the SDP annex in the IBA spec > contains this rather enlightening passage: > > IP over InfiniBand does not define a mechanism to perform an > inverse lookup (from an InfiniBand address to an IP address). It > is also possible for a single InfiniBand address to have many IP > addresses, providing a one-to-many mapping when attempting to > perform an inverse lookup. To resolve these issues, the complete > source and destination IP address is provided during connection > setup to enable mapping the destination and source LID/GID to an > IP address at the accepting peer of the connection. > > - R. > From jlentini at netapp.com Fri Jun 24 07:16:45 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 24 Jun 2005 10:16:45 -0400 (EDT) Subject: [openib-general] mapping between IP address and device name In-Reply-To: <000101c5781b$040936d0$9c5aa8c0@infiniconsys.com> References: <000101c5781b$040936d0$9c5aa8c0@infiniconsys.com> Message-ID: On Thu, 23 Jun 2005, Fab Tillier wrote: >> From: Roland Dreier [mailto:roland at topspin.com] >> Sent: Thursday, June 23, 2005 10:32 AM >> >> James> Perhaps a bit of motivation of how the GID->IP service can >> James> be used is in order. >> >> James> kDAPL uses this feature to provide the passive side of a >> James> connection with the IP address of the remote peer. kDAPL >> James> consumers can use this information as a weak authentication >> James> mechanism. >> >> This seems so weak as to be not useful, and rather expensive to boot. >> To implement this, a system receiving a connection request would have >> to perform an SA query to map the remote LID back to a GuidInfo >> record, and then for each GID attached to the remote LID, somehow >> retrieve the set of IP addresses configured for that GID (assuming >> that is somehow even possible). > > This reverse lookup was something that I worked to accommodate in my > proposed changes to expand DAPL ATS to support multiple IP > addresses. The revised DAPL ATS proposal establishes the notion of > a primary IP address that would be used for such validation. > However, I still think the reverse lookup (GID->IP) is weak as there > is no way to tell which IP the source really used. > > IMO it would be much better to put the source and destination > addresses into the CM private data, but this supposedly creates a > wire protocol which the DAT collaborative wants to avoid at all > costs. If we place the IP address into the CM private data it will create interoperability problems. Upper layer protocols like NFS-RDMA, would only be able to communicate with implementations that also placed the address in the CM private data. How would an NFS-RDMA client written directly to the IB verbs layer communicate with an NFS-RDMA server written to kDAPL? The NFS-RDMA client in this configuration would need to place the IP address in the CM private data. In effect, we've added a new (albeit small) component to the NFS-RDMA protocol. Rather than add this mechanism into every ULP, I think it would be better to provide it as a core service of the network. james From Thomas.Talpey at netapp.com Fri Jun 24 07:20:55 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Fri, 24 Jun 2005 10:20:55 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52ekatm13h.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> At 01:31 PM 6/23/2005, Roland Dreier wrote: > James> kDAPL uses this feature to provide the passive side of a > James> connection with the IP address of the remote peer. kDAPL > James> consumers can use this information as a weak authentication > James> mechanism. > >This seems so weak as to be not useful, and rather expensive to boot. >To implement this, a system receiving a connection request would have >to perform an SA query to map the remote LID back to a GuidInfo >record, and then for each GID attached to the remote LID, somehow >retrieve the set of IP addresses configured for that GID (assuming >that is somehow even possible). Yes, it's weak, but it's needed. A good example is the NFS server's "exports" function. For the last 20 or so years, NFS servers have a table which assigns access rights to filesystems by IP address, for example restricting access, making it r/o, etc to certain classes of client. (man exports for gory detail). The NFS daemons inspect the peer address of incoming connections and requests to compare them against this list. When the endpoint is a socket, they can simply use getpeername() and a DNS op. When it's an IB endpoint (without IPoIB or SDP), what can they use? The requirement is that there needs to be a way to track a connection back to a traditional hostname and/or address. Today in the Linux NFS/RDMA work we use ATS to provide the getpeername() function. There are stronger authentication techniques NFS can use of course. But the vast majority of NFS users don't bother and just stuff DNS names into their exports. Replacing these with GIDs is not acceptable (just try asking a sysadmin if he or she wants to put mac addresses in this file!). Tom. From jlentini at netapp.com Fri Jun 24 07:42:03 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 24 Jun 2005 10:42:03 -0400 (EDT) Subject: [openib-general] Re: [openib-commits] r2695 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <1119618180.4477.3372.camel@localhost.localdomain> References: <20050623191815.CA02B2283E1@openib.ca.sandia.gov> <1119618180.4477.3372.camel@localhost.localdomain> Message-ID: On Fri, 24 Jun 2005, Hal Rosenstock wrote: > On Thu, 2005-06-23 at 15:18, jlentini at openib.org wrote: >> Author: jlentini >> Date: 2005-06-23 12:18:14 -0700 (Thu, 23 Jun 2005) >> New Revision: 2695 >> >> Modified: >> gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c >> Log: >> Retry normal packets but not the CM messages. CM message retries are >> linked to dat_ep_connect()'s timeout value > >> Signed-off-by: James Lentini >> >> >> Modified: gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c >> =================================================================== >> -- gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c 2005-06-23 19:15:12 UTC (rev 2694) >> +++ gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c 2005-06-23 19:18:14 UTC (rev 2695) >> @@ -39,10 +39,10 @@ > ... >> #define DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ >> -#define DAPL_IB_MAX_CM_RETRIES 4 >> +#define DAPL_IB_MAX_CM_RETRIES 0 > > Until the algorithm previously discussed is implemented, changing > DAPL_IB_MAX_CM_RETRIES to 0 makes this more frail (in the case of a lost > REQ, etc.). I made this change because this is what we intended to do in the original "timeout implementation" change. I've added an item to the TODO list to account for address resolution time and CM retries in the timeout value. In my testing, the lack of retries hasn't been an issue. If it is, we can boost it up and account for it in the timeout value. From halr at voltaire.com Fri Jun 24 08:08:48 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 11:08:48 -0400 Subject: [openib-general] Re: [openib-commits] r2695 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: References: <20050623191815.CA02B2283E1@openib.ca.sandia.gov> <1119618180.4477.3372.camel@localhost.localdomain> Message-ID: <1119625728.4480.118.camel@localhost.localdomain> On Fri, 2005-06-24 at 10:42, James Lentini wrote: > In my testing, the lack of retries hasn't been an issue. If it is, we > can boost it up and account for it in the timeout value. For what it's worth, I've observed this. I think others have too. It can occur on a noisy network due to cable or other issues. -- Hal From jlentini at netapp.com Fri Jun 24 08:27:57 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 24 Jun 2005 11:27:57 -0400 (EDT) Subject: [openib-general] RE: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to dapl _provider.c In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35D0C@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35D0C@mtvex01.mtv.mtl.com> Message-ID: I think we are in violent agreement. :) If we keep a dapl_hca structure, then it should be consistent managed in its own file for consistency. As we analyze how it is used, there doesn't seem to be a reason to keep it as an independent object. I think grouping the information in dapl_hca, dapl_provider_entry, and the dat_provider into a new "dapl_provider" structure would be a good solution. Keeping a list of the provider's IAs in this structure sounds like a good idea. uDAPL used the dapl_hca structure's ia_list to cleanup IA resources when the user space process forked. When the code was ported to the kernel, the list was retained, but the cleanup wasn't. Suppose a kDAPL consumer (another kernel module) is unloaded and forgets to close an IA. What do we want to have happen when the kDAPL module is unloaded? Don't we want the resources associated with any open IA's cleaned up? james On Fri, 24 Jun 2005, Itamar Rabenstein wrote: > hi James, > > I think that you are worng. > There is no functionalty to this struct. > it is only a struct that hold some data together. > if we look into the files dapl_hca_util.[h.c] > we will find 4 functions: > 1) dapl_hca_alloc -> which is kmalloc and set to 4 data members of this > struct > 2) dapl_hca_free -> only 1 line == kfree > 3) dapl_hca_link_ia -> which is only a call to list_add > 4) dapl_hca_unlink_ia -> which is only a call to list_del > > you have found out that functions 3 & 4 are not needed so we need to delete > them. > > so i dont see why we should have 2 files for kmalloc and kfree > > HCA is not like ia,evd,ep, ... > the others are dapl spec structs so they should be in separete files , > hca is just a name that come from IB it is part of the provider code. > I believe that in other protocols implementaions they will not use "HCA". > > Itamar > >> -----Original Message----- >> From: James Lentini [mailto:jlentini at netapp.com] >> Sent: Thursday, June 23, 2005 11:27 PM >> To: Itamar Rabenstein >> Cc: openib-general >> Subject: Re: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to >> dapl_provider.c >> >> >> >> For consistency, I think we should keep the HCA object in its own >> file. >> >> However, I'm not sure we need an HCA object. Is there a better way to >> organize the data being stored in the dapl_hca structure? I have this >> feeling that we should merge all the structures that we create on a >> per-hca basis (the dapl_provider_entry, dat_provider table, and >> dapl_hca). >> >> james >> >> On Mon, 20 Jun 2005, Itamar Rabenstein wrote: >> >> itamar> ntegrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c >> itamar> (no need for 2 files just for 2 simple function that >> kmalloc and kfree. >> itamar> There is not any special logic in this functions >> that need to separate >> itamar> them into different files) >> itamar> >> itamar> Signed-off-by: Itamar Rabenstein >> itamar> >> itamar> diff -Nurp -X dontdiff dat-provider_link_ia/Makefile >> dat-provider/Makefile >> itamar> --- dat-provider_link_ia/Makefile Sun Jun 19 18:43:16 2005 >> itamar> +++ dat-provider/Makefile Sun Jun 19 19:45:15 2005 >> itamar> @@ -20,7 +20,6 @@ PROVIDER_MODULES := \ >> itamar> dapl_cr \ >> itamar> dapl_ep \ >> itamar> dapl_evd \ >> itamar> - dapl_hca_util \ >> itamar> dapl_ia \ >> itamar> dapl_lmr \ >> itamar> dapl_provider \ >> itamar> diff -Nurp -X dontdiff >> dat-provider_link_ia/dapl_hca_util.c dat-provider/dapl_hca_util.c >> itamar> --- dat-provider_link_ia/dapl_hca_util.c Sun Jun >> 19 18:43:16 2005 >> itamar> +++ dat-provider/dapl_hca_util.c Thu Jan 1 02:00:00 1970 >> itamar> @@ -1,90 +0,0 @@ >> itamar> -/* >> itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. >> All rights reserved. >> itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All >> rights reserved. >> itamar> - * >> itamar> - * This Software is licensed under one of the >> following licenses: >> itamar> - * >> itamar> - * 1) under the terms of the "Common Public License >> 1.0" a copy of which is >> itamar> - * available from the Open Source Initiative, see >> itamar> - * http://www.opensource.org/licenses/cpl.php. >> itamar> - * >> itamar> - * 2) under the terms of the "The BSD License" a >> copy of which is >> itamar> - * available from the Open Source Initiative, see >> itamar> - * http://www.opensource.org/licenses/bsd-license.php. >> itamar> - * >> itamar> - * 3) under the terms of the "GNU General Public >> License (GPL) Version 2" a >> itamar> - * copy of which is available from the Open >> Source Initiative, see >> itamar> - * http://www.opensource.org/licenses/gpl-license.php. >> itamar> - * >> itamar> - * Licensee has the right to choose one of the above >> licenses. >> itamar> - * >> itamar> - * Redistributions of source code must retain the >> above copyright >> itamar> - * notice and one of the license notices. >> itamar> - * >> itamar> - * Redistributions in binary form must reproduce >> both the above copyright >> itamar> - * notice, one of the license notices in the documentation >> itamar> - * and/or other materials provided with the distribution. >> itamar> - */ >> itamar> - >> itamar> -/* >> itamar> - * $Id: dapl_hca_util.c 2640 2005-06-16 16:22:46Z jlentini $ >> itamar> - */ >> itamar> - >> itamar> -#include "dapl.h" >> itamar> -#include "dapl_openib_util.h" >> itamar> -#include "dapl_provider.h" >> itamar> -#include "dapl_hca_util.h" >> itamar> - >> itamar> -/* >> itamar> - * dapl_hca_alloc >> itamar> - * >> itamar> - * alloc and initialize an HCA struct >> itamar> - * >> itamar> - * Input: >> itamar> - * name >> itamar> - * port >> itamar> - * >> itamar> - * Output: >> itamar> - * hca >> itamar> - * >> itamar> - * Returns: >> itamar> - * none >> itamar> - * >> itamar> - */ >> itamar> -struct dapl_hca *dapl_hca_alloc(char *name, struct >> ib_device *device, u8 port) >> itamar> -{ >> itamar> - struct dapl_hca *hca; >> itamar> - int malloc_size = sizeof *hca + strlen(name) + 1; >> itamar> - >> itamar> - hca = kmalloc(malloc_size, GFP_ATOMIC); >> itamar> - if (hca) { >> itamar> - memset(hca, 0, malloc_size); >> itamar> - spin_lock_init(&hca->lock); >> itamar> - INIT_LIST_HEAD(&hca->ia_list); >> itamar> - hca->name = (char *)hca + sizeof *hca; >> itamar> - strcpy(hca->name, name); >> itamar> - hca->ib_hca_handle = device; >> itamar> - hca->port_num = port; >> itamar> - } >> itamar> - return hca; >> itamar> -} >> itamar> - >> itamar> -/* >> itamar> - * dapl_hca_free >> itamar> - * >> itamar> - * free an IA INFO struct >> itamar> - * >> itamar> - * Input: >> itamar> - * hca >> itamar> - * >> itamar> - * Output: >> itamar> - * none >> itamar> - * >> itamar> - * Returns: >> itamar> - * none >> itamar> - * >> itamar> - */ >> itamar> -void dapl_hca_free(struct dapl_hca *hca) >> itamar> -{ >> itamar> - kfree(hca); >> itamar> -} >> itamar> diff -Nurp -X dontdiff >> dat-provider_link_ia/dapl_hca_util.h dat-provider/dapl_hca_util.h >> itamar> --- dat-provider_link_ia/dapl_hca_util.h Sun Jun >> 19 18:43:16 2005 >> itamar> +++ dat-provider/dapl_hca_util.h Thu Jan 1 02:00:00 1970 >> itamar> @@ -1,42 +0,0 @@ >> itamar> -/* >> itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. >> All rights reserved. >> itamar> - * Copyright (c) 2005 Sun Microsystems, Inc. All >> rights reserved. >> itamar> - * >> itamar> - * This Software is licensed under one of the >> following licenses: >> itamar> - * >> itamar> - * 1) under the terms of the "Common Public License >> 1.0" a copy of which is >> itamar> - * available from the Open Source Initiative, see >> itamar> - * http://www.opensource.org/licenses/cpl.php. >> itamar> - * >> itamar> - * 2) under the terms of the "The BSD License" a >> copy of which is >> itamar> - * available from the Open Source Initiative, see >> itamar> - * http://www.opensource.org/licenses/bsd-license.php. >> itamar> - * >> itamar> - * 3) under the terms of the "GNU General Public >> License (GPL) Version 2" a >> itamar> - * copy of which is available from the Open >> Source Initiative, see >> itamar> - * http://www.opensource.org/licenses/gpl-license.php. >> itamar> - * >> itamar> - * Licensee has the right to choose one of the above >> licenses. >> itamar> - * >> itamar> - * Redistributions of source code must retain the >> above copyright >> itamar> - * notice and one of the license notices. >> itamar> - * >> itamar> - * Redistributions in binary form must reproduce >> both the above copyright >> itamar> - * notice, one of the license notices in the documentation >> itamar> - * and/or other materials provided with the distribution. >> itamar> - */ >> itamar> - >> itamar> -/* >> itamar> - * $Id: dapl_hca_util.h 2640 2005-06-16 16:22:46Z jlentini $ >> itamar> - */ >> itamar> - >> itamar> -#ifndef DAPL_HCA_UTIL_H >> itamar> -#define DAPL_HCA_UTIL_H >> itamar> - >> itamar> -#include "dapl.h" >> itamar> - >> itamar> -struct dapl_hca *dapl_hca_alloc(char *name, struct >> ib_device *device, u8 port); >> itamar> - >> itamar> -void dapl_hca_free(struct dapl_hca *hca); >> itamar> - >> itamar> -#endif >> itamar> diff -Nurp -X dontdiff >> dat-provider_link_ia/dapl_provider.c dat-provider/dapl_provider.c >> itamar> --- dat-provider_link_ia/dapl_provider.c Sun Jun >> 19 18:43:16 2005 >> itamar> +++ dat-provider/dapl_provider.c Sun Jun 19 19:47:16 2005 >> itamar> @@ -35,7 +35,6 @@ >> itamar> #include >> itamar> >> itamar> #include "dapl.h" >> itamar> -#include "dapl_hca_util.h" >> itamar> #include "dapl_provider.h" >> itamar> #include "dapl_util.h" >> itamar> #include "dapl_openib_util.h" >> itamar> @@ -246,6 +245,24 @@ static void dapl_provider_info_init(stru >> itamar> provider_info->ia_name[i+1] = '\0'; >> itamar> } >> itamar> >> itamar> +static struct dapl_hca *dapl_hca_alloc(char *name, >> struct ib_device *device, u8 port) >> itamar> +{ >> itamar> + struct dapl_hca *hca; >> itamar> + int malloc_size = sizeof *hca + strlen(name) + 1; >> itamar> + >> itamar> + hca = kmalloc(malloc_size, GFP_ATOMIC); >> itamar> + if (hca) { >> itamar> + memset(hca, 0, malloc_size); >> itamar> + spin_lock_init(&hca->lock); >> itamar> + INIT_LIST_HEAD(&hca->ia_list); >> itamar> + hca->name = (char *)hca + sizeof *hca; >> itamar> + strcpy(hca->name, name); >> itamar> + hca->ib_hca_handle = device; >> itamar> + hca->port_num = port; >> itamar> + } >> itamar> + return hca; >> itamar> +} >> itamar> + >> itamar> static void dapl_add_port(struct ib_device *device, u8 port) >> itamar> { >> itamar> struct dat_provider_info provider_info; >> itamar> @@ -305,7 +322,7 @@ error: >> itamar> >> (void)dapl_provider_list_remove(provider_info.ia_name); >> itamar> >> itamar> if (NULL != hca) >> itamar> - dapl_hca_free(hca); >> itamar> + kfree(hca); >> itamar> } >> itamar> } >> itamar> >> itamar> @@ -338,7 +355,7 @@ static void dapl_remove_port(struct ib_d >> itamar> provider_info.ia_name); >> itamar> } >> itamar> >> itamar> - dapl_hca_free(provider->extension); >> itamar> + kfree(provider->extension); >> itamar> >> itamar> dapl_provider_list_remove(provider_info.ia_name); >> itamar> } >> itamar> -- >> itamar> Itamar >> itamar> >> > From roland at topspin.com Fri Jun 24 08:58:23 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 24 Jun 2005 08:58:23 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <1119620402.4480.26.camel@localhost.localdomain> (Hal Rosenstock's message of "24 Jun 2005 09:40:02 -0400") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <1119620402.4480.26.camel@localhost.localdomain> Message-ID: <52ekarkaqo.fsf@topspin.com> Hal> I think this part is simpler than this. Aren't the Hal> primary/alternate GIDs in the CM REQ ? Yes, but the remote peer could lie, right? - R. From tduffy at sun.com Fri Jun 24 08:58:26 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 24 Jun 2005 08:58:26 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: remove dat_os_realloc() In-Reply-To: References: <1119568476.21702.13.camel@duffman> Message-ID: <1119628706.8022.3.camel@duffman> On Thu, 2005-06-23 at 22:37 -0400, James Lentini wrote: > What do you think about doing it this way? (see attached) Yeah, I noticed you did the subtract after kfree but figured the pointer value would not be change, just that the data it pointed to would be (potentially) gone. Go with your patch. It is cleaner. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Fri Jun 24 09:08:57 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 12:08:57 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52ekarkaqo.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <1119620402.4480.26.camel@localhost.localdomain> <52ekarkaqo.fsf@topspin.com> Message-ID: <1119629140.14146.1.camel@localhost.localdomain> On Fri, 2005-06-24 at 11:58, Roland Dreier wrote: > Hal> I think this part is simpler than this. Aren't the > Hal> primary/alternate GIDs in the CM REQ ? > > Yes, but the remote peer could lie, right? So there needs to be one more SA lookup to validate that the GID is associated with the LID. Same issue for APM too. -- Hal From roland at topspin.com Fri Jun 24 09:19:25 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 24 Jun 2005 09:19:25 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> (Thomas Talpey's message of "Fri, 24 Jun 2005 10:20:55 -0400") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> Message-ID: <524qbnk9rm.fsf@topspin.com> Thomas> Yes, it's weak, but it's needed. A good example is the NFS Thomas> server's "exports" function. For the last 20 or so years, Thomas> NFS servers have a table which assigns access rights to Thomas> filesystems by IP address, for example restricting access, Thomas> making it r/o, etc to certain classes of client. (man Thomas> exports for gory detail). Sure, I understand why NFS/RDMA wants the peer address. However, forcing this into kDAPL and then making kDAPL go through contortions to provide it seems like the wrong way around. We end up with gross hacks like ATS, which only works for IPv4 and doesn't support multiple IP addresses on a single interface. It seems far preferable to me to just define the wire protocol of NFS/RDMA for IB such that a client passes its IP address as part of the connection request. This scheme was used for SDP to avoid precisely the complications that we're discussing now. - R. From halr at voltaire.com Fri Jun 24 09:28:19 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 12:28:19 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <524qbnk9rm.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> Message-ID: <1119630499.4389.5.camel@hal.voltaire.com> On Fri, 2005-06-24 at 12:19, Roland Dreier wrote: > Sure, I understand why NFS/RDMA wants the peer address. However, > forcing this into kDAPL and then making kDAPL go through contortions > to provide it seems like the wrong way around. We end up with gross > hacks like ATS, which only works for IPv4 and doesn't support multiple > IP addresses on a single interface. Why do you say ATS only works for IPv4 and doesn't support multiple IP addresses on a single interface ? It certainly does the later. Both IPv4 and v6 addresses are accomodated in the ATS SR definition. Any IPv4ness of the some ATS APIs (and the underlying implementation) need to be enhanced for IPv6. That is on the TODO list. -- Hal From Thomas.Talpey at netapp.com Fri Jun 24 09:34:41 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Fri, 24 Jun 2005 12:34:41 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <524qbnk9rm.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> At 12:19 PM 6/24/2005, Roland Dreier wrote: >It seems far preferable to me to just define the wire protocol of >NFS/RDMA for IB such that a client passes its IP address as part of >the connection request. This scheme was used for SDP to avoid >precisely the complications that we're discussing now. But that's totally and completely insecure. The goal of /etc/exports is to place at least part of the client authentication in the network rather than the supplied credentials. NFS has quite enough of a history with AUTH_SYS to prove the issues there. Some of the exports options (e.g. the *_squash ones) are specifically because of this. I don't care about ATS either, by the way. I'm looking for an interoperable alternative. Tom. From roland at topspin.com Fri Jun 24 09:42:37 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 24 Jun 2005 09:42:37 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> (Thomas Talpey's message of "Fri, 24 Jun 2005 12:34:41 -0400") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> Message-ID: <52vf43iu4i.fsf@topspin.com> Thomas> But that's totally and completely insecure. The goal of Thomas> /etc/exports is to place at least part of the client Thomas> authentication in the network rather than the supplied Thomas> credentials. NFS has quite enough of a history with Thomas> AUTH_SYS to prove the issues there. Some of the exports Thomas> options (e.g. the *_squash ones) are specifically because Thomas> of this. ATS is completely insecure too, right? A client can create any old service record in the subnet administrator's database and claim that its GID has whatever IP address it wants. - R. From jlentini at netapp.com Fri Jun 24 09:45:22 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 24 Jun 2005 12:45:22 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove dat_os_realloc() In-Reply-To: <1119628706.8022.3.camel@duffman> References: <1119568476.21702.13.camel@duffman> <1119628706.8022.3.camel@duffman> Message-ID: Committed in revision 2707. On Fri, 24 Jun 2005, Tom Duffy wrote: tduffy> On Thu, 2005-06-23 at 22:37 -0400, James Lentini wrote: tduffy> > What do you think about doing it this way? (see attached) tduffy> tduffy> Yeah, I noticed you did the subtract after kfree but figured the pointer tduffy> value would not be change, just that the data it pointed to would be tduffy> (potentially) gone. tduffy> tduffy> Go with your patch. It is cleaner. tduffy> tduffy> -tduffy tduffy> From roland.list at gmail.com Fri Jun 24 10:00:56 2005 From: roland.list at gmail.com (Roland Dreier) Date: Fri, 24 Jun 2005 10:00:56 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <1119630499.4389.5.camel@hal.voltaire.com> References: <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <1119630499.4389.5.camel@hal.voltaire.com> Message-ID: > Why do you say ATS only works for IPv4 and doesn't support multiple IP > addresses on a single interface ? It certainly does the later. Both IPv4 > and v6 addresses are accomodated in the ATS SR definition. Any IPv4ness > of the some ATS APIs (and the underlying implementation) need to be > enhanced for IPv6. That is on the TODO list. Sorry, I haven't looked at ATS for a while. I guess I remembered incorrectly. - R. From jay.rosser at hp.com Fri Jun 24 10:02:51 2005 From: jay.rosser at hp.com (Jay Rosser) Date: Fri, 24 Jun 2005 10:02:51 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> Message-ID: <42BC3CBB.6030909@hp.com> On the subject of NFS/RDMA, what is the IB ServiceID space that is used? If I recall correctly, I have seen simply the value 2049 (i.e. the standard TCP/UDP port number) used in some implementations (i.e. 00 00 00 00 00 00 20 49). Is there a mapping onto an IB ServiceID defined? Thanks, Jay >At 12:19 PM 6/24/2005, Roland Dreier wrote: > > >>It seems far preferable to me to just define the wire protocol of >>NFS/RDMA for IB such that a client passes its IP address as part of >>the connection request. This scheme was used for SDP to avoid >>precisely the complications that we're discussing now. >> >> > >But that's totally and completely insecure. The goal of /etc/exports >is to place at least part of the client authentication in the network >rather than the supplied credentials. NFS has quite enough of a >history with AUTH_SYS to prove the issues there. Some of the >exports options (e.g. the *_squash ones) are specifically because >of this. > >I don't care about ATS either, by the way. I'm looking for an >interoperable alternative. > >Tom. >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > > -- jay.rosser at hp.com 408-447-3175 From caitlin.bestler at gmail.com Fri Jun 24 10:05:49 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Fri, 24 Jun 2005 10:05:49 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52vf43iu4i.fsf@topspin.com> References: <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> Message-ID: <469958e0050624100536b89306@mail.gmail.com> The presumption behind authenticating by remote address is that the local subnet is sufficiently administered so as to prevent address spoofing. That would require firewalls, configured switches and other techniques for an IP network, and for IB ensuring that the IB subnet administration is not simply accessible to anyone. I am certain that somewhere in the NFS specs it clearly states that the application SHOULD NOT use AUTH_SYS unless these sorts of assurances are in place. Right. But it's not our place at the transport layer to tell the application that their security policy is flawed. We have existing applications that want a somewhat authenticated remote IP address delivered to them. Actual use of the wire protocol address *will* be required to support iWARP. The question is what InfiniBand should do. The DAT specification was deliberately written so that the actual GID can be reported here *if* the GID can also be used to connect to. That suggests that address translation is only really needed for IPv4 addresses. Tom can comment on whether an IPv6 format address meets the needs of NFS. It obviously SHOULD, but he'd no better than I do on how common IPv6 format support is in NFS code. On 6/24/05, Roland Dreier wrote: > Thomas> But that's totally and completely insecure. The goal of > Thomas> /etc/exports is to place at least part of the client > Thomas> authentication in the network rather than the supplied > Thomas> credentials. NFS has quite enough of a history with > Thomas> AUTH_SYS to prove the issues there. Some of the exports > Thomas> options (e.g. the *_squash ones) are specifically because > Thomas> of this. > > ATS is completely insecure too, right? A client can create any old > service record in the subnet administrator's database and claim that > its GID has whatever IP address it wants. > > - R. > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From halr at voltaire.com Fri Jun 24 10:13:12 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 13:13:12 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52vf43iu4i.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> Message-ID: <1119633191.4388.17.camel@hal.voltaire.com> On Fri, 2005-06-24 at 12:42, Roland Dreier wrote: > Thomas> But that's totally and completely insecure. The goal of > Thomas> /etc/exports is to place at least part of the client > Thomas> authentication in the network rather than the supplied > Thomas> credentials. NFS has quite enough of a history with > Thomas> AUTH_SYS to prove the issues there. Some of the exports > Thomas> options (e.g. the *_squash ones) are specifically because > Thomas> of this. > > ATS is completely insecure too, right? A client can create any old > service record in the subnet administrator's database and claim that > its GID has whatever IP address it wants. The first level of IB trust in terms of the SA (authenticaing the requestor) is restrictions based on access (partitioning). This is true for a number of SA attributes which is more than (just) ServiceRecords. But we do trust the kernel, right ? So the only issue would be user space creation and deletion of these records. There could be checking in the kernel as to the registration/deregistration being appropriate for the configuration. I think this is the same issue whatever way it is done (whether by SRs or CM private data). -- Hal From Thomas.Talpey at netapp.com Fri Jun 24 10:20:01 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Fri, 24 Jun 2005 13:20:01 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <42BC3CBB.6030909@hp.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <42BC3CBB.6030909@hp.com> Message-ID: <6.2.1.2.2.20050624131145.060437b0@exnane01.nane.netapp.com> At 01:02 PM 6/24/2005, Jay Rosser wrote: >On the subject of NFS/RDMA, what is the IB ServiceID space that is used? >If I recall correctly, I have seen simply the value 2049 (i.e. the >standard TCP/UDP port number) used in some implementations (i.e. 00 00 >00 00 00 00 20 49). Is there a mapping onto an IB ServiceID defined? We aren't currently using the portmapper to discover the serviceid that the NFS/RDMA server is listening on. Brent Callaghan chose serviceid 2049 as a convenience in Sun's first implementation, and so far it has stuck. Theoretically the server can listen on any endpoint it chooses, this is how NFS/TCP and NFS/UDP work. But typically all servers use the well known port. It's probably a good idea to define a better default mapping. Tom. From rolandd at cisco.com Fri Jun 24 10:22:33 2005 From: rolandd at cisco.com (Roland Dreier) Date: Fri, 24 Jun 2005 10:22:33 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <1119633191.4388.17.camel@hal.voltaire.com> (Hal Rosenstock's message of "24 Jun 2005 13:13:12 -0400") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <1119633191.4388.17.camel@hal.voltaire.com> Message-ID: <527jgjis9y.fsf@topspin.com> Hal> The first level of IB trust in terms of the SA (authenticaing Hal> the requestor) is restrictions based on access Hal> (partitioning). This is true for a number of SA attributes Hal> which is more than (just) ServiceRecords. Right, but at least for now the SA has no way of checking the IP address in a request to decide whether or not it should allow creating an ATS record. Hal> But we do trust the kernel, right ? No, an NFS server can't trust anything coming from a remote client. - R. From Thomas.Talpey at netapp.com Fri Jun 24 10:22:44 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Fri, 24 Jun 2005 13:22:44 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52vf43iu4i.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050624132030.06040730@exnane01.nane.netapp.com> At 12:42 PM 6/24/2005, Roland Dreier wrote: > Thomas> But that's totally and completely insecure. The goal of > Thomas> /etc/exports is to place at least part of the client > Thomas> authentication in the network rather than the supplied > Thomas> credentials. NFS has quite enough of a history with > Thomas> AUTH_SYS to prove the issues there. Some of the exports > Thomas> options (e.g. the *_squash ones) are specifically because > Thomas> of this. > >ATS is completely insecure too, right? A client can create any old >service record in the subnet administrator's database and claim that >its GID has whatever IP address it wants. As I said - I am not attached to ATS. I would welcome an alternative. But in the absence of one, I like what we have. Also, I do not want to saddle the NFS/RDMA transport with carrying an IP address purely for the benefit of a missing transport facility. After all NFS/RDMA works on iWARP too. Tom. From rolandd at cisco.com Fri Jun 24 10:24:27 2005 From: rolandd at cisco.com (Roland Dreier) Date: Fri, 24 Jun 2005 10:24:27 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050624131145.060437b0@exnane01.nane.netapp.com> (Thomas Talpey's message of "Fri, 24 Jun 2005 13:20:01 -0400") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <42BC3CBB.6030909@hp.com> <6.2.1.2.2.20050624131145.060437b0@exnane01.nane.netapp.com> Message-ID: <521x6ris6s.fsf@topspin.com> Thomas> We aren't currently using the portmapper to discover the Thomas> serviceid that the NFS/RDMA server is listening on. Brent Thomas> Callaghan chose serviceid 2049 as a convenience in Sun's Thomas> first implementation, and so far it has stuck. Ugh, according to the IBA spec a service ID beginning with 0x00 belongs to the IBTA-administered range. This should probably be fixed before it gets to entrenched. - R. From Thomas.Talpey at netapp.com Fri Jun 24 10:27:59 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Fri, 24 Jun 2005 13:27:59 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <527jgjis9y.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <1119633191.4388.17.camel@hal.voltaire.com> <527jgjis9y.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050624132453.042786c0@exnane01.nane.netapp.com> At 01:22 PM 6/24/2005, Roland Dreier wrote: > Hal> But we do trust the kernel, right ? > >No, an NFS server can't trust anything coming from a remote client. Well, the server can't trust untrusted information coming from the client. NFS has many forms of strong authentication. But many, many users ignore it and simply apply careful administration in its place. I just don't want to go backwards when introducing NFS/RDMA. If I have to tell someone "well it works just like regular NFS except for..." guess what I'll hear. Oh and by the way, iWARP won't have this issue. Tom. From halr at voltaire.com Fri Jun 24 10:25:43 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 13:25:43 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <527jgjis9y.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <1119633191.4388.17.camel@hal.voltaire.com> <527jgjis9y.fsf@topspin.com> Message-ID: <1119633943.4388.26.camel@hal.voltaire.com> On Fri, 2005-06-24 at 13:22, Roland Dreier wrote: > Hal> The first level of IB trust in terms of the SA (authenticaing > Hal> the requestor) is restrictions based on access > Hal> (partitioning). This is true for a number of SA attributes > Hal> which is more than (just) ServiceRecords. > > Right, but at least for now the SA has no way of checking the IP > address in a request to decide whether or not it should allow creating > an ATS record. In fact, the SA does not know it is an IP address in the ServiceData of the ServiceRecord. -- Hal From rolandd at cisco.com Fri Jun 24 10:30:38 2005 From: rolandd at cisco.com (Roland Dreier) Date: Fri, 24 Jun 2005 10:30:38 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050624132030.06040730@exnane01.nane.netapp.com> (Thomas Talpey's message of "Fri, 24 Jun 2005 13:22:44 -0400") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <6.2.1.2.2.20050624132030.06040730@exnane01.nane.netapp.com> Message-ID: <52vf43hdc1.fsf@topspin.com> Thomas> As I said - I am not attached to ATS. I would welcome an Thomas> alternative. Sure, understood. I'm suggesting a slight tweak to the IB wire protocol. I don't think there's a difference in the security provided, and carrying the peer address in the CM private data avoids a lot of the conceptual and implementation difficulties of ATS. Thomas> But in the absence of one, I like what we have. Also, I do Thomas> not want to saddle the NFS/RDMA transport with carrying an Thomas> IP address purely for the benefit of a missing transport Thomas> facility. After all NFS/RDMA works on iWARP too. I'm not sure I understand this objection. We wouldn't be saddling the transport with anything -- simply specifying in the binding of NFS/RDMA to IB that certain information is carried in the private data fields of the CM messages used to establish a connection. Clearly iWARP would use its own mechanism for providing the peer address. This would be exactly analogous to the situation for SDP -- obviously SDP running on iWARP does not use the IB CM to exchange IP address information in the same way the SDP over IB does. - R. From jay.rosser at hp.com Fri Jun 24 10:34:54 2005 From: jay.rosser at hp.com (Jay Rosser) Date: Fri, 24 Jun 2005 10:34:54 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050624131145.060437b0@exnane01.nane.netapp.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <42BC3CBB.6030909@hp.com> <6.2.1.2.2.20050624131145.060437b0@exnane01.nane.netapp.com> Message-ID: <42BC443E.6040802@hp.com> >>On the subject of NFS/RDMA, what is the IB ServiceID space that is used? >>If I recall correctly, I have seen simply the value 2049 (i.e. the >>standard TCP/UDP port number) used in some implementations (i.e. 00 00 >>00 00 00 00 20 49). Is there a mapping onto an IB ServiceID defined? >> >> > >We aren't currently using the portmapper to discover the serviceid that >the NFS/RDMA server is listening on. Brent Callaghan chose serviceid 2049 >as a convenience in Sun's first implementation, and so far it has stuck. > >Theoretically the server can listen on any endpoint it chooses, this is >how NFS/TCP and NFS/UDP work. But typically all servers use the well >known port. It's probably a good idea to define a better default mapping. > > Using a well-known port seems okay to me. However, if I understand correctly, the 2049 ServiceId is in the IBTA's ServiceID space (i.e. for IBTA-defined well-known ServiceIDs - see IB1.2 Vol1 pg 1181 A3.2.3.1). It would probably make sense to use the IETF ServiceID space (A3.2.3.2) but that would require someone to administer it. Thanks, Jay >Tom. > > > -- jay.rosser at hp.com 408-447-3175 From rolandd at cisco.com Fri Jun 24 10:35:29 2005 From: rolandd at cisco.com (Roland Dreier) Date: Fri, 24 Jun 2005 10:35:29 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <1119633943.4388.26.camel@hal.voltaire.com> (Hal Rosenstock's message of "24 Jun 2005 13:25:43 -0400") References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <1119633191.4388.17.camel@hal.voltaire.com> <527jgjis9y.fsf@topspin.com> <1119633943.4388.26.camel@hal.voltaire.com> Message-ID: <52ll4zhd3y.fsf@topspin.com> Roland> Right, but at least for now the SA has no way of checking Roland> the IP address in a request to decide whether or not it Roland> should allow creating an ATS record. Hal> In fact, the SA does not know it is an IP address in the Hal> ServiceData of the ServiceRecord. Right, which means that for an NFS server, looking up a remote peer's ATS record and checking against an exports file provides zero security. The remote peer can put any IP address it wants for itself into the SA's database. For ATS to be useful in this setting, the SA needs to know about ATS records and have some way of checking the IP addresses they contain. And configuring that seems likely to be quite painful. - R. From Thomas.Talpey at netapp.com Fri Jun 24 10:51:26 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Fri, 24 Jun 2005 13:51:26 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52vf43hdc1.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <6.2.1.2.2.20050624132030.06040730@exnane01.nane.netapp.com> <52vf43hdc1.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050624134131.05c7b0e0@exnane01.nane.netapp.com> At 01:30 PM 6/24/2005, Roland Dreier wrote: > Thomas> But in the absence of one, I like what we have. Also, I do > Thomas> not want to saddle the NFS/RDMA transport with carrying an > Thomas> IP address purely for the benefit of a missing transport > Thomas> facility. After all NFS/RDMA works on iWARP too. > >I'm not sure I understand this objection. We wouldn't be saddling the >transport with anything -- simply specifying in the binding of >NFS/RDMA to IB that certain information is carried in the private data >fields of the CM messages used to establish a connection. Clearly >iWARP would use its own mechanism for providing the peer address. > >This would be exactly analogous to the situation for SDP -- obviously >SDP running on iWARP does not use the IB CM to exchange IP address >information in the same way the SDP over IB does. Oh - I thought you meant that NFS/RDMA should have a HELLO message carrying an IP address, like SDP/IB. That's a nonstarter for the reason I mentioned, plus the fact that it links this state to the connection, which might break and require reconnect. In fact, NFSv4 and our Sessions proposal addresses this, but it doesn't help NFSv3, which is the predominant use today. On the other hand, placing a mandatory content in the CM exchange brings in a whole different raft of interoperability questions, as James mentioned earlier. For better or for worse, the ATS approach is easily administered and does not impact any protocol layers outside of its own. I think of it as ARP for IB. Tom. From caitlinb at siliquent.com Fri Jun 24 11:11:48 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 24 Jun 2005 11:11:48 -0700 Subject: [openib-general] ATS and DAT "IA Address" / Connection Establishment Requirements Message-ID: <8508251A6FC08A489844A94261D3693A057B87@fiona.siliquent.com> I think it should be made clear that ATS is *not* a required part of DAT. It is merely a commonly adopted solution. I am attaching a document prepared for the DAT Collaborative while the definition of IA Address was being resolved. A quick recap on the requirements of the "IA Address" are as follows: 1) It walks, talks and quacks like an IPv6 address -- but nothing says it actually has to be one. 2) IPv6 Addresses already include IPv4. 3) A Connection Request reports the local address that was requested, and the remote address. In an IP network those are expected to the be actual addresses from the IP header, subject to all authentication features the OS and/or local subnet offers. It is not a value that can be made up by a remote user at will with no risk of detection. 4) Generally, it should be usable to set up a reverse connection. The generally is mostly a caveat about IP firewalls and PNAT. The important point here is that it is totally valid to identify the remote endpoint with a GID. ATS was developed largely because the sourceforge reference implementation could not rely upon the host OS providing DNS support for IPv6 format addresses. That is not an issue here. Therefore the reported address *could* be the actual remote GID, as long as that would be accepted in a dat_ep_connect() call to go in the opposite direction. Delivering the GID would provide an address that was just as authenticated as an IP Address, and therefore be just as good for the intended purpose of providing authentication when the application trusts the local network administrator. Tunneling alleged IP Addresses supplied in user-mode that are invisible to the network administrator does not achieve that goal. I also believe that ATS achieves that goal, but I'm not an expert on IB subnet administration. But wouldn't entry of invalid data to the ATS database be at least visisble to the network administrator? -------------- next part -------------- A non-text attachment was scrubbed... Name: IPv6_andor_GID.pdf Type: application/octet-stream Size: 43166 bytes Desc: IPv6_andor_GID.pdf URL: From halr at voltaire.com Fri Jun 24 11:27:30 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 14:27:30 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050624134131.05c7b0e0@exnane01.nane.netapp.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <6.2.1.2.2.20050624132030.06040730@exnane01.nane.netapp.com> <52vf43hdc1.fsf@topspin.com> <6.2.1.2.2.20050624134131.05c7b0e0@exnane01.nane.netapp.com> Message-ID: <1119636743.4388.116.camel@hal.voltaire.com> On Fri, 2005-06-24 at 13:51, Talpey, Thomas wrote: > On the other hand, placing a mandatory content in the CM exchange > brings in a whole different raft of interoperability questions, as James > mentioned earlier. For better or for worse, the ATS approach is easily > administered and does not impact any protocol layers outside of its > own. I think of it as ARP for IB. reverse ARP for IB From Thomas.Talpey at netapp.com Fri Jun 24 12:14:37 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Fri, 24 Jun 2005 15:14:37 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <1119636743.4388.116.camel@hal.voltaire.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <6.2.1.2.2.20050624132030.06040730@exnane01.nane.netapp.com> <52vf43hdc1.fsf@topspin.com> <6.2.1.2.2.20050624134131.05c7b0e0@exnane01.nane.netapp.com> <1119636743.4388.116.camel@hal.voltaire.com> Message-ID: <6.2.1.2.2.20050624143638.0575e090@exnane01.nane.netapp.com> At 02:27 PM 6/24/2005, Hal Rosenstock wrote: >On Fri, 2005-06-24 at 13:51, Talpey, Thomas wrote: >> mentioned earlier. For better or for worse, the ATS approach is easily >> administered and does not impact any protocol layers outside of its >> own. I think of it as ARP for IB. > >reverse ARP for IB Actually, I did mean ARP. It's only necessary to answer the questions "please send this to address foo" (for making connections), along with "what address sent this request" (for accepting connections and export checking). There is no need for RARP, which would answer the question "who is supposed to belong to this MAC address". In fact, I don't want to know the answer to that! :-) That's link layer stuff, three floors down. Tom. From halr at voltaire.com Fri Jun 24 12:30:54 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 15:30:54 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050624143638.0575e090@exnane01.nane.netapp.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <6.2.1.2.2.20050624132030.06040730@exnane01.nane.netapp.com> <52vf43hdc1.fsf@topspin.com> <6.2.1.2.2.20050624134131.05c7b0e0@exnane01.nane.netapp.com> <1119636743.4388.116.camel@hal.voltaire.com> <6.2.1.2.2.20050624143638.0575e090@exnane01.nane.netapp.com> Message-ID: <1119641380.4347.23.camel@hal.voltaire.com> On Fri, 2005-06-24 at 15:14, Talpey, Thomas wrote: > At 02:27 PM 6/24/2005, Hal Rosenstock wrote: > >On Fri, 2005-06-24 at 13:51, Talpey, Thomas wrote: > >> mentioned earlier. For better or for worse, the ATS approach is easily > >> administered and does not impact any protocol layers outside of its > >> own. I think of it as ARP for IB. > > > >reverse ARP for IB > > Actually, I did mean ARP. It's only necessary to answer the questions > "please send this to address foo" (for making connections), This is the ARP side (get the link layer address (GID + QPN) for an IP address) I'm connecting to. > along with > "what address sent this request" (for accepting connections and export > checking). I think this is the RARP side (get the IP address that corresponds to a partial link layer address (GID)). This was the side being discussed on the list, right ? Do I have this backwards ? > There is no need for RARP, which would answer the question "who is > supposed to belong to this MAC address". In fact, I don't want to know > the answer to that! :-) That's link layer stuff, three floors down. I think we are just discussing terminology. Perhaps I should have said reverse ARP for IP (not IB). You are calling that ARP for IB. Doesn't RARP say for a given MAC (hardware) address give me the IP address and that's exactly what ATS is doing (where the MAC address is equivalent to the GID) in the case under discussion ? ARP is the other way 'round: for an IP address give me the link layer address. -- Hal From mamidala at cse.ohio-state.edu Fri Jun 24 12:55:11 2005 From: mamidala at cse.ohio-state.edu (amith rajith mamidala) Date: Fri, 24 Jun 2005 15:55:11 -0400 (EDT) Subject: [openib-general] Error destroying CQ Message-ID: Hi, I am getting an error while destroying the CQ. I am not able to figure out the reason why I am getting this error. I printed out the error code and it is 16. Is there anyway by which I can capture the error message and not just the code ? Thanks, Amith From rolandd at cisco.com Fri Jun 24 13:05:10 2005 From: rolandd at cisco.com (Roland Dreier) Date: Fri, 24 Jun 2005 13:05:10 -0700 Subject: [openib-general] Error destroying CQ In-Reply-To: (amith rajith mamidala's message of "Fri, 24 Jun 2005 15:55:11 -0400 (EDT)") References: Message-ID: <52wtojed1l.fsf@topspin.com> amith> I am getting an error while destroying the CQ. I am not amith> able to figure out the reason why I am getting this amith> error. I printed out the error code and it is 16. Is there amith> anyway by which I can capture the error message and not amith> just the code ? There isn't really an error message. However error code 16 is EBUSY, which would seem to indicate that your CQ still has work queues associated to it. Are you sure you've destroyed all the QPs that are using the CQ? - R. From rolandd at cisco.com Fri Jun 24 14:54:42 2005 From: rolandd at cisco.com (Roland Dreier) Date: Fri, 24 Jun 2005 14:54:42 -0700 Subject: [openib-general] [PATCH 1/3] IB: Fix race in sa_query Message-ID: <20056241454.JSnV6qzt9RST2IRw@cisco.com> Use a copy of the id we'll return to the consumer so that we don't dereference query->sa_query after calling send_mad(). A completion may occur very quickly and end up freeing the query before we get to do anything after send_mad(). Signed-off-by: Roland Dreier --- drivers/infiniband/core/sa_query.c | 18 +++++++++++++----- 1 files changed, 13 insertions(+), 5 deletions(-) --- linux-export2.orig/drivers/infiniband/core/sa_query.c 2005-06-23 13:16:25.000000000 -0700 +++ linux-export2/drivers/infiniband/core/sa_query.c 2005-06-24 14:49:43.592455970 -0700 @@ -507,7 +507,13 @@ spin_unlock_irqrestore(&idr_lock, flags); } - return ret; + /* + * It's not safe to dereference query any more, because the + * send may already have completed and freed the query in + * another context. So use wr.wr_id, which has a copy of the + * query's id. + */ + return ret ? ret : wr.wr_id; } static void ib_sa_path_rec_callback(struct ib_sa_query *sa_query, @@ -598,14 +604,15 @@ rec, query->sa_query.mad->data); *sa_query = &query->sa_query; + ret = send_mad(&query->sa_query, timeout_ms); - if (ret) { + if (ret < 0) { *sa_query = NULL; kfree(query->sa_query.mad); kfree(query); } - return ret ? ret : query->sa_query.id; + return ret; } EXPORT_SYMBOL(ib_sa_path_rec_get); @@ -674,14 +681,15 @@ rec, query->sa_query.mad->data); *sa_query = &query->sa_query; + ret = send_mad(&query->sa_query, timeout_ms); - if (ret) { + if (ret < 0) { *sa_query = NULL; kfree(query->sa_query.mad); kfree(query); } - return ret ? ret : query->sa_query.id; + return ret; } EXPORT_SYMBOL(ib_sa_mcmember_rec_query); From rolandd at cisco.com Fri Jun 24 14:54:42 2005 From: rolandd at cisco.com (Roland Dreier) Date: Fri, 24 Jun 2005 14:54:42 -0700 Subject: [openib-general] [PATCH 3/3] MAINTAINERS: Update Roland Dreier's email In-Reply-To: <20056241454.C4xXsKbhkkqxG17N@cisco.com> Message-ID: <20056241454.37h1OxG0D6Q1DrAv@cisco.com> Cisco bought Topspin, so I'm now a shiny happy Cisco employee. Update my entry in MAINTAINERS. Signed-off-by: Roland Dreier --- linux-export2.orig/MAINTAINERS 2005-06-23 13:16:09.000000000 -0700 +++ linux-export2/MAINTAINERS 2005-06-24 14:50:27.932893628 -0700 @@ -1150,7 +1150,7 @@ INFINIBAND SUBSYSTEM P: Roland Dreier -M: roland at topspin.com +M: rolandd at cisco.com P: Sean Hefty M: mshefty at ichips.intel.com P: Hal Rosenstock From rolandd at cisco.com Fri Jun 24 14:54:42 2005 From: rolandd at cisco.com (Roland Dreier) Date: Fri, 24 Jun 2005 14:54:42 -0700 Subject: [openib-general] [PATCH 2/3] IB: Fix pack/unpack when size_bits == 64 In-Reply-To: <20056241454.JSnV6qzt9RST2IRw@cisco.com> Message-ID: <20056241454.C4xXsKbhkkqxG17N@cisco.com> Fix handling of fields with size_bits == 64. Pointed out by Hal Rosenstock. Signed-off-by: Roland Dreier --- drivers/infiniband/core/packer.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) --- linux-export2.orig/drivers/infiniband/core/packer.c 2005-06-23 13:16:25.000000000 -0700 +++ linux-export2/drivers/infiniband/core/packer.c 2005-06-24 14:49:44.448271407 -0700 @@ -96,7 +96,7 @@ else val = 0; - mask = cpu_to_be64(((1ull << desc[i].size_bits) - 1) << shift); + mask = cpu_to_be64((~0ull >> (64 - desc[i].size_bits)) << shift); addr = (__be64 *) ((__be32 *) buf + desc[i].offset_words); *addr = (*addr & ~mask) | (cpu_to_be64(val) & mask); } else { @@ -176,7 +176,7 @@ __be64 *addr; shift = 64 - desc[i].offset_bits - desc[i].size_bits; - mask = ((1ull << desc[i].size_bits) - 1) << shift; + mask = (~0ull >> (64 - desc[i].size_bits)) << shift; addr = (__be64 *) buf + desc[i].offset_words; val = (be64_to_cpup(addr) & mask) >> shift; value_write(desc[i].struct_offset_bytes, From halr at voltaire.com Fri Jun 24 15:27:23 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 Jun 2005 18:27:23 -0400 Subject: [openib-general] [PATCH] [kdapl] Update EP state before disconnect Message-ID: <1119652043.4347.115.camel@hal.voltaire.com> [kdapl] Make update of EP state consistent. Update the EP state before disconnecting. Signed-off-by: Hal Rosenstock Index: dapl_evd.c =================================================================== -- dapl_evd.c (revision 2713) +++ dapl_evd.c (working copy) @@ -789,11 +789,11 @@ * reset the state to DISCONNECTED as we don't * expect a callback on an ABRUPT disconnect. */ - dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); spin_lock_irqsave(&ep->common.lock, ep->common.flags); ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; spin_unlock_irqrestore(&ep->common.lock, ep->common.flags); + dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); } } From eitan at mellanox.co.il Sat Jun 25 12:25:20 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Sat, 25 Jun 2005 22:25:20 +0300 Subject: [openib-general] IB Diagnositic Tools Message-ID: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com> Hi Fabian, Hal, Following the discussion about the debug tools, I would like to propose using OpenSM Vendor layer as a common layer for developing the debug tools. Since this layer is already available on both Windows and Linux stacks it could allow us to have the same code tree for both. Also I would like to propose developing an enhanced functionality for some of the tools. Especially adding the concept of reporting using "system names" rather then GUIDs and LIDs. The discovery tool would also be enhanced to perform some basic health checks for the fabric. As we (Mellanox) already have a "MADs" and "Topology" manipulations layers implemented we plan to open them in the OpenIB repository as well as develop the enhanced debug capability in OpenIB. If there is an interest in these tools we can provide an open version of those in week or two. Eitan Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -------------- next part -------------- An HTML attachment was scrubbed... URL: From itamar at mellanox.co.il Sun Jun 26 00:10:08 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Sun, 26 Jun 2005 10:10:08 +0300 Subject: [openib-general] mapping between IP address and device name Message-ID: <91DB792C7985D411BEC300B40080D29CC35D0E@mtvex01.mtv.mtl.com> > On the other hand, placing a mandatory content in the CM exchange > brings in a whole different raft of interoperability > questions, as James > mentioned earlier. For better or for worse, the ATS approach is easily > administered and does not impact any protocol layers outside of its > own. I think of it as ARP for IB. > > Tom. > _______________________________________________ But the ATS will not solve the problem of "many to one". What will the nfs module will do if the the result from the ATS will be a list of "IP's" which only one of them is has permission to the nfs ? ATS cant tell you who is the source IP. Itamar From itamar at mellanox.co.il Sun Jun 26 00:53:23 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Sun, 26 Jun 2005 10:53:23 +0300 Subject: [openib-general] RE: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to dapl _provider.c Message-ID: <91DB792C7985D411BEC300B40080D29CC35D0F@mtvex01.mtv.mtl.com> > uDAPL used the dapl_hca structure's ia_list to cleanup IA > resources when the user space process forked. When the code was ported > to the kernel, the list was retained, but the cleanup wasn't. > Suppose a kDAPL consumer (another kernel module) is unloaded and > forgets to close an IA. What do we want to have happen when the kDAPL > module is unloaded? Don't we want the resources associated with any > open IA's cleaned up? > > james > NO!!! If other modules has a bug (forgot to close the IA) we are not going to hide it. ib_dat_provider module will not be able to go down (be unloaded) until the bug will fixed. Kernel is not a place where you clean other modules BUGS. Udapl need this feature not Kdapl. Please delete this list. Itamar From turpentine at tpnet.pl Sun Jun 26 00:05:59 2005 From: turpentine at tpnet.pl (Darrin Frederick) Date: Sun, 26 Jun 2005 09:05:59 +0200 Subject: [openib-general] re: 22. Message-ID: <07GUK0USPCVM6WW4GW@UCBEH.SAN.UC.EDU> Hello, You have been chosen to participate in an invitation only limited time event! Are you currently paying over 3% for your mortgage? STOP! We can help you lower that today! Answer only a few questions and we can give you an approval in under 30 seconds � it�s that simple! http://BXwT.h3lp-m3.com/p2.asp And stop fighting for lenders � let them fight for you! Make them work for your business by giving you the lowest rates around! $230,000 loans are available for only $340/month! WE�RE PRACTICALLY GIVING AWAY MONEY! Think your credit is too bad to get a deal like this? THINK AGAIN! We will have you saving your money in no time! Are you ready to save your money? http://fFW.h3lp-m3.com/p2.asp http://FbXo.h3lp-m3.com/p1.asp Regards, Darrin Frederick From hch at lst.de Sun Jun 26 03:49:20 2005 From: hch at lst.de (Christoph Hellwig) Date: Sun, 26 Jun 2005 12:49:20 +0200 Subject: [openib-general] Re: [PATCH] cleanup dat provider registration In-Reply-To: References: <20050617163211.GA19071@lst.de> Message-ID: <20050626104920.GA24144@lst.de> On Wed, Jun 22, 2005 at 05:13:03PM -0400, James Lentini wrote: > > This is an excellent simplification. Committed in revision 2682 with a > few minor modifications: > > - kept printouts in dat_registry_add_provider, > dat_registry_remove_provider, and dat_registry_list_providers > > - updated printout in dat_ia_close (this wasn't something you > changed) > > - removed parens around sizeof kernel style is to have parants around it, but all of the openib code is different. well, let's keep it that way. > - removed space in front of labels lots of new kernel code uses the space, but again it's okay to stick to the surrounding code. > The last two are for consistency with the coding style we've been > using. If we've deviated from what is acceptable, let us know. > > Given this simplification, I can think of a few more changes: > > - rename api.c to registry.c Note yet. The code will get some major surgery still, and as part of that split into different files again maybe, just on very different boundaries. > - remove all "dictionary" references: rename dat_dictionary_search to > dat_provider_list_search, rename struct dat_dictionary_entry to > struct dat_provider_list_entry, rename > dat_dictionary_key_is_equal() to dat_provider_info_is_equal() I'll plan bigger changes in that area that will kill the dictionary term, so a simple search and replace is probably not worth it. From hch at lst.de Sun Jun 26 03:51:32 2005 From: hch at lst.de (Christoph Hellwig) Date: Sun, 26 Jun 2005 12:51:32 +0200 Subject: [openib-general] Re: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to dapl_provider.c In-Reply-To: References: <20050620085010.GA7206@mellanox.co.il> Message-ID: <20050626105132.GC24144@lst.de> On Thu, Jun 23, 2005 at 04:27:09PM -0400, James Lentini wrote: > However, I'm not sure we need an HCA object. Is there a better way to > organize the data being stored in the dapl_hca structure? I have this > feeling that we should merge all the structures that we create on a > per-hca basis (the dapl_provider_entry, dat_provider table, and > dapl_hca). Absolutely. dat_provider should become just a read-only method table, and then there should be a dat_hca object per hca, as explained a little in my other email From hch at lst.de Sun Jun 26 03:50:54 2005 From: hch at lst.de (Christoph Hellwig) Date: Sun, 26 Jun 2005 12:50:54 +0200 Subject: [openib-general] Re: [PATCH] cleanup dat provider registratio n In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35D07@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35D07@mtvex01.mtv.mtl.com> Message-ID: <20050626105054.GB24144@lst.de> On Thu, Jun 23, 2005 at 04:52:55PM +0300, Itamar Rabenstein wrote: > i have a question: > after we have cleaned up the dat_provider_info struct and now it is define > as : > > struct dat_provider_info { > char ia_name[DAT_NAME_MAX_LENGTH]; > }; > > why cant we replace it with a simple char * > it is only a string that the only thing we do with it > is to store it and compare with others strings. Two think here. One is that the dat_provider_info is should probably go away. My scheme so far it to introduce a dat_hca object that will contain the ia_name, and all of dictionary entry, and the linked list of ia objects that I wan't to move from the dat-provider to the dat layer. As part of that the registration/unregistration path will be changed a lot aswell. From rep.nop at aon.at Mon Jun 27 00:43:18 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Mon, 27 Jun 2005 09:43:18 +0200 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050622162856.GA18883@esmail.cup.hp.com> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622162856.GA18883@esmail.cup.hp.com> Message-ID: <20050627074318.GB18494@aon.at> On Wed, Jun 22, 2005 at 09:28:56AM -0700, Grant Grundler wrote: >I also don't know how to express > if target == static then > MORECFLAGS = -rdynamic... > endif $ cat Makefile BAR:=-Os ifeq ($(MAKECMDGOALS),static) BAR += -rdynamic endif #default: all static: all all: @echo "making ${MAKECMDGOALS}: $(BAR)" hth, Bernhard From mst at mellanox.co.il Mon Jun 27 01:19:32 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 27 Jun 2005 11:19:32 +0300 Subject: [openib-general] Re: [PATCH] rdma_lat-09 and results In-Reply-To: <20050627074318.GB18494@aon.at> References: <20050602005228.GP25321@esmail.cup.hp.com> <20050602170734.GB29040@esmail.cup.hp.com> <20050602234519.GA29972@mellanox.co.il> <20050604003301.GB2538@esmail.cup.hp.com> <20050604181237.GA18586@mellanox.co.il> <20050622065510.GC17224@esmail.cup.hp.com> <20050622071928.GA32485@mellanox.co.il> <20050622162856.GA18883@esmail.cup.hp.com> <20050627074318.GB18494@aon.at> Message-ID: <20050627081932.GA17004@mellanox.co.il> Quoting r. Bernhard Fischer : > Subject: Re: [openib-general] Re: [PATCH] rdma_lat-09 and results > > On Wed, Jun 22, 2005 at 09:28:56AM -0700, Grant Grundler wrote: > > >I also don't know how to express > > if target == static then > > MORECFLAGS = -rdynamic... > > endif > > $ cat Makefile > BAR:=-Os > ifeq ($(MAKECMDGOALS),static) > BAR += -rdynamic > endif > #default: all > static: all > all: > @echo "making ${MAKECMDGOALS}: $(BAR)" > > > hth, > Bernhard > Or better: static: BAR += -rdynamic -- MST From glebn at voltaire.com Mon Jun 27 01:27:45 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Mon, 27 Jun 2005 11:27:45 +0300 Subject: [openib-general] [PATCH] process locked in D state. Message-ID: <20050627082745.GJ15312@minantech.com> Hello, Summary: If I call ibv_close_device() on the context with unregistered memory process hangs in D state. This is what happens: ibv_close_device() close cmd_fd and then calls free_context(). free_context() calls munmap to unmap doorbell registers. In kernel sys_munmap gets mm->mmap_sem semaphore and calls do_munmap. do_munmap is the last user of the file so it calls release method of the file (ib_uverbs_close() in our case). ib_uverbs_close() calls ib_dealloc_ucontext(). ib_dealloc_ucontext() notices that there is unregistered memory on the file and calls ib_umem_release(). And there we are trying to acquire mm->mmap_sem on more time. One way to solve this problem is to call munmap before close in ibv_close_device() but this will not stop malicious user so this is not enough. In attached patch I use down_write_trylock() instead of down_write() in ib_umem_release(). If semaphore is already locked we will not update locked_vm statistics. This way malicious user can only cause harm to itself. The solution is not ideal since if some other process holds mmap_sem (for instance do 'cat /proc/pid/maps') we will not be able to update locked_vm counter but the chances this happening are close to zero. Index: trunk/src/userspace/libibverbs/src/device.c =================================================================== --- trunk/src/userspace/libibverbs/src/device.c (revision 2715) +++ trunk/src/userspace/libibverbs/src/device.c (working copy) @@ -121,10 +121,9 @@ close(context->async_fd); for (i = 0; i < context->num_comp; ++i) close(context->cq_fd[i]); + context->device->ops.free_context(context); close(context->cmd_fd); - context->device->ops.free_context(context); - return 0; } Index: trunk/src/linux-kernel/infiniband/core/uverbs_mem.c =================================================================== --- trunk/src/linux-kernel/infiniband/core/uverbs_mem.c (revision 2715) +++ trunk/src/linux-kernel/infiniband/core/uverbs_mem.c (working copy) @@ -163,18 +163,19 @@ void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) { struct mm_struct *mm; + int semlocked = 0; mm = get_task_mm(current); - if (mm) { - down_write(&mm->mmap_sem); + if (mm && (semlocked = down_write_trylock (&mm->mmap_sem))) { mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; } __ib_umem_release(dev, umem, 1); if (mm) { - up_write(&mm->mmap_sem); + if (semlocked) + up_write(&mm->mmap_sem); mmput(mm); } } -- Gleb. -------------- next part -------------- Index: trunk/src/userspace/libibverbs/src/device.c =================================================================== --- trunk/src/userspace/libibverbs/src/device.c (revision 2715) +++ trunk/src/userspace/libibverbs/src/device.c (working copy) @@ -121,10 +121,9 @@ close(context->async_fd); for (i = 0; i < context->num_comp; ++i) close(context->cq_fd[i]); + context->device->ops.free_context(context); close(context->cmd_fd); - context->device->ops.free_context(context); - return 0; } Index: trunk/src/linux-kernel/infiniband/core/uverbs_mem.c =================================================================== --- trunk/src/linux-kernel/infiniband/core/uverbs_mem.c (revision 2715) +++ trunk/src/linux-kernel/infiniband/core/uverbs_mem.c (working copy) @@ -163,18 +163,19 @@ void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) { struct mm_struct *mm; + int semlocked = 0; mm = get_task_mm(current); - if (mm) { - down_write(&mm->mmap_sem); + if (mm && (semlocked = down_write_trylock (&mm->mmap_sem))) { mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; } __ib_umem_release(dev, umem, 1); if (mm) { - up_write(&mm->mmap_sem); + if (semlocked) + up_write(&mm->mmap_sem); mmput(mm); } } From rep.nop at aon.at Mon Jun 27 04:01:47 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Mon, 27 Jun 2005 13:01:47 +0200 Subject: [openib-general] Re: [openib-commits] r2629 - in gen2/branches/shaharf-ibat/src/linux-kernel/infiniband: core include In-Reply-To: <20050615192718.0750F2283DD@openib.ca.sandia.gov> References: <20050615192718.0750F2283DD@openib.ca.sandia.gov> Message-ID: <20050627110147.GA19226@aon.at> Hal, On Wed, Jun 15, 2005 at 12:27:18PM -0700, halr at openib.org wrote: >Support callback for route by ip >Modified: gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/uat.c >=================================================================== >--- gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/uat.c 2005-06-15 18:22:01 UTC (rev 2628) >+++ gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/uat.c 2005-06-15 19:27:17 UTC (rev 2629) >@@ -414,13 +495,30 @@ > printk(KERN_ERR "ib_uat_event: uevent %p ctx %p status %d not completed\n", uevent, uevent->ctx, uevent->ctx->status); > uevent->ctx->rec_num = -EIO; > } >- /* Copy path records returned from SA back to userspace */ >- if (copy_to_user(uevent->ctx->user_path_arr, >- uevent->ctx->path_arr, >- uevent->ctx->user_length)) >- uevent->ctx->status = IB_USER_AT_STATUS_ERROR; >- kfree(uevent->ctx->path_arr); >- uevent->ctx->path_arr = NULL; >+ switch (uevent->type) { >+ case IB_UAT_PATH_EVENT: >+ /* Copy path records returned from SA to userspace */ >+ if (copy_to_user(uevent->ctx->user_path_arr, >+ uevent->ctx->path_arr, >+ uevent->ctx->user_length)) Should { result be set to FAULT here >+ uevent->ctx->status = IB_USER_AT_STATUS_ERROR; } >+ kfree(uevent->ctx->path_arr); >+ uevent->ctx->path_arr = NULL; >+ break; >+ case IB_UAT_ROUTE_EVENT: >+ if (copy_to_user(uevent->ctx->user_ib_route, >+ uevent->ctx->ib_route, >+ sizeof(*uevent->ctx->user_ib_route))) .. and here { >+ uevent->ctx->status = IB_USER_AT_STATUS_ERROR; } >+ kfree(uevent->ctx->ib_route); >+ uevent->ctx->ib_route = NULL; >+ break; >+ case IB_UAT_ATS_EVENT: >+ default: >+ printk(KERN_ERR "ib_uat_event: type %d not handled\n", >+ uevent->type); >+ break; >+ } > uevent->resp.callback = (u64)(unsigned long)uevent->ctx->user_callback; > uevent->resp.context = (u64)(unsigned long)uevent->ctx->user_context; > uevent->resp.req_id = uevent->ctx->req_id; > .. and take result into account before putting the response back to the user or is setting the status like you did above enough? Thanks for clarification. From halr at voltaire.com Mon Jun 27 05:55:12 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 Jun 2005 08:55:12 -0400 Subject: [openib-general] Re: [openib-commits] r2629 - in gen2/branches/shaharf-ibat/src/linux-kernel/infiniband: core include In-Reply-To: <20050627110147.GA19226@aon.at> References: <20050615192718.0750F2283DD@openib.ca.sandia.gov> <20050627110147.GA19226@aon.at> Message-ID: <1119876912.4848.17.camel@hal.voltaire.com> On Mon, 2005-06-27 at 07:01, Bernhard Fischer wrote: > Hal, > > On Wed, Jun 15, 2005 at 12:27:18PM -0700, halr at openib.org wrote: > > >Support callback for route by ip > > >Modified: gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/uat.c > >=================================================================== > >--- gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/uat.c 2005-06-15 18:22:01 UTC (rev 2628) > >+++ gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/uat.c 2005-06-15 19:27:17 UTC (rev 2629) > > >@@ -414,13 +495,30 @@ > > printk(KERN_ERR "ib_uat_event: uevent %p ctx %p status %d not completed\n", uevent, uevent->ctx, uevent->ctx->status); > > uevent->ctx->rec_num = -EIO; > > } > >- /* Copy path records returned from SA back to userspace */ > >- if (copy_to_user(uevent->ctx->user_path_arr, > >- uevent->ctx->path_arr, > >- uevent->ctx->user_length)) > >- uevent->ctx->status = IB_USER_AT_STATUS_ERROR; > >- kfree(uevent->ctx->path_arr); > >- uevent->ctx->path_arr = NULL; > >+ switch (uevent->type) { > >+ case IB_UAT_PATH_EVENT: > >+ /* Copy path records returned from SA to userspace */ > >+ if (copy_to_user(uevent->ctx->user_path_arr, > >+ uevent->ctx->path_arr, > >+ uevent->ctx->user_length)) > Should { result be set to FAULT here > >+ uevent->ctx->status = IB_USER_AT_STATUS_ERROR; > } > >+ kfree(uevent->ctx->path_arr); > >+ uevent->ctx->path_arr = NULL; > > > > >+ break; > >+ case IB_UAT_ROUTE_EVENT: > >+ if (copy_to_user(uevent->ctx->user_ib_route, > >+ uevent->ctx->ib_route, > >+ sizeof(*uevent->ctx->user_ib_route))) > > .. and here { > >+ uevent->ctx->status = IB_USER_AT_STATUS_ERROR; > } > >+ kfree(uevent->ctx->ib_route); > >+ uevent->ctx->ib_route = NULL; > > > > >+ break; > >+ case IB_UAT_ATS_EVENT: > >+ default: > >+ printk(KERN_ERR "ib_uat_event: type %d not handled\n", > >+ uevent->type); > >+ break; > >+ } > > uevent->resp.callback = (u64)(unsigned long)uevent->ctx->user_callback; > > uevent->resp.context = (u64)(unsigned long)uevent->ctx->user_context; > > uevent->resp.req_id = uevent->ctx->req_id; > > > .. and take result into account before putting the response back to the > user or is setting the status like you did above enough? I just changed this to deal with this error as you suggested. -- Hal From jlentini at netapp.com Mon Jun 27 07:18:16 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 27 Jun 2005 10:18:16 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl] Update EP state before disconnect In-Reply-To: <1119652043.4347.115.camel@hal.voltaire.com> References: <1119652043.4347.115.camel@hal.voltaire.com> Message-ID: Committed in revision 2718. On Fri, 24 Jun 2005, Hal Rosenstock wrote: halr> [kdapl] Make update of EP state consistent. Update the EP state before halr> disconnecting. halr> halr> Signed-off-by: Hal Rosenstock halr> halr> halr> Index: dapl_evd.c halr> =================================================================== halr> -- dapl_evd.c (revision 2713) halr> +++ dapl_evd.c (working copy) halr> @@ -789,11 +789,11 @@ halr> * reset the state to DISCONNECTED as we don't halr> * expect a callback on an ABRUPT disconnect. halr> */ halr> - dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); halr> spin_lock_irqsave(&ep->common.lock, ep->common.flags); halr> ep->param.ep_state = DAT_EP_STATE_DISCONNECTED; halr> spin_unlock_irqrestore(&ep->common.lock, halr> ep->common.flags); halr> + dapl_ib_disconnect(ep, DAT_CLOSE_ABRUPT_FLAG); halr> } halr> } halr> halr> From mst at mellanox.co.il Mon Jun 27 08:03:08 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 27 Jun 2005 18:03:08 +0300 Subject: [openib-general] [PATCH] mthca: report board id in sysfs Message-ID: <20050627150308.GD25304@mellanox.co.il> 4 last words in query adapter include the board id (byte-swapped). Show this board id in sysfs. Signed-off-by: Michael S. Tsirkin Index: hw/mthca/mthca_dev.h =================================================================== --- hw/mthca/mthca_dev.h (revision 2719) +++ hw/mthca/mthca_dev.h (working copy) @@ -66,6 +66,10 @@ enum { }; enum { + MTHCA_BOARD_ID_LEN = 0x10 +}; + +enum { MTHCA_EQ_CONTEXT_SIZE = 0x40, MTHCA_CQ_CONTEXT_SIZE = 0x40, MTHCA_QP_CONTEXT_SIZE = 0x200, @@ -245,6 +249,7 @@ struct mthca_dev { unsigned long device_cap_flags; u32 rev_id; + u8 board_id[MTHCA_BOARD_ID_LEN]; /* firmware info */ u64 fw_ver; Index: hw/mthca/mthca_main.c =================================================================== --- hw/mthca/mthca_main.c (revision 2719) +++ hw/mthca/mthca_main.c (working copy) @@ -284,6 +284,7 @@ static int __devinit mthca_init_tavor(st mdev->eq_table.inta_pin = adapter.inta_pin; mdev->rev_id = adapter.revision_id; + memcpy(mdev->board_id, adapter.board_id, sizeof mdev->board_id); return 0; Index: hw/mthca/mthca_provider.c =================================================================== --- hw/mthca/mthca_provider.c (revision 2719) +++ hw/mthca/mthca_provider.c (working copy) @@ -955,14 +955,23 @@ static ssize_t show_hca(struct class_dev } } +static ssize_t show_board(struct class_device *cdev, char *buf) +{ + struct mthca_dev *dev = container_of(cdev, struct mthca_dev, ib_dev.class_dev); + return sprintf(buf, "%.*s\n", MTHCA_BOARD_ID_LEN, dev->board_id); +} + + static CLASS_DEVICE_ATTR(hw_rev, S_IRUGO, show_rev, NULL); static CLASS_DEVICE_ATTR(fw_ver, S_IRUGO, show_fw_ver, NULL); static CLASS_DEVICE_ATTR(hca_type, S_IRUGO, show_hca, NULL); +static CLASS_DEVICE_ATTR(board_id, S_IRUGO, show_board, NULL); static struct class_device_attribute *mthca_class_attributes[] = { &class_device_attr_hw_rev, &class_device_attr_fw_ver, - &class_device_attr_hca_type + &class_device_attr_hca_type, + &class_device_attr_board_id }; int mthca_register_device(struct mthca_dev *dev) Index: hw/mthca/mthca_cmd.c =================================================================== --- hw/mthca/mthca_cmd.c (revision 2719) +++ hw/mthca/mthca_cmd.c (working copy) @@ -1087,13 +1087,14 @@ int mthca_QUERY_ADAPTER(struct mthca_dev { struct mthca_mailbox *mailbox; u32 *outbox; - int err; + int i, err; #define QUERY_ADAPTER_OUT_SIZE 0x100 #define QUERY_ADAPTER_VENDOR_ID_OFFSET 0x00 #define QUERY_ADAPTER_DEVICE_ID_OFFSET 0x04 #define QUERY_ADAPTER_REVISION_ID_OFFSET 0x08 #define QUERY_ADAPTER_INTA_PIN_OFFSET 0x10 +#define QUERY_ADAPTER_BOARD_ID_OFFSET 0xF0 mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); if (IS_ERR(mailbox)) @@ -1111,6 +1112,9 @@ int mthca_QUERY_ADAPTER(struct mthca_dev MTHCA_GET(adapter->revision_id, outbox, QUERY_ADAPTER_REVISION_ID_OFFSET); MTHCA_GET(adapter->inta_pin, outbox, QUERY_ADAPTER_INTA_PIN_OFFSET); + for (i = 0; i < sizeof adapter->board_id / 4; ++i) + adapter->board_id[i] = __swab32p(outbox + i + QUERY_ADAPTER_BOARD_ID_OFFSET / 4); + out: mthca_free_mailbox(dev, mailbox); return err; Index: hw/mthca/mthca_cmd.h =================================================================== --- hw/mthca/mthca_cmd.h (revision 2719) +++ hw/mthca/mthca_cmd.h (working copy) @@ -187,6 +187,7 @@ struct mthca_adapter { u32 device_id; u32 revision_id; u8 inta_pin; + u32 board_id[MTHCA_BOARD_ID_LEN / 4]; }; struct mthca_init_hca_param { -- MST From rolandd at cisco.com Mon Jun 27 09:20:41 2005 From: rolandd at cisco.com (Roland Dreier) Date: Mon, 27 Jun 2005 09:20:41 -0700 Subject: [openib-general] [PATCH] mthca: report board id in sysfs In-Reply-To: <20050627150308.GD25304@mellanox.co.il> (Michael S. Tsirkin's message of "Mon, 27 Jun 2005 18:03:08 +0300") References: <20050627150308.GD25304@mellanox.co.il> Message-ID: <527jgf93fq.fsf@topspin.com> Michael> 4 last words in query adapter include the board id Michael> (byte-swapped). Show this board id in sysfs. Thanks, looks useful. Is the VSD really always byte-swapped or just in big-endian order? In other words, should this: + adapter->board_id[i] = __swab32p(outbox + i + QUERY_ADAPTER_BOARD_ID_OFFSET / 4); use be32_to_cpup() instead? - R. From rolandd at cisco.com Mon Jun 27 09:24:45 2005 From: rolandd at cisco.com (Roland Dreier) Date: Mon, 27 Jun 2005 09:24:45 -0700 Subject: [openib-general] [PATCH] process locked in D state. In-Reply-To: <20050627082745.GJ15312@minantech.com> (Gleb Natapov's message of "Mon, 27 Jun 2005 11:27:45 +0300") References: <20050627082745.GJ15312@minantech.com> Message-ID: <523br3938y.fsf@topspin.com> Gleb> This is what happens: ibv_close_device() close cmd_fd and Gleb> then calls free_context(). free_context() calls munmap to Gleb> unmap doorbell registers. In kernel sys_munmap gets Gleb> mm->mmap_sem semaphore and calls do_munmap. do_munmap is Gleb> the last user of the file so it calls release method of the Gleb> file (ib_uverbs_close() in our case). ib_uverbs_close() Gleb> calls ib_dealloc_ucontext(). ib_dealloc_ucontext() notices Gleb> that there is unregistered memory on the file and calls Gleb> ib_umem_release(). And there we are trying to acquire Gleb> mm->mmap_sem on more time. Thanks for the good debugging work. Gleb> In attached patch I use down_write_trylock() instead of Gleb> down_write() in ib_umem_release(). If semaphore is already Gleb> locked we will not update locked_vm statistics. This way Gleb> malicious user can only cause harm to itself. I don't like this solution -- as you point out, down_write_trylock() may fail if there is even momentary contention on the mmap_sem. So for example a different malicious user could poll on /proc//maps and cause our locked_vm to continue to grow. How about if we use schedule_work() to defer the modification of locked_vm? - R. From mshefty at ichips.intel.com Mon Jun 27 09:51:23 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 27 Jun 2005 09:51:23 -0700 Subject: [openib-general] [Fwd: Returned mail: see transcript for details] In-Reply-To: <1118857749.3845.8.camel@duffman> References: <1118857749.3845.8.camel@duffman> Message-ID: <42C02E8B.1070506@ichips.intel.com> Tom Duffy wrote: > I have grepped the core and include logs for any files I have changed > and added a Sun copyright. > > Signed-off-by: Tom Duffy > Assuming that these haven't been applied yet, I will try to get to this today or tomorrow. - Sean From steven.merker at gmail.com Mon Jun 27 10:16:30 2005 From: steven.merker at gmail.com (Steve Merker) Date: Mon, 27 Jun 2005 10:16:30 -0700 Subject: [openib-general] compile issues w/sdp on 2.6.11: /gen2/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_pass.c Message-ID: Hello, I'm having problems getting the sdp kernel module to build against 2.6.11 and/or 2.6.11.12. Looks like the latest version (2665) of sdp modified some of the sdp debugging and socket debugging. The last sdp module that I can get to work is r2663. Wondering if I missed a patch for my sock.h include. -Steven [root at cluster4 linux-2.6.11.12]# make modules CHK include/linux/version.h make[1]: `arch/i386/kernel/asm-offsets.s' is up to date. CC [M] drivers/infiniband/ulp/sdp/sdp_pass.o drivers/infiniband/ulp/sdp/sdp_pass.c: In function `sdp_cm_listen_lookup': drivers/infiniband/ulp/sdp/sdp_pass.c:299: error: `SOCK_DBG' undeclared (first use in this function) drivers/infiniband/ulp/sdp/sdp_pass.c:299: error: (Each undeclared identifier is reported only once drivers/infiniband/ulp/sdp/sdp_pass.c:299: error: for each function it appears in.) drivers/infiniband/ulp/sdp/sdp_pass.c:301: error: `SOCK_LOCALROUTE' undeclared (first use in this function) drivers/infiniband/ulp/sdp/sdp_pass.c:307: error: `SOCK_RCVTSTAMP' undeclared (first use in this function) make[3]: *** [drivers/infiniband/ulp/sdp/sdp_pass.o] Error 1 make[2]: *** [drivers/infiniband/ulp/sdp] Error 2 make[1]: *** [drivers/infiniband] Error 2 make: *** [drivers] Error 2 ------------------------------ [root at cluster4 ulp]# svn info sdp/sdp_pass.c Path: sdp/sdp_pass.c Name: sdp_pass.c URL: https://openib.org/svn/gen2/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_pass.c Repository UUID: 21a7a0b7-18d7-0310-8e21-e8b31bdbf5cd Revision: 2719 Node Kind: file Schedule: normal Last Changed Author: libor Last Changed Rev: 2665 Last Changed Date: 2005-06-20 22:14:58 -0700 (Mon, 20 Jun 2005) Text Last Updated: 2005-06-27 09:48:44 -0700 (Mon, 27 Jun 2005) Properties Last Updated: 2005-06-24 10:07:28 -0700 (Fri, 24 Jun 2005) Checksum: 714d1584bd100396a287538ff65f01d0 From rolandd at cisco.com Mon Jun 27 10:38:04 2005 From: rolandd at cisco.com (Roland Dreier) Date: Mon, 27 Jun 2005 10:38:04 -0700 Subject: [openib-general] compile issues w/sdp on 2.6.11: /gen2/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_pass.c In-Reply-To: (Steve Merker's message of "Mon, 27 Jun 2005 10:16:30 -0700") References: Message-ID: <521x6n7lab.fsf@topspin.com> Steve> Hello, I'm having problems getting the sdp kernel module to Steve> build against 2.6.11 and/or 2.6.11.12. Looks like the Steve> latest version (2665) of sdp modified some of the sdp Steve> debugging and socket debugging. The last sdp module that I Steve> can get to work is r2663. Now that 2.6.12 is out, the SDP code in the svn tree has been updated to match the new sockets API in the 2.6.12 kernel. - R> From rep.nop at aon.at Mon Jun 27 10:39:51 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Mon, 27 Jun 2005 19:39:51 +0200 Subject: [openib-general] [PATCH][dapl] cleanup dapl_cookie Message-ID: <20050627173951.GB19279@aon.at> Hi James, untested. - cleanup dapl_cookie.c: remove unneeded local variables and simplify branches to be consistent with dapl_rmr_cookie_alloc(). Signed-off-by: Bernhard Fischer thank you, Bernhard -------------- next part -------------- Index: users/jlentini/linux-kernel/dat-provider/dapl_cookie.c =================================================================== --- users/jlentini/linux-kernel/dat-provider/dapl_cookie.c (revision 2715) +++ users/jlentini/linux-kernel/dat-provider/dapl_cookie.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Bernhard Fischer, All rights reserved. * * This Software is licensed under one of the following licenses: * @@ -136,8 +137,8 @@ u32 dapl_cb_create(struct dapl_cookie_bu } return DAT_SUCCESS; - } else - return DAT_INSUFFICIENT_RESOURCES; + } + return DAT_INSUFFICIENT_RESOURCES; } /* @@ -157,8 +158,7 @@ u32 dapl_cb_create(struct dapl_cookie_bu */ void dapl_cb_free(struct dapl_cookie_buffer *buffer) { - if (NULL != buffer->pool) - kfree(buffer->pool); + kfree(buffer->pool); } /* @@ -181,24 +181,19 @@ void dapl_cb_free(struct dapl_cookie_buf u32 dapl_cb_get(struct dapl_cookie_buffer *buffer, struct dapl_cookie **cookie_ptr) { - u32 dat_status; int new_head; BUG_ON(cookie_ptr == NULL); new_head = (atomic_read(&buffer->head) + 1) % buffer->pool_size; - if (new_head == atomic_read(&buffer->tail)) { - dat_status = DAT_INSUFFICIENT_RESOURCES; - goto bail; - } else { - atomic_set(&buffer->head, new_head); + if (new_head == atomic_read(&buffer->tail)) + return DAT_INSUFFICIENT_RESOURCES; - *cookie_ptr = &buffer->pool[atomic_read(&buffer->head)]; - dat_status = DAT_SUCCESS; - } -bail: - return dat_status; + atomic_set(&buffer->head, new_head); + + *cookie_ptr = &buffer->pool[atomic_read(&buffer->head)]; + return DAT_SUCCESS; } /* @@ -224,24 +219,19 @@ u32 dapl_rmr_cookie_alloc(struct dapl_co struct dapl_cookie **cookie_ptr) { struct dapl_cookie *cookie; - u32 dat_status; if (DAT_SUCCESS != dapl_cb_get(buffer, &cookie)) { *cookie_ptr = NULL; - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; + return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); } - dat_status = DAT_SUCCESS; cookie->type = DAPL_COOKIE_TYPE_RMR; cookie->val.rmr.rmr = rmr; cookie->val.rmr.cookie = user_cookie; *cookie_ptr = cookie; - -bail: - return dat_status; + return DAT_SUCCESS; } /* From mst at mellanox.co.il Mon Jun 27 10:42:41 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 27 Jun 2005 20:42:41 +0300 Subject: [openib-general] [PATCH] mthca: report board id in sysfs In-Reply-To: <527jgf93fq.fsf@topspin.com> References: <20050627150308.GD25304@mellanox.co.il> <527jgf93fq.fsf@topspin.com> Message-ID: <20050627174241.GA25939@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [openib-general] [PATCH] mthca: report board id in sysfs > > Michael> 4 last words in query adapter include the board id > Michael> (byte-swapped). Show this board id in sysfs. > > Thanks, looks useful. > > Is the VSD really always byte-swapped or just in big-endian order? In > other words, should this: > > + adapter->board_id[i] = __swab32p(outbox + i + QUERY_ADAPTER_BOARD_ID_OFFSET / 4); > > use be32_to_cpup() instead? > > - R. > Its really an ASCII string, so it looks the same (byte-swapped) on all architectures. -- MST From rolandd at cisco.com Mon Jun 27 10:48:08 2005 From: rolandd at cisco.com (Roland Dreier) Date: Mon, 27 Jun 2005 10:48:08 -0700 Subject: [openib-general] [PATCH] mthca: report board id in sysfs In-Reply-To: <20050627174241.GA25939@mellanox.co.il> (Michael S. Tsirkin's message of "Mon, 27 Jun 2005 20:42:41 +0300") References: <20050627150308.GD25304@mellanox.co.il> <527jgf93fq.fsf@topspin.com> <20050627174241.GA25939@mellanox.co.il> Message-ID: <52u0jj6693.fsf@topspin.com> Michael> Its really an ASCII string, so it looks the same Michael> (byte-swapped) on all architectures. I'm confused... if it's just a string, why don't we just use memcpy() to get it from the mailbox? - R. From ftillier at silverstorm.com Mon Jun 27 10:55:38 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Mon, 27 Jun 2005 10:55:38 -0700 Subject: [openib-general] RE: IB Diagnositic Tools In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com> Message-ID: <000501c57b41$6e367300$9c5aa8c0@infiniconsys.com> > From: Eitan Zahavi [mailto:eitan at mellanox.co.il] > Sent: Saturday, June 25, 2005 12:25 PM > > Following the discussion about the debug tools, I would like to propose using > OpenSM Vendor layer as a common layer for developing the debug tools. Since > this layer is already available on both Windows and Linux stacks it could > allow us to have the same code tree for both. I think this is a decent idea. My only reservations are that it would require everyone to learn the OSM Vendor Layer API. It might also not allow testing nuances in the access layer APIs, which might be useful. So I think it would be useful to have the test run over each low level MAD API, as well as to the OSM Vendor Layer. I'm a bit weary of adding extra layers between the tests and the access layer - it just creates more areas where things can go wrong. That said, I'm not dead set on this and could be convinced otherwise, but I just don't know enough about the OSM Vendor Layer at the moment and don't have many cycles to learn it. > Also I would like to propose developing an enhanced functionality for some of > the tools. > > Especially adding the concept of reporting using "system names" rather then > GUIDs and LIDs. By system names, you mean node descriptions? > The discovery tool would also be enhanced to perform some basic health checks > for the fabric. I think this would be valuable. > As we (Mellanox) already have a "MADs" and "Topology" manipulations layers > implemented we plan to open them in the OpenIB repository as well as develop > the enhanced debug capability in OpenIB. > > If there is an interest in these tools we can provide an open version of those > in week or two. Sounds good, thanks! - Fab From halr at voltaire.com Mon Jun 27 11:04:16 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 Jun 2005 14:04:16 -0400 Subject: [openib-general] Re: IB Diagnositic Tools In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com> Message-ID: <1119895456.4848.353.camel@hal.voltaire.com> Hi Eitan, On Sat, 2005-06-25 at 15:25, Eitan Zahavi wrote: > Following the discussion about the debug tools, I would like to > propose using OpenSM Vendor layer as a common layer for developing the > debug tools. Is the OpenSM vendor layer available in Windows for OpenIB or is this something which needs to be developed ? > Since this layer is already available on both Windows and Linux stacks > it could allow us to have the same code tree for both. Will the Linux distros take it this way (with #ifdef OS) > Also I would like to propose developing an enhanced functionality for > some of the tools. > > Especially adding the concept of reporting using "system names" rather > then GUIDs and LIDs. Not sure exactly what you have in mind here but there is something like this on the TODO list. How are GUIDs and LIDs aggregated into a name ? Is this SystemImageGUID ? > The discovery tool would also be enhanced to perform some basic health > checks for the fabric. Sure that can be an additional option. What are the basic health checks you want to add ? > As we (Mellanox) already have a "MADs" and "Topology" manipulations > layers implemented we plan to open them in the OpenIB repository as > well as develop the enhanced debug capability in OpenIB. Not sure what you mean exactly by MAD and topology manipulations layers. Are there different tools ? What do they provide different from the current OpenIB diagnostics ? > If there is an interest in these tools we can provide an open version > of those in week or two. Will the development then be done in the OpenIB tree or will the drop model be used ? On the Linux side, the community desires the tools to be built with autotools. -- Hal > Eitan > > Eitan Zahavi > > Design Technology Director > > Mellanox Technologies LTD > > Tel:+972-4-9097208 > Fax:+972-4-9593245 > > P.O. Box 586 Yokneam 20692 ISRAEL > From libor at topspin.com Mon Jun 27 11:17:21 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 27 Jun 2005 11:17:21 -0700 Subject: [openib-general] Re: SDP: still getting sk_alloc() panic, any ideas? In-Reply-To: <9d3b7de7050623130658aa92@mail.gmail.com>; from tomduffy@gmail.com on Thu, Jun 23, 2005 at 01:06:47PM -0700 References: <9d3b7de7050623130658aa92@mail.gmail.com> Message-ID: <20050627111721.A26240@topspin.com> On Thu, Jun 23, 2005 at 01:06:47PM -0700, Tom Duffy wrote: > I am still getting the panic when you try to connect to a machine and > it is not listening (but has ib_sdp loaded): > > [root at sins-stinger-10 ~]# ----------- [cut here ] --------- [please > bite here ] --------- > Kernel BUG at "/build1/tduffy/openib-work/linux-2.6.12-openib/in:352 > invalid operand: 0000 [1] SMP > CPU 1 > > Any idea about this? Sorry, I've been out of the office for a few days. The problem is that each call to sk_alloc() is grabbing a reference to the module, but it checks to make sure that there already is at least one reference, if not the top BUG is triggered. In the case of the passive connection there are no other references to the module. You can see that the problem goes away if you open just one socket, even if you don't listen on it, and then try the failing passive connect. When a socket is created it actually grabs two references to the module, one at the sock level and one at the sk level. The first reference at the sock level does not trigger the BUG since it's through another code path. (try_module_get vs. __module_get) This is why we only hit this during passive connect to a system that has no active SDP sockets. Not sure the right way to fix this, maybe check to see if the socket table size (dev_root_s.sk_entry) is greater then 0 in sdp_cm_req_handler() before even performing the alloc... -Libor From eitan at mellanox.co.il Mon Jun 27 11:20:50 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Mon, 27 Jun 2005 21:20:50 +0300 Subject: [openib-general] RE: IB Diagnositic Tools Message-ID: <506C3D7B14CDD411A52C00025558DED607C303C5@mtlex01.yok.mtl.com> Hi Fabian > > I think this is a decent idea. My only reservations are that it would require > everyone to learn the OSM Vendor Layer API. It might also not allow testing > nuances in the access layer APIs, which might be useful. [EZ] This is true. But the API is simple. The MAD flow API is: bind - to get a handle for sending mads of specific class and registering callbacks send - to send a mad get_mad - to get a mad buffer put_mad - to return it to the driver The rest can be found in the OpenSM repository under osm_vendor_api.h > > So I think it would be useful to have the test run over each low level MAD API, > as well as to the OSM Vendor Layer. I'm a bit weary of adding extra layers > between the tests and the access layer - it just creates more areas where things > can go wrong. That said, I'm not dead set on this and could be convinced > otherwise, but I just don't know enough about the OSM Vendor Layer at the moment > and don't have many cycles to learn it. [EZ] I agree. Code testing should be done in all layers. But writing cluster debug tools is easier with a higher abstraction layer (callbacks vs. polling or blocking reads). > > > By system names, you mean node descriptions? [EZ] If the user provide a file describing the topology in terms of systems then the code uses the names provided in the file in its reports. For example: Assuming you have a cluster built of a 288port switch and 288 HCAs. The topology description could then be: IBSW288 mySwitch Leaf1/P1 -> HCA Rack1-Node1 P1 Leaf1/P2 -> HCA Rack1-Node2 P1 ... Leaf1/P12 -> HCA Rack2-Node3 P1 Leaf2/P1 -> HCA anyNameYouWant P2 .... Then any error report can be provided in these names like: Error with cable from mySwitch/Leaf2/P1 to anyNameYouWant/P1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From eitan at mellanox.co.il Mon Jun 27 11:29:38 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Mon, 27 Jun 2005 21:29:38 +0300 Subject: [openib-general] RE: IB Diagnositic Tools Message-ID: <506C3D7B14CDD411A52C00025558DED607C303C6@mtlex01.yok.mtl.com> Hi Hal, > > Is the OpenSM vendor layer available in Windows for OpenIB or is this > something which needs to be developed ? [EZ] It is available (the code was part of IBAL but needed some fixes etc). > > > Since this layer is already available on both Windows and Linux stacks > > it could allow us to have the same code tree for both. > > Will the Linux distros take it this way (with #ifdef OS) [EZ] The different implementations are already included in the OpenSM code on the OpenIB trunk. They are not named Win/Linux but IBAL, TS. I propose we perform some restructuring where each "vendor" has its own package and all of them are actually made into the same "lib" name. Such that for OpenSM build it should not make any difference which vendor it is linked with. > > Not sure exactly what you have in mind here but there is something like > this on the TODO list. How are GUIDs and LIDs aggregated into a name ? > Is this SystemImageGUID ? [EZ] I mean using topology file to map discovered topology to specified topology and thus enable the use of the user given names for the various systems (instead of guids). > > > Not sure what you mean exactly by MAD and topology manipulations layers. > Are there different tools ? What do they provide different from the > current OpenIB diagnostics ? [EZ] The MAD layer provides scripting interface for sending receiving mads of the various classes. The analysis layers perform topology matching, LFT traversals, credit loop analysis, MFT connectivity checks, routing hops histograms, etc. > > > If there is an interest in these tools we can provide an open version > > of those in week or two. > > Will the development then be done in the OpenIB tree or will the drop > model be used ? On the Linux side, the community desires the tools to be > built with autotools. [EZ] The idea is to move the current tools into OpenIB and use OpenIB as the development environment. This includes OpenSM and the MAD layers. All should be autotools. -------------- next part -------------- An HTML attachment was scrubbed... URL: From libor at topspin.com Mon Jun 27 11:35:21 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 27 Jun 2005 11:35:21 -0700 Subject: [openib-general] SDP sk_data_ready() callback In-Reply-To: <1119429583.8208.28.camel@confield>; from arne.redlich@xiranet.com on Wed, Jun 22, 2005 at 10:39:43AM +0200 References: <1119429583.8208.28.camel@confield> Message-ID: <20050627113521.B26240@topspin.com> On Wed, Jun 22, 2005 at 10:39:43AM +0200, Arne Redlich wrote: > Hi, > > I'm trying to use SDP from within the kernel. My problem is that the > code relies on sk_data_ready() (this callback is modified to wake up a > Rx thread before executing the original function), but sk_data_ready() > apparently never gets called. Is there any way to fix this? You mean that you replace sk->sk_data_ready, with a similar but slightly modified version and so rely on sk->sk_data_ready() being called? You're right it's currently not being called, instead we're calling the function to which it's pointing directly, and for no real reason. Also, the function that's being called is a duplicate of the function sock_def_readable() in net/sock.c. I'll look at correcting these issues. -Libor From halr at voltaire.com Mon Jun 27 11:44:52 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 Jun 2005 14:44:52 -0400 Subject: [openib-general] user_mad: Support RMPP on receive side Message-ID: <1119897892.4848.396.camel@hal.voltaire.com> Hi, I am in the process of enabling receive side RMPP in user_mad. There is an open issue about read (length returned when supplied buffer is too small) and failing doing it that way, there would be an additional ioctl. I would prefer to do it the read way as that seems more optimal in that a mad agent doing receives can do a normal MAD sized read and if a larger RMPP size packet comes in, he just does a read with the larger size. With the ioctl, the user would need to do an ioctl to get the size for the read if any RMPPs are possible. Any objections ? Thanks. -- Hal -----Forwarded Message----- From: Hal Rosenstock To: Roland Dreier Cc: openib-general at openib.org Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side Date: 20 May 2005 09:23:00 -0400 On Wed, 2005-05-18 at 19:04, Roland Dreier wrote: > This looks OK to check in with one small comment on the following: > > - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) > + if (copy_to_user(buf, &packet->mad, > + min(count, packet->length + > + sizeof (struct ib_user_mad)))) > ret = -EFAULT; > else > - ret = sizeof packet->mad; > + ret = count; > > This code will truncate a received MAD that is bigger than the buffer > passed into read(), but return the full size of the packet. I don't > think read() is allowed to do this: the return value can be at most > the count value passed in by the user. > > I think we have two options: truncate and return the actual amount of > data read to the user, or return an error if the user's buffer is too > small. The man page for read states: read() attempts to read up to count bytes from file descriptor fd into the buffer starting at buf. RETURN VALUE On success, the number of bytes read is returned (zero indicates end of file), and the file position is advanced by this number. For RMPP reads (next set of changes to this), it would eliminate an additional call (ioctl) if on a read that a return length larger than the count supplied indicates that the read did not occur and the count (buffer size) to be used for the next read to get the entire packet. It appears to me that there is nothing that enforces the man page behavior above in Linux so this is a "convention". Is this something we can take advantage of or do we need to require the additional call to get the buffer length ? -- Hal _______________________________________________ openib-general mailing list openib-general at openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From halr at voltaire.com Mon Jun 27 11:58:28 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 Jun 2005 14:58:28 -0400 Subject: [openib-general] RE: IB Diagnositic Tools In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C303C6@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C303C6@mtlex01.yok.mtl.com> Message-ID: <1119898707.4848.414.camel@hal.voltaire.com> Hi again Eitan, On Mon, 2005-06-27 at 14:29, Eitan Zahavi wrote: > Hi Hal, > > > > > Is the OpenSM vendor layer available in Windows for OpenIB or is > this > > something which needs to be developed ? > [EZ] It is available (the code was part of IBAL but needed some fixes > etc). Needed or still needs some fixes ? > > > Since this layer is already available on both Windows and Linux > stacks > > > it could allow us to have the same code tree for both. > > > > Will the Linux distros take it this way (with #ifdef OS) > [EZ] The different implementations are already included in the OpenSM > code on the OpenIB trunk. They are not named Win/Linux but IBAL, TS. I > propose we perform some restructuring where each "vendor" has its own > package and all of them are actually made into the same "lib" name. > Such that for OpenSM build it should not make any difference which > vendor it is linked with. Currently, the OpenIB build (for Linux) just deals with the OpenIB vendor layer. The others are there for tracking to OpenSM releases and are not part of the build (or current requirements for the build). > > Not sure exactly what you have in mind here but there is something > like > > this on the TODO list. How are GUIDs and LIDs aggregated into a name > ? > > Is this SystemImageGUID ? > [EZ] I mean using topology file to map discovered topology to > specified topology and thus enable the use of the user given names for > the various systems (instead of guids). The current OpenIB topology file has a place where these annotations can be made (and displayed). > > Not sure what you mean exactly by MAD and topology manipulations > layers. > > Are there different tools ? What do they provide different from the > > current OpenIB diagnostics ? > [EZ] The MAD layer provides scripting interface for sending receiving > mads of the various classes. > The analysis layers perform topology matching, LFT traversals, credit > loop analysis, MFT connectivity checks, routing hops histograms, etc. This brings in more things that are not currently ported to OpenIB and also there are some issues with some of these tools. > > > If there is an interest in these tools we can provide an open > version > > > of those in week or two. > > > > Will the development then be done in the OpenIB tree or will the > drop > > model be used ? On the Linux side, the community desires the tools > to be > > built with autotools. > [EZ] The idea is to move the current tools into OpenIB and use OpenIB > as the development environment. This includes OpenSM and the MAD > layers. > All should be autotools. Glad to hear it. -- Hal From Thomas.Talpey at netapp.com Mon Jun 27 12:23:46 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Mon, 27 Jun 2005 15:23:46 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35D0E@mtvex01.mtv.mtl.com > References: <91DB792C7985D411BEC300B40080D29CC35D0E@mtvex01.mtv.mtl.com> Message-ID: <6.2.1.2.2.20050627151532.04a0a480@exnane01.nane.netapp.com> At 03:10 AM 6/26/2005, Itamar Rabenstein wrote: >But the ATS will not solve the problem of "many to one". >What will the nfs module will do if the the result from the ATS will be >a list of "IP's" which only one of them is has permission to the nfs ? >ATS cant tell you who is the source IP. The NFS server exports will function just fine in such a case. This is no different from any other multihomed client, and /etc/exports can be configured appropriately. What wouldn't be useful would be to use MAC addresses (GIDs) for mounting, exports, etc. Can you imagine administering a network where hardware addresses were the only naming? No sysadmin would even entertain such an idea. Tom. From eitan at mellanox.co.il Mon Jun 27 12:27:25 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Mon, 27 Jun 2005 22:27:25 +0300 Subject: [openib-general] RE: IB Diagnositic Tools Message-ID: <506C3D7B14CDD411A52C00025558DED607C303C9@mtlex01.yok.mtl.com> > > [EZ] It is available (the code was part of IBAL but needed some fixes > > etc). > > Needed or still needs some fixes ? [EZ] Under work but really close to completion > > > The current OpenIB topology file has a place where these annotations can > be made (and displayed). [EZ] How would you define the internal structure of a 288port switch in the existing topology file? Would it support writing code that is able to report something like "board spine2 of system mySwitch is missing"? The code that supports all that is part of the simulator code I have posted long ago. Please give it a look. Especially the Fabric.h, SysDef.h, ibnl_parser.yy in https://openib.org/svn/gen2/utils/src/linux-user/ibdm/datamodel > > > This brings in more things that are not currently ported to OpenIB and > also there are some issues with some of these tools. [EZ] Never heard of any specific issue. Can you describe these issues? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Mon Jun 27 12:34:42 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 27 Jun 2005 22:34:42 +0300 Subject: [openib-general] [PATCH] mthca: report board id in sysfs In-Reply-To: <52u0jj6693.fsf@topspin.com> References: <20050627150308.GD25304@mellanox.co.il> <527jgf93fq.fsf@topspin.com> <20050627174241.GA25939@mellanox.co.il> <52u0jj6693.fsf@topspin.com> Message-ID: <20050627193442.GA26669@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [openib-general] [PATCH] mthca: report board id in sysfs > > Michael> Its really an ASCII string, so it looks the same > Michael> (byte-swapped) on all architectures. > > I'm confused... if it's just a string, why don't we just use memcpy() > to get it from the mailbox? > > - R. > Because firmware puts it byte-swapped in the mailbox. E.g. if you put in there "foo" in the mailbox you get "\0oof". Board ID should be things like "MT_00A0000001", mailbox has things like "0_TM00A00000\0\0\01". -- MST From jcarr at linuxmachines.com Mon Jun 27 13:26:03 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 27 Jun 2005 13:26:03 -0700 Subject: [openib-general] A new simple ulp (SPTS) In-Reply-To: <42B99546.2070703@linuxmachines.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> Message-ID: <42C060DB.4040409@linuxmachines.com> On 06/22/05 09:43, Jeff Carr wrote: > On 06/21/2005 12:50 PM, Roland Dreier wrote: > > >>What happens if you try replacing the send_flags line with the one you >>have commented out? >> >>+ // send_wr.send_flags = IB_SEND_SIGNALED; > > > Thanks, you are correct. IB_SEND_SIGNALED gives me the behavior I was > expecting. Here is an updated version and a simple perl script that tests it's performance. With 2K messages, these were the performance numbers between 2 systems (3.6ghz Xeon/w 133mhz/64bit pci). root at delta:~# ./fast_test.pl 20 starting sends 0 messages/sec (0 Mb/sec) 131072 messages/sec (2047 Mb/sec) 131072 messages/sec (2047 Mb/sec) 131072 messages/sec (2047 Mb/sec) 174762 messages/sec (2730 Mb/sec) 163840 messages/sec (2559 Mb/sec) 196608 messages/sec (3071 Mb/sec) 183500 messages/sec (2867 Mb/sec) 174762 messages/sec (2730 Mb/sec) 196618 messages/sec (3072 Mb/sec) 187254 messages/sec (2925 Mb/sec) 180232 messages/sec (2816 Mb/sec) 196616 messages/sec (3072 Mb/sec) 189333 messages/sec (2958 Mb/sec) 183507 messages/sec (2867 Mb/sec) 196614 messages/sec (3072 Mb/sec) 190656 messages/sec (2978 Mb/sec) 202571 messages/sec (3165 Mb/sec) 138785 messages/sec (2168 Mb/sec) 131075 messages/sec (2048 Mb/sec) Jeff -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Kconfig URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Makefile URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: client_start.c Type: text/x-csrc Size: 1064 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cm_spts.c Type: text/x-csrc Size: 13343 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fast.c Type: text/x-csrc Size: 5619 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fast_test.pl Type: application/x-perl Size: 446 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: restart_spts.pl Type: application/x-perl Size: 1607 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: spts.c Type: text/x-csrc Size: 15286 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: spts.h Type: text/x-chdr Size: 3891 bytes Desc: not available URL: From mst at mellanox.co.il Mon Jun 27 13:28:54 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 27 Jun 2005 23:28:54 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <523br3938y.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <523br3938y.fsf@topspin.com> Message-ID: <20050627202854.GA26909@mellanox.co.il> Quoting r. Roland Dreier : > How about if we use schedule_work() to defer the modification of > locked_vm? Good idea. This would also reduce the time during which we hold the mm semaphore, which is nice. -- MST From mst at mellanox.co.il Mon Jun 27 13:32:45 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 27 Jun 2005 23:32:45 +0300 Subject: [openib-general] Re: A new simple ulp (SPTS) In-Reply-To: <42C060DB.4040409@linuxmachines.com> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <42C060DB.4040409@linuxmachines.com> Message-ID: <20050627203245.GB26909@mellanox.co.il> Quoting r. Jeff Carr : > Here is an updated version and a simple perl script that tests it's > performance. With 2K messages, these were the performance numbers > between 2 systems (3.6ghz Xeon/w 133mhz/64bit pci). > > root at delta:~# ./fast_test.pl 20 > starting sends > 0 messages/sec (0 Mb/sec) > 131072 messages/sec (2047 Mb/sec) > 131072 messages/sec (2047 Mb/sec) > 131072 messages/sec (2047 Mb/sec) > 174762 messages/sec (2730 Mb/sec) > 163840 messages/sec (2559 Mb/sec) > 196608 messages/sec (3071 Mb/sec) > 183500 messages/sec (2867 Mb/sec) > 174762 messages/sec (2730 Mb/sec) > 196618 messages/sec (3072 Mb/sec) > 187254 messages/sec (2925 Mb/sec) > 180232 messages/sec (2816 Mb/sec) > 196616 messages/sec (3072 Mb/sec) > 189333 messages/sec (2958 Mb/sec) > 183507 messages/sec (2867 Mb/sec) > 196614 messages/sec (3072 Mb/sec) > 190656 messages/sec (2978 Mb/sec) > 202571 messages/sec (3165 Mb/sec) > 138785 messages/sec (2168 Mb/sec) > 131075 messages/sec (2048 Mb/sec) > > Jeff > Does this mean the bandwidth is 200-300 MByte/sec? What is the CPU utilization number (easy to sample with e.g. vstat)? -- MST From rep.nop at aon.at Mon Jun 27 14:13:46 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Mon, 27 Jun 2005 23:13:46 +0200 Subject: [openib-general] Re: IB Diagnositic Tools In-Reply-To: <1119895456.4848.353.camel@hal.voltaire.com> References: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com> <1119895456.4848.353.camel@hal.voltaire.com> Message-ID: <20050627211346.GA17011@aon.at> On Mon, Jun 27, 2005 at 02:04:16PM -0400, Hal Rosenstock wrote: >Hi Eitan, > >On Sat, 2005-06-25 at 15:25, Eitan Zahavi wrote: >> Following the discussion about the debug tools, I would like to >> propose using OpenSM Vendor layer as a common layer for developing the >> debug tools. Hal, This is kinda offtopic, but (iirc) i once stumbled over the issue of "port" vs. "mgmt port" [back then i had access to two 2-port cards] where you may have said something along the lines of \"There is clearly a bug for multi HCAs in osm_vendor_get_all_port_attr which is in the vendor layer. This needs to be fixed and is our problem. So I am close to being able to commit what I now have for this and fix this later (as there are other multi HCA issues).\" Just curious.. did somebody already have a chance to touch those or not? On a related note (just the same thing, i tend to think) /* "local ports" is(?) phys, shouldn't this exclude port 0 then ? */ I'm not too familiar with these kindof questions, which might be define in a spec, so any hint on this would be well received, at least from my part. anyone? Eitan? -- thank you, Bernhard From tduffy at sun.com Mon Jun 27 14:27:54 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 27 Jun 2005 14:27:54 -0700 Subject: [openib-general] Re: SDP: still getting sk_alloc() panic, any ideas? In-Reply-To: <20050627111721.A26240@topspin.com> References: <9d3b7de7050623130658aa92@mail.gmail.com> <20050627111721.A26240@topspin.com> Message-ID: <1119907675.6148.7.camel@duffman> On Mon, 2005-06-27 at 11:17 -0700, Libor Michalek wrote: > The problem is that each call to sk_alloc() is grabbing a reference to > the module, but it checks to make sure that there already is at least one > reference, if not the top BUG is triggered. In the case of the passive > connection there are no other references to the module. You can see that > the problem goes away if you open just one socket, even if you don't > listen on it, and then try the failing passive connect. When a socket is > created it actually grabs two references to the module, one at the sock > level and one at the sk level. The first reference at the sock level does > not trigger the BUG since it's through another code path. (try_module_get > vs. __module_get) This is why we only hit this during passive connect > to a system that has no active SDP sockets. > > Not sure the right way to fix this, maybe check to see if the socket > table size (dev_root_s.sk_entry) is greater then 0 in sdp_cm_req_handler() > before even performing the alloc... Hrm. That seems ugly. How about a patch to upstream changing sk_alloc() to use try_module_get(). Right now, we could make SDP depend on !CONFIG_MODULE_UNLOAD. That is even uglier! -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From rolandd at cisco.com Mon Jun 27 14:34:56 2005 From: rolandd at cisco.com (Roland Dreier) Date: Mon, 27 Jun 2005 14:34:56 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <6.2.1.2.2.20050627151532.04a0a480@exnane01.nane.netapp.com> (Thomas Talpey's message of "Mon, 27 Jun 2005 15:23:46 -0400") References: <91DB792C7985D411BEC300B40080D29CC35D0E@mtvex01.mtv.mtl.com> <6.2.1.2.2.20050627151532.04a0a480@exnane01.nane.netapp.com> Message-ID: <523br34h6n.fsf@topspin.com> Itamar> But the ATS will not solve the problem of "many to one". Itamar> What will the nfs module will do if the the result from Itamar> the ATS will be a list of "IP's" which only one of them is Itamar> has permission to the nfs ? ATS cant tell you who is the Itamar> source IP. Thomas> The NFS server exports will function just fine in such a Thomas> case. This is no different from any other multihomed Thomas> client, and /etc/exports can be configured appropriately. I'm not sure I understand this. At best, ATS can give you back a list of IPs. How do you decide which one to check against the exports? In a pure IP world, every packet from a multihomed client carries a source IP address. So a server can use getpeername() to determine which address a client is connecting from. This is fundamentally different from ATS. - R. From halr at voltaire.com Mon Jun 27 14:30:27 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 Jun 2005 17:30:27 -0400 Subject: [openib-general] Re: IB Diagnositic Tools In-Reply-To: <20050627211346.GA17011@aon.at> References: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com> <1119895456.4848.353.camel@hal.voltaire.com> <20050627211346.GA17011@aon.at> Message-ID: <1119907826.4848.486.camel@hal.voltaire.com> On Mon, 2005-06-27 at 17:13, Bernhard Fischer wrote: > Hal, > > This is kinda offtopic, Yes, this is different topic(s). > but (iirc) i once stumbled over the issue of > "port" vs. "mgmt port" [back then i had access to two 2-port cards] > where you may have said something along the lines of > \"There is clearly a bug for multi HCAs in osm_vendor_get_all_port_attr > which is in the vendor layer. This needs to be fixed and is our problem. > So I am close to being able to commit what I now have for this and fix > this later (as there are other multi HCA issues).\" > Just curious.. did somebody already have a chance to touch those or not? I don't think it has been fixed :-( I will need to refresh myself on this again. > On a related note (just the same thing, i tend to think) > /* "local ports" is(?) phys, shouldn't this exclude port 0 then ? */ I guess it depends on what local ports is being used for. This is a switch issue where there a 2 types of port 0s: base and enhanced. Neither of these are physical ports. Enhanced port 0 is like an HCA port. This is relevant as to PortInfo components and also which counters might also be available. In general, architecturally speaking, local ports would include all local ports whether physical IB ports or not so on a switch port 0 is included. Is port 0 causing an issue somewhere ? -- Hal > I'm not too familiar with these kindof questions, which might be define > in a spec, so any hint on this would be well received, at least from my > part. > > anyone? Eitan? From jlentini at netapp.com Mon Jun 27 15:12:24 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 27 Jun 2005 18:12:24 -0400 (EDT) Subject: [openib-general] RE: [PATCH][kdapl] Integrate dapl_hca_alloc/dapl_hca_free to dapl _provider.c In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35D0F@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35D0F@mtvex01.mtv.mtl.com> Message-ID: Ok. Removed in revision 2723. On Sun, 26 Jun 2005, Itamar Rabenstein wrote: >> uDAPL used the dapl_hca structure's ia_list to cleanup IA >> resources when the user space process forked. When the code was ported >> to the kernel, the list was retained, but the cleanup wasn't. >> Suppose a kDAPL consumer (another kernel module) is unloaded and >> forgets to close an IA. What do we want to have happen when the kDAPL >> module is unloaded? Don't we want the resources associated with any >> open IA's cleaned up? >> >> james >> > > NO!!! > > If other modules has a bug (forgot to close the IA) we are not going to hide > it. > ib_dat_provider module will not be able to go down (be unloaded) until the > bug will fixed. > Kernel is not a place where you clean other modules BUGS. > > Udapl need this feature not Kdapl. > > Please delete this list. > > Itamar > From ardavis at ichips.intel.com Mon Jun 27 15:14:53 2005 From: ardavis at ichips.intel.com (Arlin Davis) Date: Mon, 27 Jun 2005 15:14:53 -0700 Subject: [openib-general] Re: [PATCH] [udapl] fix build for x86_64 Message-ID: <42C07A5D.5000201@ichips.intel.com> Signed-off-by: Arlin Davis Index: dapl/openib/dapl_ib_dto.h =================================================================== --- dapl/openib/dapl_ib_dto.h (revision 2720) +++ dapl/openib/dapl_ib_dto.h (working copy) @@ -88,7 +88,7 @@ total_len = 0; wr.next = 0; wr.num_sge = 0; - wr.wr_id = (uint64_t)cookie; + wr.wr_id = (uint64_t)(unsigned long)cookie; wr.sg_list = ds_array_p; for (i = 0; i < segments; i++ ) { @@ -162,7 +162,7 @@ wr.opcode = op_type; wr.num_sge = 0; wr.send_flags = 0; - wr.wr_id = (uint64_t)cookie; + wr.wr_id = (uint64_t)(unsigned long)cookie; wr.sg_list = ds_array_p; total_len = 0; From jlentini at netapp.com Mon Jun 27 15:15:35 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 27 Jun 2005 18:15:35 -0400 (EDT) Subject: [openib-general] Re: [PATCH] cleanup dat provider registration In-Reply-To: <20050626104920.GA24144@lst.de> References: <20050617163211.GA19071@lst.de> <20050626104920.GA24144@lst.de> Message-ID: On Sun, 26 Jun 2005, Christoph Hellwig wrote: > On Wed, Jun 22, 2005 at 05:13:03PM -0400, James Lentini wrote: >> >> This is an excellent simplification. Committed in revision 2682 with a >> few minor modifications: >> >> - kept printouts in dat_registry_add_provider, >> dat_registry_remove_provider, and dat_registry_list_providers >> >> - updated printout in dat_ia_close (this wasn't something you >> changed) >> >> - removed parens around sizeof > > kernel style is to have parants around it, but all of the openib code > is different. well, let's keep it that way. > >> - removed space in front of labels > > lots of new kernel code uses the space, but again it's okay to stick > to the surrounding code. > >> The last two are for consistency with the coding style we've been >> using. If we've deviated from what is acceptable, let us know. >> >> Given this simplification, I can think of a few more changes: >> >> - rename api.c to registry.c > > Note yet. The code will get some major surgery still, and as part of > that split into different files again maybe, just on very different > boundaries. > >> - remove all "dictionary" references: rename dat_dictionary_search to >> dat_provider_list_search, rename struct dat_dictionary_entry to >> struct dat_provider_list_entry, rename >> dat_dictionary_key_is_equal() to dat_provider_info_is_equal() > > I'll plan bigger changes in that area that will kill the dictionary > term, so a simple search and replace is probably not worth it. Oh well, I already did the cleanup. I'll add an item to the TODO list to update the API with a register/callback interface for accessing the provider. From tduffy at sun.com Mon Jun 27 15:18:10 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 27 Jun 2005 15:18:10 -0700 Subject: [openib-general] [Fwd: Returned mail: see transcript for details] In-Reply-To: <42C02E8B.1070506@ichips.intel.com> References: <1118857749.3845.8.camel@duffman> <42C02E8B.1070506@ichips.intel.com> Message-ID: <1119910690.25388.0.camel@duffman> On Mon, 2005-06-27 at 09:51 -0700, Sean Hefty wrote: > Assuming that these haven't been applied yet, I will try to get to this > today or tomorrow. Great, I don't think they have. Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Mon Jun 27 15:25:40 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 27 Jun 2005 15:25:40 -0700 Subject: [openib-general] [PATCH][RFC] nfsordma: initial port of nfsrdma to 2.6 and james'sss kdapl Message-ID: <1119911140.25388.6.camel@duffman> I have done some initial work to port nfsrdma to 2.6 and to James's kDAPL. This builds now inside the kernel. You will need to follow the kDAPL directions first to put that in your kernel tree, then slap this patch over top of that. So you have 2.6.12 + svn drivers/infiniband + kdapl from james's tree + this patch to get it to build. Oh you will also need to patch the rpc header to get it to build. I think it is time to open up a tree in openib repository. Tom, is netapp willing to GPL this code? Signed-off-by: Tom Duffy Index: drivers/infiniband/ulp/nfsrdma/rdma_kdapl.c =================================================================== --- drivers/infiniband/ulp/nfsrdma/rdma_kdapl.c (revision 0) +++ drivers/infiniband/ulp/nfsrdma/rdma_kdapl.c (revision 0) @@ -0,0 +1,1353 @@ +/* + * Copyright (c) 2003, 2004, Network Appliance, Inc. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * Neither the name of the Network Appliance, Inc. nor the names of + * its contributors may be used to endorse or promote products + * derived from this software without specific prior written + * permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* + * rdma_kdapl.c + * + * Implements an interface to kDAPL that is used by the RPC RDMA + * transport. Encapsulates the major functions managing: + * o adapters + * o endpoints + * o connections + * o buffer memory + * + * This file does not depend on any modified Linux kernel code. + * It depends on kDAPL, and some basic Linux facilities. + */ + +#include /* kmalloc, kfree */ +#include /* tasklet */ +#include /* num_physpages */ + +#include "rdma_kdapl.h" + +/* + * Globals/Macros + */ + +#undef Dprintk +#if RPCRDMA_DEBUG +#define Dprintk(cond, x) if (rdma_kdapl_debug >= cond) printk x +int rdma_kdapl_debug; +#else +#define Dprintk(n, x) +#endif + +/* + * local types and constants + */ + +/* handle replies in tasklet context, using a single, global list */ +static void rdma_run_tasklet(unsigned long data); +DECLARE_TASKLET(rdma_tasklet_g, rdma_run_tasklet, 0UL); + +static spinlock_t rdma_tk_lock_g = SPIN_LOCK_UNLOCKED; +static LIST_HEAD(rdma_tasklets_g); + +/* + * local function prototypes + */ +static void rdma_clean_evd(struct dat_evd *, char *); +static void rdma_async_evd_upcall(void *, const struct dat_event *, boolean_t); +static void rdma_event_evd_upcall(void *, const struct dat_event *, boolean_t); +static void rdma_conn_evd_upcall(void *, const struct dat_event *, boolean_t); + +static inline void +rdma_schedule_tasklet(rdma_rep_t *rep) +{ + unsigned int lock_flags; + + spin_lock_irqsave(&rdma_tk_lock_g, lock_flags); + list_add_tail(&rep->rr_list, &rdma_tasklets_g); + spin_unlock_irqrestore(&rdma_tk_lock_g, lock_flags); + tasklet_schedule(&rdma_tasklet_g); +} + + +static inline u32 +rdma_wait_conn(rdma_ep_t *ep, rdma_ia_t *ia, unsigned long to) +{ + /* TBD handle timeout */ + wait_event_interruptible(ep->rep_connect_wait, ep->rep_connected == 1); + /* TBD handle errors here? */ + return DAT_SUCCESS; +} + +static inline u32 +rdma_wait_disconn(rdma_ep_t *ep, rdma_ia_t *ia, unsigned long to) +{ + /* TBD handle timeout */ + wait_event_interruptible(ep->rep_connect_wait, ep->rep_connected != 1); + /* TBD handle errors here? */ + return DAT_SUCCESS; +} + +#if RPCRDMA_DEBUG +static const char * +ststatus(unsigned int status) +{ + static const char * const ststrings[] = { + "success", + "flushed", + "local length error", + "local endpoint error", + "local protection error", + "bad response", + "remote access error", + "remote responder error", + "transport error", + "receiver not ready", + "partial packet", + "rmr operation error" + + }; + if (status <= 11) + return ststrings[status]; + return "unknown"; +} +#endif + +/* + * Exported functions. + */ + +/* + * Open and initialize an Interface Adapter. + * o initializes fields of rdma_ia_t, including + * interface and provider attributes and protection zone. + */ +rdma_ia_t * +rdma_ia_init(rdma_ia_t *ia, char *ia_name, int memreg) +{ + u32 datstatus; + struct dat_upcall_object upcall; + DAT_REGION_DESCRIPTION region; + DAT_RMR_CONTEXT *rmr_contextp; + enum dat_mem_priv_flags mem_priv; + u64 pgcount; + + memset(ia, 0, sizeof *ia); + /* open IA */ + datstatus = dat_ia_open(ia_name, 4, + &ia->ri_async_evd_handle, + &ia->ri_ia_handle); + if (datstatus != DAT_SUCCESS) { + Dprintk(0, ("rdma_ia_init: dat_ia_open failed on %s, status 0x%x\n", + ia_name, datstatus)); + goto out; + } + + /* j.i.c., re-vector the async evd. Not fatal. */ + upcall.instance_data = ia; + upcall.upcall_func = rdma_async_evd_upcall; + datstatus = dat_evd_modify_upcall(ia->ri_async_evd_handle, + DAT_UPCALL_SINGLE_INSTANCE, &upcall); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ia_init: dat_evd_modify_upcall (async) failed: 0x%x\n", + datstatus)); + + /* get IA attributes */ + datstatus = dat_ia_query(ia->ri_ia_handle, + &ia->ri_async_evd_handle, + &ia->ri_ia_attr, + &ia->ri_pv_attr); + if (datstatus != DAT_SUCCESS) { + Dprintk(0, ("rdma_ia_init: dat_ia_query failed: 0x%x\n", + datstatus)); + goto out; + } + + /* TBD provider or endpoint attributes to check? */ + + /* create protection zone for IA */ + datstatus = dat_pz_create(ia->ri_ia_handle, &ia->ri_pz_handle); + if (datstatus != DAT_SUCCESS) { + Dprintk(0, ("rdma_ia_init: dat_pz_create failed: 0x%x\n", + datstatus)); + goto out; + } + + /* + * Optionally register an underlying physical identity mapping in + * order to do high performance dat_rmr_bind. This base registration + * is protected from remote access - that is enabled only by binding + * for the specific bytes targeted during each RPC operation, and + * revoked after the corresponding completion similar to a storage + * adapter. + */ + if (memreg > 1) { + /* round up pgcount by 1MB */ + pgcount = ((u64) num_physpages + 255ULL) & ~255ULL, + region.for_va = (void *)0; + mem_priv = DAT_MEM_PRIV_LOCAL_READ_FLAG | + DAT_MEM_PRIV_LOCAL_WRITE_FLAG; + rmr_contextp = NULL; +#if RPCRDMA_DEBUG + if (memreg == 4) { + mem_priv = DAT_MEM_PRIV_ALL_FLAG; + rmr_contextp = &ia->ri_bind_rmr; + } +#endif + datstatus = dat_lmr_kcreate(ia->ri_ia_handle, + DAT_MEM_TYPE_PHYSICAL, + region, + pgcount, + ia->ri_pz_handle, + mem_priv, + DAT_MEM_OPTIMIZE_DONT_CARE, + &ia->ri_bind_mem, + &ia->ri_bind_iov.lmr_context, + rmr_contextp, + &ia->ri_bind_iov.segment_length, + &ia->ri_bind_iov.virtual_address); + if (datstatus != DAT_SUCCESS) { + printk("rdma_ia_init: dat_lmr_kcreate for " + "fast register failed with 0x%x\n\t" + "Will continue with degraded performance\n", + datstatus); + datstatus = DAT_SUCCESS; + memreg = 1; + } else { + Dprintk(1, ("rdma_ia_init: preregistered %llu physical" + " pages for rmr creation (0->0x%llx, %lluMB)\n", + ia->ri_bind_iov.segment_length >> PAGE_SHIFT, + ia->ri_bind_iov.segment_length - 1, + ia->ri_bind_iov.segment_length >> 20)); + } + } + + /* Else will do lmr_kcreate/lmr_free for each chunk */ + ia->ri_memreg_strategy = memreg; +out: + if (datstatus != DAT_SUCCESS) { + if (ia->ri_ia_handle != NULL) { + rdma_ia_close(ia); + } + ia = NULL; + } + return ia; +} + +/* + * Clean up/close an IA. + * o if event handles and PZ have been initialized, free them. + * o close the IA + */ +void +rdma_ia_close(rdma_ia_t *ia) +{ + u32 datstatus; + + Dprintk(1, ("rdma_ia_close: entering\n")); + if (ia->ri_bind_mem != NULL) { + datstatus = dat_lmr_free(ia->ri_bind_mem); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ia_close: dat_lmr_free(phys) returned 0x%x\n", + datstatus)); + } + if (ia->ri_pz_handle != NULL) { + datstatus = dat_pz_free(ia->ri_pz_handle); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ia_close: dat_pz_free returned 0x%x\n", + datstatus)); + } + if (ia->ri_async_evd_handle != NULL) { + rdma_clean_evd(ia->ri_async_evd_handle, "async"); + /* Do not destroy - provider owns this. */ + } + + if (ia->ri_ia_handle != NULL) { + datstatus = dat_ia_close(ia->ri_ia_handle, DAT_CLOSE_ABRUPT_FLAG); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ia_close: dat_ia_close returned 0x%x\n", + datstatus)); + } +} + +/* + * Initialize default attributes for an endpoint, in preparation for create. + */ +void +rdma_ep_default_attr(rdma_ep_t *ep, rdma_ia_t *ia, + struct rdma_create_data_internal *cdata) +{ + memset(&ep->rep_attr, 0, sizeof(ep->rep_attr)); + ep->rep_attr.max_message_size = max(cdata->inline_rsize, cdata->inline_wsize); + ep->rep_attr.max_message_size += MAX_RPCHDR + MAX_RDMAHDR; + ep->rep_attr.max_rdma_size = max(cdata->rsize, cdata->wsize); + ep->rep_attr.max_recv_dtos = cdata->max_requests; + ep->rep_attr.max_request_dtos = cdata->max_requests; + if (ia->ri_bind_mem != NULL && ia->ri_memreg_strategy != 4) { + /* Add room for rmr_binds+unbinds - overkill! */ + ep->rep_attr.max_request_dtos++; + ep->rep_attr.max_request_dtos *= (2 * RDMA_MAX_SEGS); + } + if (ep->rep_attr.max_request_dtos > ia->ri_ia_attr.max_dto_per_ep) { + ep->rep_attr.max_request_dtos = ia->ri_ia_attr.max_dto_per_ep; + } + ep->rep_attr.max_recv_iov = 1; + ep->rep_attr.max_request_iov = (cdata->padding ? 4 : 2); + ep->rep_attr.max_rdma_read_in = cdata->max_requests * (RDMA_MAX_SEGS/2); + if (ep->rep_attr.max_rdma_read_in > ia->ri_ia_attr.max_rdma_read_per_ep_in) { + ep->rep_attr.max_rdma_read_in = ia->ri_ia_attr.max_rdma_read_per_ep_in; + } + ep->rep_attr.max_rdma_read_out = 0; /* always */ + ep->rep_attr.recv_completion_flags = DAT_COMPLETION_DEFAULT_FLAG; + ep->rep_attr.request_completion_flags = DAT_COMPLETION_SUPPRESS_FLAG; + + /* set trigger for requesting send completion */ + ep->rep_cqinit = ep->rep_attr.max_request_dtos - 1; + if (ia->ri_bind_mem != NULL && ia->ri_memreg_strategy != 4) + ep->rep_cqinit -= RDMA_MAX_SEGS; + if (ep->rep_cqinit <= 2) + ep->rep_cqinit = 0; + INIT_CQCOUNT(ep); + spin_lock_init(&ep->rep_postlock); + ep->rep_ia = ia; + init_waitqueue_head(&ep->rep_connect_wait); +} + +/* + * Create unconnected endpoint. + */ +rdma_ep_t * +rdma_ep_create(rdma_ep_t *ep, rdma_ia_t *ia, + struct rdma_create_data_internal *cdata) +{ + struct dat_upcall_object upcall; + u32 datstatus; + struct dat_ep_param params; + + /* + * Create a single evd for receive dto and rmr_bind (only ever + * care about unbind, really). Send completions are suppressed. + * Use single threaded upcalls to maintain ordering. Operation + * is parallel via tasklet. + */ + upcall.instance_data = NULL; + upcall.upcall_func = rdma_event_evd_upcall; + datstatus = dat_evd_kcreate(ia->ri_ia_handle, + ep->rep_attr.max_recv_dtos + + ep->rep_attr.max_request_dtos + 1, + DAT_UPCALL_SINGLE_INSTANCE, + &upcall, + DAT_EVD_DTO_FLAG| + DAT_EVD_RMR_BIND_FLAG, + &ep->rep_evd_handle); + if (datstatus != DAT_SUCCESS) { + Dprintk(0, ("rdma_ep_create: dat_evd_kcreate (event) failed: 0x%x\n", + datstatus)); + return NULL; + } + + /* + * Create a second evd for connection events. This avoids any + * performance issues with completion evd sharing. + */ + upcall.instance_data = ep; + upcall.upcall_func = rdma_conn_evd_upcall; + datstatus = dat_evd_kcreate(ia->ri_ia_handle, + 4, + DAT_UPCALL_SINGLE_INSTANCE, + &upcall, + DAT_EVD_CONNECTION_FLAG, + &ep->rep_conn_handle); + if (datstatus != DAT_SUCCESS) { + Dprintk(0, ("rdma_ep_create: dat_evd_kcreate (conn) failed: 0x%x\n", + datstatus)); + rdma_ep_destroy(ep, ia); + return NULL; + } + + datstatus = dat_ep_create(ia->ri_ia_handle, + ia->ri_pz_handle, + ep->rep_evd_handle, /* recv */ + ep->rep_evd_handle, /* rqst */ + ep->rep_conn_handle, /* conn */ + &ep->rep_attr, &ep->rep_handle); + + if (datstatus != DAT_SUCCESS) { + Dprintk(0, ("dat_ep_create failed with 0x%x\n", datstatus)); + rdma_ep_destroy(ep, ia); + return NULL; + } + /* Get the actual ep attributes */ + datstatus = dat_ep_query(ep->rep_handle, ¶ms); + if (datstatus == DAT_SUCCESS) { + ep->rep_attr = params.ep_attr; + Dprintk(1, ("rdma_ep_create: max: msg %lld, rdma %lld; " + "dtos: send %d recv %d; iovs: send %d recv %d; " + "rdma reads: in %d out %d\n", + ep->rep_attr.max_message_size, + ep->rep_attr.max_rdma_size, + ep->rep_attr.max_request_dtos, + ep->rep_attr.max_recv_dtos, + ep->rep_attr.max_request_iov, + ep->rep_attr.max_recv_iov, + ep->rep_attr.max_rdma_read_in, + ep->rep_attr.max_rdma_read_out)); + + if (ep->rep_attr.max_recv_dtos < cdata->max_requests) { + Dprintk(0, ("rdma_ep_create: reducing " + "max_requests to %d to match available DTOs\n", + ep->rep_attr.max_recv_dtos)); + cdata->max_requests = ep->rep_attr.max_recv_dtos; + } + /* TBD presume sizes, send DTOs, rdma reads etc ok */ + } else { + Dprintk(0, ("dat_ep_query failed with 0x%x\n", datstatus)); + } + return ep; +} + +/* + * rdma_ep_destroy + * + * Disconnect and destroy endpoint. After this, the only + * valid operations on the ep are to free it (if dynamically + * allocated) or re-create it. + * + * Depending on the caller's error handling, the endpoint + * could leak if this function fails. + */ +int +rdma_ep_destroy(rdma_ep_t *ep, rdma_ia_t *ia) +{ + u32 datstatus; + + Dprintk(1, ("rdma_ep_destroy: entering, connected is %d\n", + ep->rep_connected)); + + if (ep->rep_handle) { + datstatus = rdma_ep_disconnect(ep, ia); + Dprintk(1 && datstatus != DAT_SUCCESS, + ("rdma_ep_destroy: rdma_ep_disconnect returned 0x%x\n", + datstatus)); + (void) dat_ep_reset(ep->rep_handle); + } + + ep->rep_func = 0; + + /* padding - should actually be done in rdma_buffer_destroy... */ + if (ep->rep_padhandle) { + rdma_deregister_internal(ep->rep_padhandle); + ep->rep_padhandle = NULL; + } + + if (ep->rep_handle) { + datstatus = dat_ep_free(ep->rep_handle); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ep_destroy: dat_ep_free returned 0x%x\n", + datstatus)); + } + + if (ep->rep_conn_handle) { + rdma_clean_evd(ep->rep_conn_handle, "connection"); + datstatus = dat_evd_free(ep->rep_conn_handle); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ep_destroy: dat_evd_free (conn) returned 0x%x\n", + datstatus)); + } + + rdma_clean_evd(ep->rep_evd_handle, "completion"); + datstatus = dat_evd_free(ep->rep_evd_handle); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ep_destroy: dat_evd_free returned 0x%x\n", datstatus)); + + return (datstatus != DAT_SUCCESS); +} + +/* + * Connect unconnected endpoint. + * + * TBD: + * o check for unconnected state; fail if not + * o pass in, or set DAT_CONN_QUAL + * o don't use infinite timeout + * o need better error granularity for various failure events? + */ +int +rdma_ep_connect(rdma_ep_t *ep, rdma_ia_t *ia, int nowait) +{ + u32 datstatus; + int retry_count = 0; + +retry: + ep->rep_event_num = 0; + ep->rep_connected = -1; + datstatus = dat_ep_connect(ep->rep_handle, + (struct sockaddr *)&ep->rep_remote_addr, + ep->rep_server_port, + RDMA_CONNECT_TIMEOUT, + 0, NULL, /* no private data */ + DAT_QOS_BEST_EFFORT, + DAT_CONNECT_DEFAULT_FLAG); + if (datstatus == DAT_SUCCESS) { + /* set during reconnect attempts */ + if (nowait) { + return 0; + } + + rdma_wait_conn(ep, ia, DAT_TIMEOUT_MAX); + /* + * Check state. A non-peer reject indicates no listener + * (ECONNREFUSED), which may be a transient state. All + * others indicate a transport condition which has already + * undergone a best-effort. + */ + if (ep->rep_event_num == DAT_CONNECTION_EVENT_NON_PEER_REJECTED + && ++retry_count <= RDMA_CONNECT_RETRY_MAX) { + goto retry; + } + if (ep->rep_event_num != DAT_CONNECTION_EVENT_ESTABLISHED) { + Dprintk(0, ("rdma_ep_connect: failed to connect, event 0x%x\n", + ep->rep_event_num)); + datstatus = DAT_TIMEOUT_EXPIRED; + } else { + RDMA_ASSERT(ep->rep_connected == 1, + "rdma_ep_connect connection not indicated"); + Dprintk(1, ("rdma_ep_connect: connected\n")); + } + } else { + Dprintk(0, ("dat_ep_connect failed with 0x%x\n", datstatus)); + } + ep->rep_event_num = 0; + return (datstatus != DAT_SUCCESS); +} + +/* + * rdma_ep_disconnect + * + * This is separate from destroy to facilitate the ability + * to reconnect without recreating the endpoint. + * + * This call is not reentrant, and must not be made in parallel + * on the same endpoint. + */ +int +rdma_ep_disconnect(rdma_ep_t *ep, rdma_ia_t *ia) +{ + u32 datstatus; + + rdma_clean_evd(ep->rep_evd_handle, "completion"); + ep->rep_event_num = 0; + datstatus = dat_ep_disconnect(ep->rep_handle, DAT_CLOSE_ABRUPT_FLAG); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ep_disconnect: dat_ep_disconnect status 0x%x\n", + datstatus)); + if (datstatus == DAT_SUCCESS) { + /* returns without wait if not connected */ + rdma_wait_disconn(ep, ia, DAT_TIMEOUT_MAX); + Dprintk(1, ("rdma_ep_disconnect: after wait, %sconnected\n", + (ep->rep_connected == 1) ? "still " : "dis")); + } + ep->rep_connected = 0; + /* ignore event number */ + return (datstatus != DAT_SUCCESS); +} + +/* + * re-connect failed endpoint + * + * TBD: implement synchronization on this to avoid + * multiple callers (RPC mechanism may be enough) + */ +int +rdma_ep_reconnect(rdma_ep_t *ep, rdma_ia_t *ia) +{ + u32 datstatus; + (void) rdma_ep_disconnect(ep, ia); + datstatus = dat_ep_reset(ep->rep_handle); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ep_reconnect: dat_ep_reset failed 0x%x\n", + datstatus)); + rdma_clean_evd(ep->rep_evd_handle, "completion"); + datstatus = rdma_ep_connect(ep, ia, 1); + Dprintk(1, ("rdma_ep_connect: rdma_ep_connect status 0x%x\n", + datstatus)); + return (datstatus != DAT_SUCCESS); +} + +/* + * Drain any EVD, prior to teardown. + */ +static void +rdma_clean_evd(struct dat_evd *hdl, char *which) +{ + struct dat_event event; + int count = 0; + + while (dat_evd_dequeue(hdl, &event) == DAT_SUCCESS) { + ++count; + } + Dprintk(0 && count, + ("rdma_clean_evd: flushed %d %s events (last 0x%x)\n", + count, which, event.event_number)); +} + +/* + * Initialize buffer memory + */ +rdma_buffer_t * +rdma_buffer_create(rdma_buffer_t *buf, rdma_ep_t *ep, + rdma_ia_t *ia, struct rdma_create_data_internal *cdata) +{ + u32 datstatus; + char *p; + int i, len; + + buf->rb_max_requests = cdata->max_requests; + spin_lock_init(&buf->rb_lock); + atomic_set(&buf->rb_credits, 1); + + /* Need to allocate: + * 1. arrays for send and recv pointers + * 2. arrays of rdma_req_t to fill in pointers + * 3. array of rdma_rep_t for replies + * 4. padding, if any + * 5. rmr's, if any + * Send/recv buffers in req/rep need to be registered for LMR + */ + + len = buf->rb_max_requests * (sizeof(rdma_req_t *) + sizeof(rdma_rep_t *)); +#if RPCRDMA_DEBUG + if (ia->ri_memreg_strategy != 4) +#endif + if (ia->ri_bind_mem) { + len += (buf->rb_max_requests * RDMA_MAX_SEGS * sizeof (struct rdma_rmr_entry)); + } + len += cdata->padding; + + /* allocate 1, 4 and 5 in one shot */ + p = kmalloc(len, GFP_KERNEL); + if (p == NULL) { + Dprintk(0, ("rdma_buffer_create: req_t/rep_t/pad kmalloc(%d) failed\n", len)); + goto outfail; + } + memset(p, 0, len); + buf->rb_pool = p; /* for freeing it later */ + + buf->rb_send_bufs = (rdma_req_t **) p; + p = (char *) &buf->rb_send_bufs[buf->rb_max_requests]; + buf->rb_recv_bufs = (rdma_rep_t **) p; + p = (char *) &buf->rb_recv_bufs[buf->rb_max_requests]; + + /* + * Register the zeroed pad buffer, if any. + */ + if (cdata->padding && rdma_register_internal(ia, p, cdata->padding, + &ep->rep_padhandle, &ep->rep_pad)) { + goto outfail; + } + p += cdata->padding; + + /* + * Allocate the rmr's for fast chunk registration. + * We "cycle" the rmr's in order to minimize rmr_context reuse, + * and also reduce unbind-to-bind collision. + */ + INIT_LIST_HEAD(&buf->rb_rmrs); +#if RPCRDMA_DEBUG + if (ia->ri_memreg_strategy != 4) +#endif + if (ia->ri_bind_mem) { + struct rdma_rmr_entry *r = (struct rdma_rmr_entry *)p; + /* Allocate one extra request's worth, for full cycling */ + for (i = (buf->rb_max_requests + 1) * RDMA_MAX_SEGS; i; i--) { + datstatus = dat_rmr_create(ia->ri_pz_handle, + &r->rmr_handle); + if (datstatus != DAT_SUCCESS) { + Dprintk(0, ("rdma_buffer_create: dat_rmr_create" + " failed with 0x%x\n", + datstatus)); + goto outfail; + } + list_add(&r->rmr_freelist, &buf->rb_rmrs); + ++r; + } + } + + /* + * Allocate/init the request/reply buffers. Doing this + * using kmalloc for now -- one for each buf. + */ + for (i = 0; i < buf->rb_max_requests; i++) { + rdma_req_t *req; + rdma_rep_t *rep; + /* Allocate an extra RPCHDR to satisfy RPC allocate() call */ + len = cdata->inline_wsize + (2 * MAX_RPCHDR); + req = kmalloc(len + sizeof (rdma_req_t), GFP_KERNEL); + if (req == NULL) { + Dprintk(0, ("rdma_buffer_create: request " + "buffer %d kmalloc failed\n", i)); + goto outfail; + } + memset(req, 0, sizeof (rdma_req_t)); + buf->rb_send_bufs[i] = req; + buf->rb_send_bufs[i]->rl_buffer = buf; + + if (rdma_register_internal(ia, req->rl_base, len + MAX_RDMAHDR, + &buf->rb_send_bufs[i]->rl_handle, + &buf->rb_send_bufs[i]->rl_iov)) { + goto outfail; + } + /* don't include rdma header or extra RPC header in space */ + buf->rb_send_bufs[i]->rl_size = len - MAX_RPCHDR; + + len = cdata->inline_rsize + MAX_RPCHDR; + rep = kmalloc(len + sizeof (rdma_rep_t), GFP_KERNEL); + if (rep == NULL) { + Dprintk(0, ("rdma_buffer_create: reply " + "buffer %d kmalloc failed\n", i)); + goto outfail; + } + memset(rep, 0, sizeof (rdma_rep_t)); + buf->rb_recv_bufs[i] = rep; + buf->rb_recv_bufs[i]->rr_buffer = buf; + init_waitqueue_head(&rep->rr_unbind); + + if (rdma_register_internal(ia, rep->rr_base, len + MAX_RDMAHDR, + &buf->rb_recv_bufs[i]->rr_handle, + &buf->rb_recv_bufs[i]->rr_iov)) { + goto outfail; + } + } + /* done */ + return buf; + + outfail: + if (buf) { + rdma_buffer_destroy(buf); + } + return NULL; +} + +/* + * Destroy/cleanup buffer/lmr memory. Need to deal with + * partial initialization, so it's callable from failed create. + * Must be called before destroying endpoint, as registrations + * reference it. + * + * TBD: fully account for memory that may belong to the H/W. + * means working with the EVDs. There should be events + * returning memory after the connection was closed (which must + * be done BEFORE this call is made). + */ +void +rdma_buffer_destroy(rdma_buffer_t *bufp) +{ + u32 datstatus; + int i; + + /* clean up in reverse order from create + * 1. recv lmr memory (lmr free, then kfree) + * 1a. bind rmr memory + * 2. send lmr memory (lmr free, then kfree) + * 3. padding (if any) [moved to rdma_ep_destroy] + * 4. arrays + */ + Dprintk(1, ("rdma_buffer_destroy: entering\n")); + /* _rdma_dump_buffer(bufp); */ + + for (i = 0; i < bufp->rb_max_requests; i++) { + if (bufp->rb_recv_bufs && bufp->rb_recv_bufs[i]) { + rdma_deregister_internal(bufp->rb_recv_bufs[i]->rr_handle); + kfree(bufp->rb_recv_bufs[i]); + } + if (bufp->rb_send_bufs && bufp->rb_send_bufs[i]) { + while (!list_empty(&bufp->rb_rmrs)) { + struct rdma_rmr_entry *r; + r = list_entry(bufp->rb_rmrs.next, + struct rdma_rmr_entry, rmr_freelist); + list_del(&r->rmr_freelist); + datstatus = dat_rmr_free(r->rmr_handle); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_buffer_destroy: dat_rmr_free " + "failed with 0x%x\n", + datstatus)); + } + rdma_deregister_internal(bufp->rb_send_bufs[i]->rl_handle); + kfree(bufp->rb_send_bufs[i]); + } + } + + if (bufp->rb_pool) { + kfree(bufp->rb_pool); + } +} + +/* + * Get a set of request/reply buffers. + * + * Reply buffer (if needed) is attached to send buffer upon return. + * Rule: + * rb_send_index and rb_recv_index MUST always be pointing to the + * *next* available buffer (non-NULL). They are incremented after + * removing buffers, and decremented *before* returning them. + */ +rdma_req_t * +rdma_buffer_get(rdma_buffer_t *buffers) +{ + rdma_req_t *req; + unsigned int lock_flags; + + spin_lock_irqsave(&buffers->rb_lock, lock_flags); + RDMA_ASSERT(buffers->rb_send_index < buffers->rb_max_requests, + "rdma_buffer_get over max_requests"); + req = buffers->rb_send_bufs[buffers->rb_send_index]; + if (buffers->rb_send_index < buffers->rb_recv_index) { + Dprintk(1, ("rdma_buffer_get: %d extra receives outstanding (ok)\n", + buffers->rb_recv_index - buffers->rb_send_index)); + req->rl_reply = NULL; + } else { + req->rl_reply = buffers->rb_recv_bufs[buffers->rb_recv_index]; + buffers->rb_recv_bufs[buffers->rb_recv_index++] = NULL; + } + buffers->rb_send_bufs[buffers->rb_send_index++] = NULL; + if (!list_empty(&buffers->rb_rmrs)) { + int i = RDMA_MAX_SEGS - 1; + do { + struct rdma_rmr_entry *r; + r = list_entry(buffers->rb_rmrs.next, + struct rdma_rmr_entry, rmr_freelist); + list_del(&r->rmr_freelist); + req->rl_seg_handles[i].rl_rmr = r; + } while (--i >= 0); + } + spin_unlock_irqrestore(&buffers->rb_lock, lock_flags); + return req; +} + +/* + * Put request/reply buffers back into pool. + * Pre-decrement counter/array index. + */ +void +rdma_buffer_put(rdma_req_t *req) +{ + rdma_buffer_t *buffers = req->rl_buffer; + unsigned int lock_flags; + + RDMA_ASSERT(req->rl_nsegs == 0, "rdma_buffer_put with active RDMA"); + spin_lock_irqsave(&buffers->rb_lock, lock_flags); + buffers->rb_send_bufs[--buffers->rb_send_index] = req; + req->rl_niovs = req->rl_nsegs = 0; + if (req->rl_reply) { + buffers->rb_recv_bufs[--buffers->rb_recv_index] = req->rl_reply; + init_waitqueue_head(&req->rl_reply->rr_unbind); + req->rl_reply->rr_func = NULL; + req->rl_reply = NULL; + } + if (req->rl_seg_handles[0].rl_rmr) { + /* + * Cycle rmr's back in reverse order, and "spin" them. + * This delays and scrambles reuse as much as possible. + */ + int i = 1; + do { + list_add_tail(&req->rl_seg_handles[i].rl_rmr->rmr_freelist, + &buffers->rb_rmrs); + req->rl_seg_handles[i].rl_rmr = NULL; + } while (++i < RDMA_MAX_SEGS); + list_add_tail(&req->rl_seg_handles[0].rl_rmr->rmr_freelist, + &buffers->rb_rmrs); + req->rl_seg_handles[0].rl_rmr = NULL; + } + spin_unlock_irqrestore(&buffers->rb_lock, lock_flags); +} + +/* + * Recover reply buffers from pool. + * This happens when recovering from error conditions. + * Post-increment counter/array index. + */ +void +rdma_recv_buffer_get(rdma_req_t *req) +{ + rdma_buffer_t *buffers = req->rl_buffer; + unsigned int lock_flags; + + spin_lock_irqsave(&buffers->rb_lock, lock_flags); + if (buffers->rb_recv_index < buffers->rb_max_requests) { + req->rl_reply = buffers->rb_recv_bufs[buffers->rb_recv_index]; + buffers->rb_recv_bufs[buffers->rb_recv_index++] = NULL; + } + spin_unlock_irqrestore(&buffers->rb_lock, lock_flags); +} + +/* + * Put reply buffers back into pool when not attached to + * request. This happens in error conditions, and when + * aborting unbinds. Pre-decrement counter/array index. + */ +void +rdma_recv_buffer_put(rdma_rep_t *rep) +{ + rdma_buffer_t *buffers = rep->rr_buffer; + unsigned int lock_flags; + + rep->rr_func = NULL; + spin_lock_irqsave(&buffers->rb_lock, lock_flags); + buffers->rb_recv_bufs[--buffers->rb_recv_index] = rep; + spin_unlock_irqrestore(&buffers->rb_lock, lock_flags); +} + +/* + * Wrappers for internal-use kmalloc memory registration, used by buffer code. + */ +int +rdma_register_internal(rdma_ia_t *ia, void *va, int len, + struct dat_lmr **handlep, + struct dat_lmr_triplet *triplet) +{ + u32 datstatus; + DAT_REGION_DESCRIPTION region; + + if (ia->ri_bind_mem) { + *handlep = NULL; + triplet->lmr_context = ia->ri_bind_iov.lmr_context; + triplet->segment_length = len; + triplet->virtual_address = __pa(va); + return 0; + } + + region.for_va = (void *)va; + datstatus = dat_lmr_kcreate(ia->ri_ia_handle, + DAT_MEM_TYPE_VIRTUAL, + region, + (u64) len, + ia->ri_pz_handle, + DAT_MEM_PRIV_LOCAL_READ_FLAG | + DAT_MEM_PRIV_LOCAL_WRITE_FLAG, + DAT_MEM_OPTIMIZE_DONT_CARE, + handlep, + &triplet->lmr_context, + NULL, /* no need for rmr context */ + &triplet->segment_length, + &triplet->virtual_address); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_register_internal: dat_lmr_kcreate failed with 0x%x\n", + datstatus)); + return (datstatus != DAT_SUCCESS); +} + +int +rdma_deregister_internal(void *handle) +{ + u32 datstatus; + + if (handle == NULL) { + datstatus = DAT_SUCCESS; + } else { + datstatus = dat_lmr_free(handle); + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_deregister_internal: dat_lmr_free failed with 0x%x\n", + datstatus)); + } + return (datstatus != DAT_SUCCESS); +} + +/* + * Wrappers for chunk registration, shared by read/write chunk code. + */ +int +rdma_register_external(rdma_mr_iov *seg, + rdma_mr_handle *handlep, + DAT_RMR_CONTEXT *rmr_contextp, + int writing, rdma_xprt_t *r_xprt) +{ + u32 datstatus; + rdma_ia_t *ia = r_xprt->rx_ia; + enum dat_mem_priv_flags mem_priv = (writing ? + DAT_MEM_PRIV_REMOTE_WRITE_FLAG : + DAT_MEM_PRIV_REMOTE_READ_FLAG); + unsigned int lock_flags; + +#if RPCRDMA_DEBUG + /* Persistent registration */ + if (ia->ri_memreg_strategy == 4) { + *rmr_contextp = ia->ri_bind_rmr; + return 0; + } +#endif + + /* Fast registration using rmr_bind on physical lmr */ + if (ia->ri_bind_mem != NULL) { + DAT_RMR_COOKIE cookie; + struct dat_lmr_triplet triplet; + cookie.as_ptr = 0; + triplet.virtual_address = seg->mr_base; + triplet.segment_length = seg->mr_len; + triplet.lmr_context = ia->ri_bind_iov.lmr_context; + spin_lock_irqsave(&r_xprt->rx_ep.rep_postlock, lock_flags); + DECR_CQCOUNT(&r_xprt->rx_ep); + datstatus = dat_rmr_bind(handlep->rl_rmr->rmr_handle, + handlep->rl_lmr, &triplet, + mem_priv, r_xprt->rx_ep.rep_handle, + cookie, DAT_COMPLETION_SUPPRESS_FLAG, + rmr_contextp); + spin_unlock_irqrestore(&r_xprt->rx_ep.rep_postlock, lock_flags); + + /* Default registration using a new lmr_kcreate each time */ + } else { + DAT_REGION_DESCRIPTION region; + DAT_LMR_CONTEXT lmr_context; + u64 vaddr; + u64 seglen; + region.for_va = (void *) (unsigned long) seg->mr_base; + datstatus = dat_lmr_kcreate(ia->ri_ia_handle, + DAT_MEM_TYPE_VIRTUAL, region, seg->mr_len, + ia->ri_pz_handle, mem_priv, + DAT_MEM_OPTIMIZE_DONT_CARE, + &handlep->rl_lmr, &lmr_context, + rmr_contextp, &seglen, &vaddr); + } + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_register_external failed dat_%s %llu at 0x%llx status 0x%x\n", + (ia->ri_bind_mem != NULL) ? "rmr_bind" : "lmr_kcreate", + seg->mr_len, seg->mr_base, datstatus)); + + return (datstatus != DAT_SUCCESS); +} + +int +rdma_deregister_external(rdma_mr_handle *handlep, + rdma_xprt_t *r_xprt, void *rep) +{ + u32 datstatus; + unsigned int lock_flags; + +#if RPCRDMA_DEBUG + /* Persistent registration */ + if (r_xprt->rx_ia->ri_memreg_strategy == 4) { + if (rep) { + rdma_reply_func func = ((rdma_rep_t *) rep)->rr_func; + ((rdma_rep_t *) rep)->rr_func = NULL; + func(rep); + } + return 0; + } +#endif + + if (r_xprt->rx_ia->ri_bind_mem != NULL) { + DAT_RMR_COOKIE cookie; + struct dat_lmr_triplet triplet; + cookie.as_ptr = rep; + triplet.lmr_context = 0; + triplet.segment_length = 0; + triplet.virtual_address = 0; + spin_lock_irqsave(&r_xprt->rx_ep.rep_postlock, lock_flags); + if (rep) { + INIT_CQCOUNT(&r_xprt->rx_ep); + datstatus = dat_rmr_bind(handlep->rl_rmr->rmr_handle, + handlep->rl_lmr, &triplet, + DAT_MEM_PRIV_NONE_FLAG, + r_xprt->rx_ep.rep_handle, + cookie, + DAT_COMPLETION_DEFAULT_FLAG, + NULL); + if (datstatus != DAT_SUCCESS) + ((rdma_rep_t *) rep)->rr_func = NULL; + } else { + DECR_CQCOUNT(&r_xprt->rx_ep); + datstatus = dat_rmr_bind(handlep->rl_rmr->rmr_handle, + handlep->rl_lmr, + &triplet, DAT_MEM_PRIV_NONE_FLAG, + r_xprt->rx_ep.rep_handle, cookie, + DAT_COMPLETION_SUPPRESS_FLAG, NULL); + } + spin_unlock_irqrestore(&r_xprt->rx_ep.rep_postlock, lock_flags); + } else { + datstatus = dat_lmr_free(handlep->rl_lmr); + handlep->rl_lmr = NULL; + } + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_deregister_external failed dat_%s, status 0x%x\n", + (r_xprt->rx_ia->ri_bind_mem != NULL) ? + "rmr_unbind" : "lmr_free", datstatus)); + return (datstatus != DAT_SUCCESS); +} + +/* + * Prepost any receive buffer, then post send. + * + * Receive buffer is donated to hardware, reclaimed upon recv completion. + */ +int +rdma_ep_post(rdma_ep_t *ep, rdma_req_t *req) +{ + DAT_DTO_COOKIE rcookie, scookie; + u32 datstatus; + rdma_rep_t *rep = req->rl_reply; + unsigned int lock_flags; + + /* recv cookie */ + rcookie.as_ptr = rep; + /* no send cookie */ + scookie.as_ptr = NULL; + spin_lock_irqsave(&ep->rep_postlock, lock_flags); + if (rep) { + DECR_CQCOUNT(ep); + datstatus = dat_ep_post_recv(ep->rep_handle, + 1, &rep->rr_iov, + rcookie, DAT_COMPLETION_DEFAULT_FLAG); + if (datstatus != DAT_SUCCESS) { + goto out; + } + rep = req->rl_reply = NULL; + } + + if (DECR_CQCOUNT(ep) > 0) { + datstatus = dat_ep_post_send(ep->rep_handle, + req->rl_niovs, req->rl_send_iov, + scookie, DAT_COMPLETION_SUPPRESS_FLAG); + } else { + /* Provider must take a send completion every now and then */ + INIT_CQCOUNT(ep); + datstatus = dat_ep_post_send(ep->rep_handle, + req->rl_niovs, req->rl_send_iov, + scookie, DAT_COMPLETION_DEFAULT_FLAG); + } + out: + spin_unlock_irqrestore(&ep->rep_postlock, lock_flags); + + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ep_post: dat_ep_post_%s returned 0x%x\n", + rep ? "recv" : "send", datstatus)); + + return (datstatus != DAT_SUCCESS); +} + +/* + * Re-post a receive buffer. Only used in error cases. + */ +int +rdma_ep_post_recv(rdma_ep_t *ep, rdma_rep_t *rep) +{ + DAT_DTO_COOKIE cookie; + u32 datstatus; + unsigned int lock_flags; + + cookie.as_ptr = rep; + spin_lock_irqsave(&ep->rep_postlock, lock_flags); + DECR_CQCOUNT(ep); + datstatus = dat_ep_post_recv(ep->rep_handle, 1, + &rep->rr_iov, cookie, DAT_COMPLETION_DEFAULT_FLAG); + spin_unlock_irqrestore(&ep->rep_postlock, lock_flags); + + Dprintk(0 && datstatus != DAT_SUCCESS, + ("rdma_ep_post_recv: dat_ep_post_recv returned 0x%x\n", + datstatus)); + return (datstatus != DAT_SUCCESS); +} + +/* + * internal functions + */ + +/* + * rdma_async_evd_upcall + * + * This upcall "handles" catastrophic errors. + */ +static void +rdma_async_evd_upcall(void *instance_data, + const struct dat_event *event, boolean_t bool_arg) +{ + static const char * const asyncstrings[] = { + "evd overflow", + "catastrophic error", + "endpoint broken", + "timed out", + "provider internal error" + }; + unsigned int evnum = event->event_number - DAT_ASYNC_ERROR_EVD_OVERFLOW; + printk("rdma_async_evd_upcall: nic %s %s (event 0x%x)\n", + ((rdma_ia_t *) instance_data)->ri_ia_attr.adapter_name, + evnum < 5 ? asyncstrings[evnum] : "unknown error", + event->event_number); +} + +/* + * rdma_event_evd_upcall + * + * This upcall handles DTO, (recv, send, bind and unbind) events. + * It is reentrant but has been specified using DAT_UPCALL_SINGLE_INSTANCE + * in order to maintain ordering of receives to keep server credits. + * It must also be prepared to be called from interrupt context, + * so it must not block or perform blocking calls. + * + * It is the responsibility of the scheduled tasklet to return + * recv buffers to the pool. NOTE: this affects synchronization of + * connection shutdown. That is, the structures required for + * the completion of the reply handler must remain intact until + * all memory has been reclaimed. There is some work here TBD. + * + * Note that send events are suppressed and do not result in an upcall. + */ +static void +rdma_event_evd_upcall(void *instance_data, + const struct dat_event *event, boolean_t bool_arg) +{ + const struct dat_dto_completion_event_data *dto_data; + const struct dat_rmr_bind_completion_event_data *rmr_data; + rdma_rep_t *rep; /* in cookie */ + + switch (event->event_number) { + case DAT_DTO_COMPLETION_EVENT: + dto_data = &event->event_data.dto_completion_event_data; + rep = (rdma_rep_t *) dto_data->user_cookie.as_ptr; + if (rep) { + Dprintk(2 || dto_data->status != DAT_DTO_SUCCESS, + ("rdma_event_evd_upcall: receive DTO_COMPLETION, " + "status 0x%x (%s), len %lld, cookie 0x%p\n", + dto_data->status, ststatus(dto_data->status), + dto_data->transfered_length, rep)); + if (dto_data->status == DAT_DTO_SUCCESS) { + rep->rr_len = dto_data->transfered_length; + /* Keep (only) the most recent credits, check + * their validity later */ + if (rep->rr_len >= sizeof(struct rdma_msg)) { + struct rdma_msg *p = + (struct rdma_msg *) rep->rr_base; + atomic_set(&rep->rr_buffer->rb_credits, + ntohl(p->rdma_credit)); + } + } else { + rep->rr_len = ~0U; + } + rdma_schedule_tasklet(rep); + } else { + Dprintk(2 || dto_data->status != DAT_DTO_SUCCESS, + ("rdma_event_evd_upcall: send DTO_COMPLETION, " + "status 0x%x (%s), len %lld\n", + dto_data->status, ststatus(dto_data->status), + dto_data->transfered_length)); + } + break; + + case DAT_RMR_BIND_COMPLETION_EVENT: + rmr_data = &event->event_data.rmr_completion_event_data; + rep = (rdma_rep_t *) rmr_data->user_cookie.as_ptr; + Dprintk(2 || rmr_data->status != DAT_RMR_BIND_SUCCESS, + ("rdma_event_evd_upcall: %sBIND_COMPLETION, " + "status 0x%x (%s), cookie 0x%p\n", + rep ? "UN" : "", rmr_data->status, + ststatus(rmr_data->status), rep)); + if (rep) { + rdma_schedule_tasklet(rep); + } + break; + + default: + Dprintk(0, ("rdma_event_evd_upcall: unexpected event 0x%x\n", + event->event_number)); + break; + } +} + +/* + * Connection event handler. + */ +static void +rdma_conn_evd_upcall(void *instance_data, + const struct dat_event *event, boolean_t bool_arg) +{ + rdma_ep_t *ep = instance_data; + int connstate = 0; + + switch (event->event_number) { + case DAT_CONNECTION_EVENT_ESTABLISHED: + connstate = 1; + /* fall through */ + case DAT_CONNECTION_EVENT_PEER_REJECTED: + case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: + case DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR: + case DAT_CONNECTION_EVENT_DISCONNECTED: + case DAT_CONNECTION_EVENT_BROKEN: + case DAT_CONNECTION_EVENT_TIMED_OUT: + case DAT_CONNECTION_EVENT_UNREACHABLE: + Dprintk(1, ("rdma_conn_evd_upcall: CONNECTION event 0x%x, ep 0x%p\n", + event->event_number, ep)); + if (ep->rep_connected != connstate) { + /* notify waiter, pass changed state */ + ep->rep_event_num = event->event_number; + ep->rep_connected = connstate; + Dprintk(1, ("rdma_conn_evd_upcall: %sconnected\n", + connstate ? "" : "dis")); + ep->rep_func(ep); + wake_up_all(&ep->rep_connect_wait); + } + break; + + default: + Dprintk(0, ("rdma_conn_evd_upcall: unexpected event 0x%x\n", + event->event_number)); + break; + } +} + +/* + * rdma tasklet function -- just turn around and call the func + * for all replies on the list + */ +static void +rdma_run_tasklet(unsigned long data) +{ + rdma_rep_t *rep; + rdma_reply_func func; + unsigned int lock_flags; + + data = data; + Dprintk(3, ("rdma_run_tasklet: entering\n")); + spin_lock_irqsave(&rdma_tk_lock_g, lock_flags); + while (!list_empty(&rdma_tasklets_g)) { + rep = list_entry(rdma_tasklets_g.next, rdma_rep_t, rr_list); + Dprintk(3, ("rdma_run_tasklet: found entry 0x%p\n", rep)); + list_del(&rep->rr_list); + func = rep->rr_func; + rep->rr_func = NULL; + spin_unlock_irqrestore(&rdma_tk_lock_g, lock_flags); + + if (func) { + func(rep); + } else { + Dprintk(1, ("rdma_run_tasklet: orphaned reply 0x%p\n", rep)); + rdma_recv_buffer_put(rep); + } + + spin_lock_irqsave(&rdma_tk_lock_g, lock_flags); + } + spin_unlock_irqrestore(&rdma_tk_lock_g, lock_flags); + Dprintk(3, ("rdma_run_tasklet: leaving\n")); +} Index: drivers/infiniband/ulp/nfsrdma/rdma_transport.c =================================================================== --- drivers/infiniband/ulp/nfsrdma/rdma_transport.c (revision 0) +++ drivers/infiniband/ulp/nfsrdma/rdma_transport.c (revision 0) @@ -0,0 +1,950 @@ +/* + * Copyright (c) 2003, 2004, Network Appliance, Inc. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * Neither the name of the Network Appliance, Inc. nor the names of + * its contributors may be used to endorse or promote products + * derived from this software without specific prior written + * permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* + * rdma_transport.c + * + * This file contains the top-level implementation of an RPC RDMA + * transport, implemented using the new transport switch. + * Major functions, which are required by the switch include: + * xprt_rdma_create -- create an instance (e.g. during mount) + * xprt_rdma_destroy -- destroy transport instance + * xprt_rdma_close + * xprt_rdma_allocate -- allocate message buffer(s) + * xprt_rdma_sendmsg -- actually do the send. In the case of RDMA, + * the recv is posted first, and completed async. + * xprt_rdma_reconnect + * xprt_rdma_free -- release message buffer(s) + * + * Naming convention: functions beginning with xprt_ are part of the + * transport switch. All others are RPC RDMA internal. + */ + +/* + * See "TBD" for to-be-dones + */ + +#include +#include +#include + +#include "rdma_kdapl.h" + +/* prototypes and plumbing for switch */ + +static int xprt_rdma_create(struct rpc_xprt *, struct xprt_create_data *); +static int xprt_rdma_destroy(struct rpc_xprt *); +static void xprt_rdma_close(struct rpc_xprt *); +static void *xprt_rdma_allocate(struct rpc_xprt *, struct rpc_task *, + unsigned int); +static int xprt_rdma_sendmsg(struct rpc_xprt *, struct rpc_rqst *); +static void xprt_rdma_reconnect(struct rpc_task *); +static void xprt_rdma_free(struct rpc_xprt *, struct rpc_task *, void *); + +/* Transport procedures for rdma */ +static struct rpc_transport xprt_rdma = { + "RDMA", + RPC_XPRT_RDMA, + { + xprt_rdma_allocate, + xprt_rdma_sendmsg, + xprt_rdma_free, + xprt_rdma_reconnect, + xprt_rdma_create, + xprt_rdma_destroy, + xprt_rdma_close + } +}; + +/* + * local functions + */ + +/* register rdma transport upon module load */ +static int rdma_register_xprt(void); +static int rdma_remove_xprt(void); + +/* Callbacks */ +/* perform rpc call completion, as tasklet */ +static void rdma_reply_handler(rdma_rep_t *); +/* signal rdma unbind completion */ +static void rdma_unbind_func(rdma_rep_t *); +/* called for connection-relevant events */ +static void rdma_conn_func(struct rdma_ep *); + +/* do the work of xprt creation */ +static rdma_xprt_t *rdma_create_xprt(char *, + struct rdma_create_data_internal *, + struct rpc_xprt *); + +/* RPC/RDMA parameters */ +static char *Adapter = "null"; /* TBD this could be a list */ +static int MaxRequests = 100; /* RPC_MAXREQS; Linux default */ +static int MaxInlineRead = 1024; /* 0; */ +static int MaxInlineWrite = 1024; /* 0; */ +static int Padding = 512; /* set to server's preference if any */ +static int Stream = 1; /* TCP(1)/UDP(0) emulation, testing */ + +#undef Dprintk +#ifdef RPCRDMA_DEBUG +static int Memreg = 4; /* memreg strategy higher=faster */ +#define Dprintk(n, x) if (Debug >= n) printk x +int Debug = 0; +static int RKDebug = 0; +extern int rdma_kdapl_debug; +#else +static int Memreg = 3; /* memreg strategy higher=faster */ +#define Dprintk(n, x) +#endif + +static rdma_ia_t default_ia; /* The default adapter */ +static int registered = 0; + +MODULE_LICENSE("BSD"); +MODULE_DESCRIPTION("RPC/RDMA Transport for Linux kernel NFS"); +MODULE_AUTHOR("Network Appliance, Inc."); + +MODULE_PARM(Adapter, "1-32s"); +MODULE_PARM_DESC(Adapter, "Default RDMA adapter name"); + +MODULE_PARM(MaxRequests, "1-256i"); +MODULE_PARM_DESC(MaxRequests, "Credits requested of server"); + +MODULE_PARM(MaxInlineRead, "0-65536i"); +MODULE_PARM_DESC(MaxInlineRead, "Maximum inline (non-RDMA) read size"); + +MODULE_PARM(MaxInlineWrite, "0-65536i"); +MODULE_PARM_DESC(MaxInlineWrite, "Maximum inline (non-RDMA) write size"); + +MODULE_PARM(Padding, "0-8192i"); +MODULE_PARM_DESC(Padding, "Inline write padding"); + +#if !RPCRDMA_DEBUG + +MODULE_PARM(Memreg, "0-3i"); +MODULE_PARM_DESC(Memreg, "Memreg none 0, sync/lmr 1, sync/rmr 2, async/rmr 3"); + +#else + +MODULE_PARM(Memreg, "0-4i"); +MODULE_PARM_DESC(Memreg, "Memreg none 0, sync/lmr 1, sync/rmr 2, async/rmr 3, perm 4"); + +MODULE_PARM(Stream, "0-1i"); +MODULE_PARM_DESC(Stream, "Emulation UDP 0, TCP 1, testing"); + +MODULE_PARM(Debug, "0-1i"); +MODULE_PARM_DESC(Debug, "NFS/RDMA debug"); + +MODULE_PARM(RKDebug, "0-3i"); +MODULE_PARM_DESC(RKDebug, "RPC/kDAPL debug"); +#endif + +/*********************************************************************** + * init_module + * + * Entry point for a Linux module, performs simple initialization + ***********************************************************************/ + +static int __init rdma_init(void) +{ +#if RPCRDMA_DEBUG + rdma_kdapl_debug = RKDebug; +#endif + + Dprintk(0, ("RPCRDMA Module Init, register RPC RDMA transport\n")); + + Dprintk(0, ("Defaults:\n")); + Dprintk(0, ("\tAdapter %s\n\tMaxRequests %d\n", + Adapter, MaxRequests)); + Dprintk(0, ("\tMaxInlineRead %d\n\tMaxInlineWrite %d\n", + MaxInlineRead, MaxInlineWrite)); + Dprintk(0, ("\tPadding %d\n\tMemreg %d\n\tStream %d\n", Padding, + Memreg, Stream)); + Dprintk(0, ("\tDebug %d\n\tRKDebug %d\n", Debug, RKDebug)); + + /* Open the default adapter */ + if (!rdma_ia_init(&default_ia, Adapter, Memreg)) { + Dprintk(0, ("RPCRDMA module: can't open default adapter \"%s\"\n", + Adapter)); + return -ENOENT; + } + + if (rdma_register_xprt()) { + rdma_ia_close(&default_ia); + return -EEXIST; + } + + return 0; +} + +/*********************************************************************** + * cleanup_module + * + * Entry point for a Linux module, cleans up the module on exit + ***********************************************************************/ +static void __exit rdma_exit(void) +{ + rdma_ia_close(&default_ia); + + Dprintk(0, ("RPCRDMA Module Removed, deregister RPC RDMA transport\n")); + rdma_remove_xprt(); +} + +module_init(rdma_init); +module_exit(rdma_exit); + +/* + * Routines to register/remove the transport. To be called when + * module is loaded/unloaded + */ +static int +rdma_register_xprt() +{ + if (xprt_register(&xprt_rdma)) + return -EEXIST; + registered++; + return 0; +} + +static int +rdma_remove_xprt() +{ + if (xprt_unregister(&xprt_rdma)) + return -ENOENT; + registered--; + return 0; +} + +static int +xprt_rdma_create(struct rpc_xprt *xprt, struct xprt_create_data *data) +{ + rdma_xprt_t *r_xprt; + struct rdma_create_data *datap = &data->u.rdma_data; + struct rdma_create_data_internal cdata; + + /* fill in some generic fields to keep RPC happy */ + xprt->stream = (Stream != 0); + xprt->addr.sin_port = 1; /* avoid portmap calls */ + xprt->prot = (xprt->stream ? IPPROTO_TCP : IPPROTO_UDP); + + xprt->nocong = 1; /* we do this ourselves */ + xprt->cwnd = RPC_CWNDSCALE; /* Allow one send at first */ + +#if 0 + if (datap->timeo) { + xprt->timeout = *datap->timeo; + xprt->timeout.to_current = datap->timeo->to_initval; + xprt->timeout.to_resrvval = datap->timeo->to_maxval << 1; + } else +#endif + xprt_set_timeout(&xprt->timeout, 1, 30 * HZ); + + /* Set server address(es) */ + xprt->addr = datap->srvaddr; /* IP address */ + cdata.addr = datap->addr; /* RDMA address, possibly different */ + + /* [See temporary trick in sunrpc.c xprt_create_proto()] */ + if (cdata.addr.sa_family == 0) { + cdata.addr = *(struct sockaddr *)&xprt->addr; + memcpy(&cdata.addr.sa_data[2], &xprt->addr.sin_zero[2], 4); + /* zero out the 9999 trick, lest clients like NLM come back + * with it. We only support the initial mount doing this. */ + memset(xprt->addr.sin_zero, 0, sizeof xprt->addr.sin_zero); + memset(datap->srvaddr.sin_zero, 0, sizeof xprt->addr.sin_zero); + } + /* [end trick] */ + + if (datap->port) + cdata.port = datap->port; + else + cdata.port = 2049ULL; + + /* Set max requests */ + if ((cdata.max_requests = datap->max_requests) <= 0) { + cdata.max_requests = MaxRequests; + } + + /* Set some length limits */ + if ((cdata.rsize = datap->rsize) <= 0) + cdata.rsize = 32768; /* NFS3_MAXDATA */ + if ((cdata.wsize = datap->wsize) <= 0) + cdata.wsize = 32768; /* NFS3_MAXDATA */ + + if ((cdata.inline_wsize = datap->max_inline_send) <= 0) { + cdata.inline_wsize = MaxInlineWrite; + } + if (cdata.inline_wsize > cdata.wsize) { + cdata.inline_wsize = cdata.wsize; + } + + if ((cdata.inline_rsize = datap->max_inline_recv) <= 0) { + cdata.inline_rsize = MaxInlineRead; + } + if (cdata.inline_rsize > cdata.rsize) { + cdata.inline_rsize = cdata.rsize; + } + + if ((cdata.padding = datap->padding) <= 0) { + cdata.padding = Padding; + } + + /* + * Create and connect new transport. + * TBD provide adapter choice? + */ + r_xprt = rdma_create_xprt(NULL, &cdata, xprt); + + if (r_xprt == NULL) { + return -1; + } + + /* rpciod handles stream reconnect */ + if (xprt->stream) + rpciod_up(); + + /* Overload unused "sock" as our private pointer */ + xprt->sock = (struct socket *)r_xprt; + xprt->inet = (struct sock *)~0; + return 0; +} + +/* + * xprt_rdma_destroy + * + * Destroy the xprt. + * Free all memory associated with the object, including its own. + * NOTE: none of the *destroy methods free memory for their top-level + * objects, even though they may have allocated it (they do free + * private memory). It's up to the caller to handle it. In this + * case (RDMA transport), all structure memory is inlined with the + * rdma_xprt_t. + */ +static int +xprt_rdma_destroy(struct rpc_xprt *xprt) +{ + rdma_xprt_t *r_xprt = rpcx_to_rdmax(xprt); + + Dprintk(1, ("xprt_rdma_destroy called\n")); + + xprt_clear_connected(xprt); + + if (r_xprt) { + rdma_buffer_destroy(&r_xprt->rx_buf); + (void) rdma_ep_destroy(&r_xprt->rx_ep, r_xprt->rx_ia); + } + + if (r_xprt->rx_ia != &default_ia) { + rdma_ia_close(r_xprt->rx_ia); + kfree(r_xprt->rx_ia); + } + + kfree(r_xprt); + if (xprt->stream) + rpciod_down(); + Dprintk(1, ("xprt_rdma_destroy returning\n")); + return 0; +} + +/* + * rdma_create_xprt + * + * Create a transport instance, which includes initialized + * o ia + * o endpoint + * o buffers + * Returns a pointer to the new structure. + */ +static rdma_xprt_t * +rdma_create_xprt(char *ia_name, + struct rdma_create_data_internal *cdata, struct rpc_xprt *xprt) +{ + rdma_xprt_t *new_xprt; + rdma_ep_t *new_ep = NULL; + rdma_ia_t *new_ia = NULL; + rdma_buffer_t *new_buf = NULL; + int extra = 0; + struct rpc_rqst *rqst; + + list_for_each_entry(rqst, &xprt->free, rq_list) + extra++; + if (cdata->max_requests > extra) { + /* More than provided by RPC: carve out some more slots */ + extra = (cdata->max_requests - extra) * sizeof(struct rpc_rqst); + } else { + extra = 0; + } + + new_xprt = kmalloc(sizeof(rdma_xprt_t) + extra, GFP_KERNEL); + if (!new_xprt) { + goto outfail; + } + memset(new_xprt, 0, sizeof(rdma_xprt_t) + extra); + + /* Add any extra request slot entries to table */ + rqst = (struct rpc_rqst *) (new_xprt + 1); + while ((extra -= sizeof(struct rpc_rqst)) >= 0) { + list_add(&rqst->rq_list, &xprt->free); + rqst++; + } + + if (ia_name == NULL) { + new_ia = &default_ia; + ia_name = Adapter; + } else { + new_ia = kmalloc(sizeof(rdma_ia_t), GFP_KERNEL); + if (!new_ia) { + goto outfail; + } + if (!rdma_ia_init(new_ia, ia_name, Memreg)) { + goto outfail; + } + } + new_xprt->rx_ia = new_ia; + + /* + * initialize and create ep + */ + new_ep = &new_xprt->rx_ep; + new_xprt->rx_data = *cdata; + new_ep->rep_remote_addr = cdata->addr; + new_ep->rep_server_port = cdata->port; /* (64-bit!) */ + rdma_ep_default_attr(new_ep, new_ia, &new_xprt->rx_data); + Dprintk(0, ("rdma_create_xprt: %s %d.%d.%d.%d:%lld\n", + ia_name, + (unsigned char) new_ep->rep_remote_addr.sa_data[2], + (unsigned char) new_ep->rep_remote_addr.sa_data[3], + (unsigned char) new_ep->rep_remote_addr.sa_data[4], + (unsigned char) new_ep->rep_remote_addr.sa_data[5], + new_ep->rep_server_port)); + + new_ep = rdma_ep_create(&new_xprt->rx_ep, new_ia, &new_xprt->rx_data); + if (!new_ep) { + goto outfail; + } + + /* + * Allocate pre-registered send and receive buffers for headers and + * any inline data. Also specify any padding which will be provided + * from a preregistered zero buffer. + */ + new_buf = rdma_buffer_create(&new_xprt->rx_buf, new_ep, + new_ia, &new_xprt->rx_data); + if (!new_buf) { + goto outfail; + } + + /* + * Register a callback for connection events. This is necessary because + * connection loss notification is async. We also catch connection loss + * when reaping receives. + */ + new_ep->rep_func = rdma_conn_func; + new_ep->rep_xprt = xprt; + + if (rdma_ep_connect(new_ep, new_ia, 0)) { + goto outfail; + } + return new_xprt; + + outfail: + /* clean up any work done */ + if (new_buf) { + rdma_buffer_destroy(new_buf); + } + if (new_ep) { + rdma_ep_destroy(new_ep, new_ia); + } + if (new_ia && new_ia != &default_ia) { + rdma_ia_close(new_ia); + kfree(new_ia); + } + if (new_xprt) { + kfree(new_xprt); + } + return NULL; +} + +/* + * Close a connection, during shutdown or timeout/reconnect + */ +static void +xprt_rdma_close(struct rpc_xprt *xprt) +{ + rdma_xprt_t *r_xprt = rpcx_to_rdmax(xprt); + + Dprintk(1, ("xprt_rdma_close: closing")); + xprt_clear_connected(xprt); + rdma_ep_disconnect(&r_xprt->rx_ep, r_xprt->rx_ia); +} + +/* + * The RDMA allocate/free functions need the task structure as a place + * to hide the rdma_req_t, which is necessary for the actual send/recv + * sequence. For this reason, the recv buffers are attached to send + * buffers for portions of the RPC. Note that the RPC layer allocates + * both send and receive buffers in the same call. We may register + * the receive buffer portion when using reply chunks. + */ +static void * +xprt_rdma_allocate(struct rpc_xprt *xprt, + struct rpc_task *task, unsigned int size) +{ + rdma_req_t *req, *nreq; + + req = rdma_buffer_get(&rpcx_to_rdmax(xprt)->rx_buf); + Dprintk(1, ("xprt_rdma_allocate: size %d, request 0x%p\n", size, req)); + + if (size > req->rl_size + MAX_RPCHDR) { + Dprintk(1, ("xprt_rdma_allocate: size %d too large for %d: " + "prog %d vers %d proc %d\n", + size, req->rl_size, + task->tk_client->cl_prog, task->tk_client->cl_vers, + task->tk_msg.rpc_proc->p_proc)); + /* + * Outgoing length shortage. Our inline write max must have + * been configured to perform direct i/o. + * + * This is therefore a large metadata operation, and the + * allocate call was made on the maximum possible message, + * e.g. containing long filename(s) or symlink data. In + * fact, while these metadata operations *might* carry + * large outgoing payloads, they rarely *do*. However, we + * have to commit to the request here, so reallocate and + * register it now. The data path will never require this + * reallocation. + * + * If the allocation or registration fails, the RPC framework + * will (doggedly) retry. + */ + if (rpcx_to_rdmax(xprt)->rx_ia->ri_memreg_strategy == 0) { + /* forced to "pure inline" */ + Dprintk(0, ("xprt_rdma_allocate: too much data " + "(%d) for inline\n", size)); + return NULL; + } + nreq = kmalloc(sizeof *req + size, GFP_KERNEL); + if (nreq == NULL) { + return NULL; + } + if (rdma_register_internal(rpcx_to_rdmax(xprt)->rx_ia, + nreq->rl_base, size + MAX_RDMAHDR, + &nreq->rl_handle, &nreq->rl_iov)) { + kfree(nreq); + return NULL; + } +/* TBD increment a "hard way" statistic (+= size) */ + nreq->rl_size = size; + nreq->rl_niovs = 0; + nreq->rl_nsegs = 0; + nreq->rl_buffer = (struct rdma_buffer *)req; + nreq->rl_reply = req->rl_reply; + memcpy(nreq->rl_seg_handles, req->rl_seg_handles, + sizeof nreq->rl_seg_handles); + /* flag the swap with an unused field */ + nreq->rl_iov.segment_length = 0; + req->rl_reply = NULL; + req = nreq; + } + return req + 1; +} + +/* + * This function returns all RDMA resources to the pool. + */ +static void +xprt_rdma_free(struct rpc_xprt *xprt, struct rpc_task *task, void *buf) +{ + rdma_req_t *req = (rdma_req_t *)buf - 1; + rdma_rep_t *rep = req->rl_reply; + rdma_xprt_t *r_xprt = rpcx_to_rdmax(xprt); + + Dprintk(1, ("xprt_rdma_free: called with buf 0x%p, %swait on 0x%p\n", + buf, (rep && rep->rr_func) ? "" : "no ", rep)); + + /* + * Finish the deregistration. When using rmr bind, this was + * begun in rdma_reply_handler(). When using lmr_free, we do + * it here, in thread context. The process is considered + * complete when the rr_func vector becomes NULL - this + * was put in place during rdma_reply_handler() - the wait + * call below will not block if the dereg is "done". If + * interrupted, our framework will clean up. + */ + while (req->rl_nsegs) { + rdma_deregister_external(&req->rl_seg_handles[--req->rl_nsegs], + r_xprt, NULL); + } + + if (rep && wait_event_interruptible(rep->rr_unbind, !rep->rr_func)) { + rep->rr_func = NULL; /* abandon the callback */ + req->rl_reply = NULL; + } + + if (req->rl_iov.segment_length == 0) { /* see allocate above */ + rdma_req_t *oreq = (rdma_req_t *)req->rl_buffer; + oreq->rl_reply = req->rl_reply; + rdma_deregister_internal(req->rl_handle); + kfree(req); + req = oreq; + } + + /* Put back request+reply buffers */ + rdma_buffer_put(req); + + /* Update most recent server credits and done */ + if (r_xprt->rx_ep.rep_connected == 1) { + int credits; + spin_lock(&xprt->xprt_lock); + credits = atomic_read(&req->rl_buffer->rb_credits); + if (credits > req->rl_buffer->rb_max_requests) { + credits = req->rl_buffer->rb_max_requests; + } + credits *= RPC_CWNDSCALE; + /* new credits can drop at most one/reply, but + * can increase by any amount. */ + if (credits < xprt->cwnd) { + xprt->cwnd -= RPC_CWNDSCALE; + } else { + xprt->cwnd = credits; + } +/* TBD keep a stat and note credit grant/use hiwaters */ + if (RPCXPRT_CONGESTED(xprt)) { + if (xprt->cwnd == 0) { + Dprintk(0, ("rdma_reply: server dropped credits to 0!\n")); + xprt->cwnd = RPC_CWNDSCALE; /* don't deadlock */ + } + Dprintk(1 && xprt->cong - RPC_CWNDSCALE >= xprt->cwnd, + ("rdma_reply: still flow controlled (%ld/%ld)\n", + (xprt->cong / RPC_CWNDSCALE) - 1, + xprt->cwnd / RPC_CWNDSCALE)); + } + spin_unlock(&xprt->xprt_lock); + } +} + +/* + * This function is called when an unbind we are waiting for completes. + * Just use rr_func (zeroed by upcall) to signal completion. + */ +static void +rdma_unbind_func(rdma_rep_t *rep) +{ + wake_up(&rep->rr_unbind); +} + +/* + * This function is called when an async event is posted to + * the connection which changes the connection state. All it + * does at this point is mark the connection up/down, the rpc + * timers do the rest. + */ +static void +rdma_conn_func(struct rdma_ep *ep) +{ + struct rpc_xprt *xprt = (struct rpc_xprt *) ep->rep_xprt; + const char * const conn[] = { + "connected", + "peer rejected", + "peer not listening", + "peer failed to accept", + "disconnected", + "connection broken", + "timed out", + "unreachable" + }; + unsigned int err = ep->rep_event_num - DAT_CONNECTION_EVENT_ESTABLISHED; + Dprintk(0 && !xprt->shutdown, + ("rdma_conn_func: %s: %d.%d.%d.%d:%lld (DAT event 0x%x)\n", + (err <= 7) ? conn[err] : "unknown connection error", + (unsigned char) ep->rep_remote_addr.sa_data[2], + (unsigned char) ep->rep_remote_addr.sa_data[3], + (unsigned char) ep->rep_remote_addr.sa_data[4], + (unsigned char) ep->rep_remote_addr.sa_data[5], + ep->rep_server_port, ep->rep_event_num)); + + if (ep->rep_connected) { + xprt_set_connected(xprt); + /* Only schedule one task, until credits refreshed */ + rpc_wake_up_next(&xprt->sending); + } else { + xprt_clear_connected(xprt); + xprt->cwnd = RPC_CWNDSCALE; /* Reset server credits */ + } +} + +/* + * Reconnect. This routine is TBD and under construction. + * It is only called by the RPC framework when xprt->stream + * is true and xprt_connected() is false. + */ +static void +xprt_rdma_reconnect(struct rpc_task *task) +{ + struct rpc_xprt *xprt = task->tk_xprt; + rdma_xprt_t *r_xprt = rpcx_to_rdmax(xprt); + + if (!xprt->shutdown && r_xprt->rx_ep.rep_connected != 1) { + Dprintk(0, ("xprt_rdma_reconnect: attempt reconnect\n")); + if (rdma_ep_reconnect(&r_xprt->rx_ep, r_xprt->rx_ia) == 0) + rpc_delay(task, HZ / (1000000 / RDMA_CONNECT_TIMEOUT)); + /* count retries? */ + } + if (r_xprt->rx_ep.rep_connected != 1) { + rpc_delay(task, 5 * HZ); + task->tk_status = -ENOTCONN; + } +} + +/* + * Sendmsg invokes the meat of RPC RDMA. It must do the following: + * 1. Detect any connection issues, particularly when acting like + * a UDP (connectionless) transport. + * 2. Marshal the RPC request into an RPC RDMA request, which means + * putting a header in front of data, and creating IOVs for kDAPL + * from those in the request. + * 3. In marshaling, detect opportunities for RDMA, and use them. + * 4. Post a recv message to set up asynch completion, then send + * the request (rdma_ep_post). + * 5. Return value is number of bytes which were requested to be sent, + * or error. No partial sends are supported. + * + * Synchronization: This routine is reentrant. + * The RPC subsystem ensures that a given rpc_task is + * protected. Multiple tasks can run at the same time. Shared structures + * include the endpoint and interface structures, but these are not modified + * during this operation, unless a reconnect is required, which is + * serialized by the RPC subsystem as well + * TBD: make sure that the RPC lock protects this (true for stream?). + * TBD: verify sync of destroy of rdma_xprt_t with this and other users. + */ + +static int +xprt_rdma_sendmsg(struct rpc_xprt *xprt, struct rpc_rqst *rqst) +{ + rdma_req_t *req; + rdma_xprt_t *r_xprt = rpcx_to_rdmax(xprt); + + /* + * Normally we run with xprt->stream, in which case the RPC + * framework handles reconnect. However, when running in "UDP + * mode", support it ourself. This method is TBD. + */ + if (r_xprt->rx_ep.rep_connected != 1) { + if (r_xprt->rx_ep.rep_connected == 0 && !xprt->stream) { + Dprintk(0, ("xprt_rdma_sendmsg: attempt reconnect\n")); + rdma_ep_reconnect(&r_xprt->rx_ep, r_xprt->rx_ia); + } + return -ENOTCONN; + } + + /* no retransmits */ + if (rqst->rq_bytes_sent) { + Dprintk(1, ("xprt_rdma_sendmsg: retransmit suppressed!\n")); + return 0; + } + req = rpcr_to_rdmar(rqst); + + if (req->rl_niovs == 0) { /* first time */ + /* marshal the send itself */ + if (rdma_marshal_req(rqst) != 0) { + Dprintk(0, ("xprt_rdma_sendmsg: rdma_marshal_req failed\n")); + return -EIO; + } + } else + Dprintk(1, ("xprt_rdma_sendmsg: retransmit initiated!\n")); + + if (req->rl_reply == NULL) { /* e.g. reconnection */ + rdma_recv_buffer_get(req); + } + if (req->rl_reply != NULL) { + req->rl_reply->rr_func = rdma_reply_handler; + /* this need only be done once, but... */ + req->rl_reply->rr_xprt = xprt; + } + + if (rdma_ep_post(&r_xprt->rx_ep, req)) { + xprt_rdma_close(xprt); + return -ENOTCONN; /* implies disconnect */ + } + return rqst->rq_slen; +} + +/* + * Called as a tasklet to do req/reply match and complete a request + * Errors must result in the RPC task either being awakened, or + * allowed to timeout, to discover the errors at that time. + */ +static void +rdma_reply_handler(rdma_rep_t *rep) +{ + struct rdma_msg_no_chunks *headerp; + rdma_req_t *req; + struct rpc_rqst *rqst; + struct rpc_xprt *xprt = rep->rr_xprt; + rdma_xprt_t *r_xprt = rpcx_to_rdmax(xprt); + unsigned int *iptr; + int rdmalen, status; + + /* Check status. If bad, signal disconnect and return rep to pool */ + if (rep->rr_len == ~0U) { + rdma_recv_buffer_put(rep); + if (r_xprt->rx_ep.rep_connected == 1) { + r_xprt->rx_ep.rep_event_num = ~0; + r_xprt->rx_ep.rep_connected = 0; + rdma_conn_func(&r_xprt->rx_ep); + } + return; + } + if (rep->rr_len < sizeof *headerp) { + Dprintk(0, ("rdma_reply_handler: short/invalid reply\n")); + goto repost; + } + headerp = (struct rdma_msg_no_chunks *) rep->rr_base; + + /* Get XID and try for a match. */ + rqst = xprt_lookup_rqst(xprt, headerp->rdma_xid); + if (rqst == NULL) { + Dprintk(0, ("rdma_reply_handler: reply 0x%p failed " + "to match request 0x%08x len %d\n", + rep, headerp->rdma_xid, rep->rr_len)); + repost: + rep->rr_func = rdma_reply_handler; + if (rdma_ep_post_recv(&r_xprt->rx_ep, rep)) { + rdma_recv_buffer_put(rep); + } + return; + } + + /* get request object */ + req = rpcr_to_rdmar(rqst); + + Dprintk(1, ("rdma_reply_handler: reply 0x%p completes request 0x%p" + " RPC request 0x%p xid 0x%08x\n", + rep, req, rqst, headerp->rdma_xid)); + + RDMA_ASSERT(req && !req->rl_reply, + "xprt_lookup_rqst returned bad/duplicate reply"); + + /* from here on, the reply is no longer an orphan */ + req->rl_reply = rep; + rqst->rq_task->tk_garb_retry = 0; /* no retry on invalid reply */ + + /* check for expected message types */ + /* The order of some of these tests is important. */ + switch (headerp->rdma_type) { + case __constant_htonl(RDMA_MSG): + /* never expect read chunks */ + /* never expect reply chunks (two ways to check) */ + /* never expect write chunks without offered RDMA */ + if (headerp->rdma_nochunks[0] != xdr_zero || + (headerp->rdma_nochunks[1] == xdr_zero && + headerp->rdma_nochunks[2] != xdr_zero) || + (headerp->rdma_nochunks[1] != xdr_zero && + req->rl_nsegs == 0)) { + goto badheader; + } + if (headerp->rdma_nochunks[1] != xdr_zero) { + /* count any expected write chunks in read reply */ + /* start at write chunk array count */ + iptr = &headerp->rdma_nochunks[2]; + rdmalen = rdma_count_chunks(rep, req->rl_nsegs, 1, &iptr); + /* check for validity, and no reply chunk after */ + if (rdmalen < 0 || *iptr++ != xdr_zero) { + goto badheader; + } + rep->rr_len -= + ((unsigned char *)iptr - (unsigned char *)headerp); + status = rep->rr_len + rdmalen; + } else { + /* else ordinary inline */ + iptr = (unsigned int *) (headerp + 1); + rep->rr_len -= sizeof *headerp; + status = rep->rr_len; + } + /* Fix up the rpc results for upper layer */ + rdma_inline_fixup(rqst, iptr, rep->rr_len); + break; + + case __constant_htonl(RDMA_NOMSG): + /* never expect read or write chunks, always reply chunks */ + if (headerp->rdma_nochunks[0] != xdr_zero || + headerp->rdma_nochunks[1] != xdr_zero || + headerp->rdma_nochunks[2] != xdr_one || + req->rl_nsegs == 0) { + goto badheader; + } + iptr = (unsigned int *) (headerp + 1); + rdmalen = rdma_count_chunks(rep, req->rl_nsegs, 0, &iptr); + if (rdmalen < 0) { + goto badheader; + } + /* Reply chunk buffer already is the reply vector - no fixup. */ + status = rdmalen; + break; + + default: + badheader: + Dprintk(0, ("rdma_reply_handler: invalid reply header (type %d)\n", + ntohl(headerp->rdma_type))); + status = -EIO; + break; + } + + /* If using rmr bind, start the deregister process now. */ + /* (Note: if lmr_free(), cannot perform it here, in tasklet context) */ + if (req->rl_nsegs && r_xprt->rx_ia->ri_memreg_strategy > 1) { + do { + /* Optionally wait (not here) for unbinds to complete */ + if (--req->rl_nsegs == 0 && + r_xprt->rx_ia->ri_memreg_strategy == 2) { + rep->rr_func = rdma_unbind_func; + rdma_deregister_external(&req->rl_seg_handles[0], + r_xprt, rep); + } else { + rdma_deregister_external(&req->rl_seg_handles[req->rl_nsegs], + r_xprt, NULL); + } + } while (req->rl_nsegs); + } + +#if LINUX_VERSION_CODE <= KERNEL_VERSION(2,4,19) + /* complete RPC task. */ + rqst->rq_task->tk_status = status; + rqst->rq_received = 1; + rpc_wake_up_task(rqst->rq_task); + /* Lookup returns a locked task. */ + rpc_unlock_task(rqst->rq_task); +#else + xprt_complete_rqst(xprt, rqst, status); +#endif +} Index: drivers/infiniband/ulp/nfsrdma/Kconfig =================================================================== --- drivers/infiniband/ulp/nfsrdma/Kconfig (revision 0) +++ drivers/infiniband/ulp/nfsrdma/Kconfig (revision 0) @@ -0,0 +1,5 @@ +config NFSRDMA + tristate "NFS over RDMA" + depends on INFINIBAND && INFINIBAND_IPOIB && DAT + ---help--- + NFSoRMDA Index: drivers/infiniband/ulp/nfsrdma/rdma_kdapl.h =================================================================== --- drivers/infiniband/ulp/nfsrdma/rdma_kdapl.h (revision 0) +++ drivers/infiniband/ulp/nfsrdma/rdma_kdapl.h (revision 0) @@ -0,0 +1,350 @@ +/* + * Copyright (c) 2003, 2004, Network Appliance, Inc. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * Neither the name of the Network Appliance, Inc. nor the names of + * its contributors may be used to endorse or promote products + * derived from this software without specific prior written + * permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _RDMA_KDAPL_H +#define _RDMA_KDAPL_H + +#include /* kDAPL */ +#include /* wait_queue_head_t, etc */ +#include /* spinlock_t, etc */ +#include /* atomic_t, etc */ +#include "rdma_proto.h" /* RPC/RDMA protocol */ + +#if RPCRDMA_DEBUG +#define RDMA_ASSERT(x, s) if (!(x)) panic (s) +#else +#define RDMA_ASSERT(x, s) +#endif + +/* + * Constants. Max RPC/NFS header is big enough to account for + * additional marshaling buffers passed down by Linux client. + * + * RDMA header is big enough for a fully-chunked message (read + * chunks are the largest). Note only a single chunk type per + * message is supported currently. + */ +#define MAX_RPCHDR (512 + 136) /* RPC/NFS overhead to allocate */ +#define MAX_RDMAHDR ( /* RPC/RDMA max header */ \ + sizeof (struct rdma_msg) + (2 * sizeof (uint32_t)) + \ + (sizeof (struct xdr_read_chunk) * RDMA_MAX_SEGS) + sizeof (uint32_t)) +#define RDMA_INLINE_PAD_THRESH 512 /* payload threshold to pad */ + +#define RDMA_MAX_SEGS 17 /* max scatter/gather: hdr + 16 pgs */ + +#define RDMA_CONNECT_TIMEOUT 500000 /* TBD .5 seconds */ +#define RDMA_CONNECT_RETRY_MAX 9 + +/* + * Data structures: + * rdma_ia -- interface adapter, plus pz and event handles + * rdma_ep -- represents an endpoint. + * rdma_buffer -- holds pre-registered buffer memory for inline messages. + * rdma_xprt -- encapsulates the structures above for integration with RPC. + */ + +/* + * Interface Adapter -- one per module instance + */ +typedef struct rdma_ia { + struct dat_ia *ri_ia_handle; + int ri_memreg_strategy; + struct dat_lmr *ri_bind_mem; +#if RPCRDMA_DEBUG + DAT_RMR_CONTEXT ri_bind_rmr; +#endif + struct dat_lmr_triplet ri_bind_iov; + struct dat_ia_attr ri_ia_attr; + struct dat_provider_attr ri_pv_attr; + struct dat_pz *ri_pz_handle; + struct dat_evd *ri_async_evd_handle; +} rdma_ia_t; + +/* + * RDMA Endpoint -- one per transport instance + */ +typedef struct rdma_ep { + spinlock_t rep_postlock; + int rep_cqcount; + int rep_cqinit; + int rep_connected; + rdma_ia_t *rep_ia; + struct dat_ep *rep_handle; + struct dat_evd *rep_evd_handle; + struct dat_evd *rep_conn_handle; + struct sockaddr rep_remote_addr; + DAT_CONN_QUAL rep_server_port; + struct dat_ep_attr rep_attr; + wait_queue_head_t rep_connect_wait; + enum dat_event_number rep_event_num; + struct dat_lmr_triplet rep_pad; /* holds zeroed pad */ + struct dat_lmr *rep_padhandle; /* holds zeroed pad */ + void (*rep_func)(struct rdma_ep *); + void *rep_xprt; /* for rep_func - rpc xprt */ +} rdma_ep_t; + +#define INIT_CQCOUNT(ep) ((ep)->rep_cqcount = (ep)->rep_cqinit) +#define DECR_CQCOUNT(ep) (--(ep)->rep_cqcount) + +/* + * rdma_rep_t -- this structure encapsulates state required to recv + * and complete a reply, asychronously. It needs several pieces of + * state: + * o recv buffer (posted to provider) + * o DAT_LMR_TRIPLET (also donated to provider) + * o status of reply (length, success or not) + * o bookkeeping state to get run by tasklet (list, etc) + * + * These are allocated during initialization, per-transport instance; + * however, the tasklet execution list itself is global, as it should + * always be pretty short. + * + * N of these are associated with a transport instance, and stored in + * rdma_buffer_t. N is the max number of outstanding requests. + */ + +struct rdma_rep; +struct rdma_buffer; +typedef void (*rdma_reply_func)(struct rdma_rep *); + +typedef struct rdma_rep { + unsigned int rr_len; /* actual received reply length */ + struct rdma_buffer *rr_buffer; /* home base for this structure */ + struct rpc_xprt *rr_xprt; /* needed for request/reply matching */ + rdma_reply_func rr_func; /* called by tasklet in softint */ + struct list_head rr_list; /* tasklet list */ + wait_queue_head_t rr_unbind; /* optional unbind wait */ + struct dat_lmr_triplet rr_iov; /* for posting */ + struct dat_lmr *rr_handle; /* handle for mem in rr_iov */ + unsigned char rr_base[MAX_RDMAHDR]; /* start of actual buffer */ +} rdma_rep_t; + +/* + * rdma_req_t -- this structure is central to the request/reply sequence. + * + * N of these are associated with a transport instance, and stored in + * rdma_buffer_t. N is the max number of outstanding requests. + * + * It includes pre-registered buffer memory for send AND recv. + * The recv buffer, however, is not owned by this structure, and + * is "donated" to the hardware when a recv is posted. When a + * reply is handled, the recv buffer used is given back to the + * rdma_req_t associated with the request. + * + * In addition to the basic memory, this structure includes an array + * of DAT_LMR_TRIPLET (IOV) for send operations. The reason is + * that the kDAPL spec says that the iov's passed to *post_{send,recv} + * must not be modified until the DTO completes, because behavior is + * provider-dependent. + * + * NOTES: + * o RDMA_MAX_SEGS is the max number of addressible chunk elements we + * marshal. The number needed varies depending on the iov lists that + * are passed to us, and if physical addressing is used, the layout. + */ + +typedef struct { /* chunk descriptors */ + u64 mr_base; + u64 mr_len; +} rdma_mr_iov; + +typedef union { /* chunk memory handles */ + struct dat_lmr *rl_lmr; + struct rdma_rmr_entry { + struct list_head rmr_freelist; + struct dat_rmr *rmr_handle; + } *rl_rmr; +} rdma_mr_handle; + +typedef struct rdma_req { + unsigned int rl_size; /* actual length of buffer */ + unsigned int rl_niovs; /* 0, 2 or 4 */ + unsigned int rl_nsegs; /* non-zero if chunks */ + struct rdma_buffer *rl_buffer; /* home base for this structure */ + rdma_rep_t *rl_reply; /* holder for reply buffer */ + rdma_mr_handle rl_seg_handles[RDMA_MAX_SEGS]; /* chunk segments */ + struct dat_lmr_triplet rl_send_iov[4]; /* for active requests */ + struct dat_lmr_triplet rl_iov; /* for posting */ + struct dat_lmr *rl_handle; /* handle for mem in rl_iov */ + unsigned char rl_base[MAX_RDMAHDR]; /* start of actual buffer */ +} rdma_req_t; + +#define rpcr_to_rdmar(r) ((rdma_req_t *)(r)->rq_task->tk_buffer - 1) + +/* + * rdma_buffer_t -- holds list/queue of registered memory for + * requests/replies, and client/server credits. + * + * One of these is associated with a transport instance + */ +typedef struct rdma_buffer { + spinlock_t rb_lock; /* protects indexes */ + atomic_t rb_credits; /* most recent server credits */ + int rb_max_requests;/* client max requests */ + struct list_head rb_rmrs; /* optional memory windows */ + int rb_send_index; + rdma_req_t **rb_send_bufs; + int rb_recv_index; + rdma_rep_t **rb_recv_bufs; + char *rb_pool; +} rdma_buffer_t; + +/* + * Internal structure for transport instance creation. This + * exists primarily to keep this file from including xprt.h, + * where rdma_create_data is defined. + * + * TBD: add additional fields/information for creation + */ +struct rdma_create_data_internal { + struct sockaddr addr; + DAT_CONN_QUAL port; + unsigned int max_requests; /* max requests (slots) in flight */ + unsigned int rsize; /* mount rsize - max read hdr+data */ + unsigned int wsize; /* mount wsize - max write hdr+data */ + unsigned int inline_rsize; /* max non-rdma read data payload */ + unsigned int inline_wsize; /* max non-rdma write data payload */ + unsigned int padding; /* non-rdma write header padding */ +}; + +#define RDMA_INLINE_READ_THRESHOLD(rq) \ + (rpcx_to_rdmad((rq)->rq_xprt).inline_rsize + MAX_RPCHDR) + +#define RDMA_INLINE_WRITE_THRESHOLD(rq)\ + (rpcx_to_rdmad((rq)->rq_xprt).inline_wsize + MAX_RPCHDR) + +#define RDMA_INLINE_PAD_VALUE(rq)\ + rpcx_to_rdmad((rq)->rq_xprt).padding + +/* + * Stats for RDMA + * TBD: add a /proc interface. + */ +typedef struct { +/* TBD + total read + total write + write header total + read header total + write chunk count + read chunk count + reply chunk count + fixup data copy count + hardway register count + failed marshal +*/ +} rdma_stats_t; + +/* + * RDMA transport + * + * This is the single object that bundles the state defined + * in the RDMA interface over kDAPL. + * The contained structures are embedded, not pointers, + * for convenience - except for the ia, which can be shared. + * This structure need not be visible externally. + * + * It is allocated and initialized during mount, and released + * during unmount. + */ +typedef struct { + struct rdma_ia *rx_ia; + struct rdma_ep rx_ep; + struct rdma_buffer rx_buf; + struct rdma_create_data_internal rx_data; + rdma_stats_t rx_stats; +} rdma_xprt_t; + +#define rpcx_to_rdmax(x) ((rdma_xprt_t *)(x)->sock) +#define rpcx_to_rdmad(x) (rpcx_to_rdmax(x)->rx_data) + +/* + * Interface Adapter calls + * Creation: + * o ia_t is provided by caller (zero'd first) + * Destruction/close: + * o will NOT kfree the structure + */ +rdma_ia_t *rdma_ia_init(rdma_ia_t *, char *, int); +void rdma_ia_close(rdma_ia_t *); + +/* + * Endpoint calls + * Creation: + * o ep_t is provided by caller but must have been + * initialized via a call to rdma_ep_default_attr. + * Destroy: + * o will NOT kfree the structure + */ +rdma_ep_t *rdma_ep_create(rdma_ep_t *, rdma_ia_t *, + struct rdma_create_data_internal *); +void rdma_ep_default_attr(rdma_ep_t *, rdma_ia_t *, + struct rdma_create_data_internal *); + +int rdma_ep_destroy(rdma_ep_t *, rdma_ia_t *); +int rdma_ep_connect(rdma_ep_t *, rdma_ia_t *, int); +int rdma_ep_disconnect(rdma_ep_t *, rdma_ia_t *); +int rdma_ep_reconnect(rdma_ep_t *, rdma_ia_t *); + +int rdma_ep_post(rdma_ep_t *, rdma_req_t *); +int rdma_ep_post_recv(rdma_ep_t *, rdma_rep_t *); + +/* + * Buffer calls + */ +rdma_buffer_t *rdma_buffer_create(rdma_buffer_t *, rdma_ep_t *, rdma_ia_t *, + struct rdma_create_data_internal *); +void rdma_buffer_destroy(rdma_buffer_t *); + +rdma_req_t *rdma_buffer_get(rdma_buffer_t *); +void rdma_buffer_put(rdma_req_t *); +void rdma_recv_buffer_get(rdma_req_t *); +void rdma_recv_buffer_put(rdma_rep_t *); + +int rdma_register_internal(rdma_ia_t *, void *, int, struct dat_lmr **, + struct dat_lmr_triplet *); +int rdma_deregister_internal(void *); + +int rdma_register_external(rdma_mr_iov *, rdma_mr_handle *, + DAT_RMR_CONTEXT *, int, rdma_xprt_t *); +int rdma_deregister_external(rdma_mr_handle *, rdma_xprt_t *, void *); + +/* + * NFS/RDMA calls (kernel version dependent - rdma_marshal.c) + */ +struct rpc_rqst; +int rdma_marshal_req(struct rpc_rqst *); +int rdma_count_chunks(rdma_rep_t *, int, int, unsigned int **); +void rdma_inline_fixup(struct rpc_rqst *, void *, int); + +#endif /* _RDMA_KDAPL_H */ Index: drivers/infiniband/ulp/nfsrdma/rdma_marshal.c =================================================================== --- drivers/infiniband/ulp/nfsrdma/rdma_marshal.c (revision 0) +++ drivers/infiniband/ulp/nfsrdma/rdma_marshal.c (revision 0) @@ -0,0 +1,881 @@ +/* + * Copyright (c) 2003, 2004, Network Appliance, Inc. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * Neither the name of the Network Appliance, Inc. nor the names of + * its contributors may be used to endorse or promote products + * derived from this software without specific prior written + * permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* + * rdma_marshal.c + * + * This file contains the guts of the RPC RDMA protocol, and + * does marshaling/unmarshaling, etc. It is also where any + * changes related to the various Linux kernel generations + * live. These are primarily related to memory representation + * (iovs, pages, etc). + * + * Where possible, the differences wrt specific kernels are + * discussed. + */ + +#include "rdma_kdapl.h" +#include + +/* + * local functions + */ + +typedef enum { noch = 0, readch, areadch, writech, replych } chunktype; +static int rdma_inline_pullup(struct rpc_rqst *, int); + + +/* + * Macros to hide differences from 2.4.18 and 2.4.20 in RPC/XDR code + */ + +#include +#include +#include +#include + +#if LINUX_VERSION_CODE <= KERNEL_VERSION(2,4,19) + +#define XDR_TARGET struct rpc_iov +#define pos0(r) (r)->io_vec->iov_len + +#define RPC_SEND_VECS(rqst) (rqst)->rq_snr +#define RPC_SEND_SEG0(rqst) (rqst)->rq_snd_buf.io_vec[0].iov_base +#define RPC_SEND_LEN0(rqst) (rqst)->rq_snd_buf.io_vec[0].iov_len + +static inline int +RPC_SEND_COPY(struct rpc_rqst *rqst, int n, int len, char *destp) +{ + if (len > rqst->rq_snd_buf.io_vec[n].iov_len) + len = rqst->rq_snd_buf.io_vec[n].iov_len; + memcpy(destp, rqst->rq_snd_buf.io_vec[n].iov_base, len); + return len; +} + +#define RPC_RECV_VECS(rqst) (rqst)->rq_rnr +#define RPC_RECV_SEG0(rqst) (rqst)->rq_rcv_buf.io_vec[0].iov_base +#define RPC_RECV_LEN0(rqst) (rqst)->rq_rcv_buf.io_vec[0].iov_len + +static inline int +RPC_RECV_COPY(struct rpc_rqst *rqst, int n, int len, char *src) +{ + if (len > rqst->rq_rcv_buf.io_vec[n].iov_len) + len = rqst->rq_rcv_buf.io_vec[n].iov_len; + memcpy(rqst->rq_rcv_buf.io_vec[n].iov_base, src, len); + return len; +} + +#else /* >= 2.4.20 */ + +#define XDR_TARGET struct xdr_buf +#define pos0(x) (x)->head[0].iov_len + +#define RPC_SEND_VECS(rqst) \ + (1 + (PAGE_ALIGN((rqst)->rq_snd_buf.page_base + \ + (rqst)->rq_snd_buf.page_len) >> PAGE_SHIFT) + \ + ((rqst)->rq_snd_buf.tail[0].iov_len != 0)) +#define RPC_SEND_SEG0(rqst) (rqst)->rq_svec[0].iov_base +#define RPC_SEND_LEN0(rqst) (rqst)->rq_svec[0].iov_len + +static inline int +RPC_SEND_COPY(struct rpc_rqst *rqst, int n, int len, char *dest) +{ + char *src; + int lim; + if (--n == 0) { + lim = PAGE_SIZE - rqst->rq_snd_buf.page_base; + } else { + lim = rqst->rq_snd_buf.page_base + rqst->rq_snd_buf.page_len; + lim = PAGE_ALIGN(lim); + if (n < (lim >> PAGE_SHIFT)) + lim = PAGE_SIZE; + else if (n == (lim >> PAGE_SHIFT)) + lim &= ~PAGE_MASK; + else { + if (len > rqst->rq_snd_buf.tail[0].iov_len) + len = rqst->rq_snd_buf.tail[0].iov_len; + memcpy(dest, rqst->rq_snd_buf.tail[0].iov_base, len); + return len; + } + } + if (len > lim) + len = lim; + + if (PageHighMem(rqst->rq_snd_buf.pages[n])) { + src = kmap(rqst->rq_snd_buf.pages[n]); + memcpy(dest, src, len); + kunmap(rqst->rq_snd_buf.pages[n]); + } else { + src = page_address(rqst->rq_snd_buf.pages[n]) + + (n ? 0 : rqst->rq_snd_buf.page_base); + memcpy(dest, src, len); + } + return len; +} + +#define RPC_RECV_VECS(rqst) \ + (1 + (PAGE_ALIGN((rqst)->rq_rcv_buf.page_base + \ + (rqst)->rq_rcv_buf.page_len) >> PAGE_SHIFT) + \ + ((rqst)->rq_rcv_buf.tail[0].iov_len != 0)) +#define RPC_RECV_SEG0(rqst) (rqst)->rq_rcv_buf.head[0].iov_base +#define RPC_RECV_LEN0(rqst) (rqst)->rq_rcv_buf.head[0].iov_len + +static inline int +RPC_RECV_COPY(struct rpc_rqst *rqst, int n, int len, char *src) +{ + char *dest; + int lim; + if (--n == 0) { + lim = PAGE_SIZE - rqst->rq_rcv_buf.page_base; + } else { + lim = rqst->rq_rcv_buf.page_base + rqst->rq_rcv_buf.page_len; + lim = PAGE_ALIGN(lim); + if (n < (lim >> PAGE_SHIFT)) + lim = PAGE_SIZE; + else if (n == (lim >> PAGE_SHIFT)) + lim &= ~PAGE_MASK; + else { + if (len > rqst->rq_rcv_buf.tail[0].iov_len) + len = rqst->rq_rcv_buf.tail[0].iov_len; + memcpy(rqst->rq_rcv_buf.tail[0].iov_base, src, len); + return len; + } + } + if (len > lim) + len = lim; + + if (PageHighMem(rqst->rq_rcv_buf.pages[n])) { + dest = kmap(rqst->rq_rcv_buf.pages[n]); + memcpy(dest, src, len); + kunmap(rqst->rq_rcv_buf.pages[n]); + } else { + dest = page_address(rqst->rq_rcv_buf.pages[n]) + + (n ? 0 : rqst->rq_rcv_buf.page_base); + memcpy(dest, src, len); + } + return len; +} + +#endif + +static unsigned int rdma_create_chunks(struct rpc_rqst *, XDR_TARGET *, + struct rdma_msg *, chunktype); +static int rdma_convert_virt(XDR_TARGET *, int, rdma_mr_iov *, int); +static int rdma_convert_phys(XDR_TARGET *, int, rdma_mr_iov *, int); + +#undef Dprintk +#if RPCRDMA_DEBUG +#define Dprintk(n, x) if (Debug >= n) printk x +extern int Debug; +#else +#define Dprintk(n, x) +#endif + +/* + * Marshal a request: the primary job of this routine is to choose + * the transfer modes. See comments below. + * + * Uses multiple kDAPL IOVs for a request: + * [0] -- RPC RDMA header, which uses memory from the *start* of the + * preregistered buffer that already holds the RPC data in + * its middle. + * [1] -- the RPC header/data, marshaled by RPC and the NFS protocol. + * [2] -- optional padding. + * [3] -- if padded, header only in [1] and data here. + */ + +#if RPCRDMA_DEBUG +static const char transfertypes[][12] = { + "pure inline", /* no chunks */ + " read chunk", /* some argument via rdma read */ + "*read chunk", /* entire request via rdma read */ + "write chunk", /* some result via rdma write */ + "reply chunk" /* entire reply via rdma write */ +}; +#endif + +int +rdma_marshal_req(struct rpc_rqst *rqst) +{ + struct rpc_xprt *xprt = rqst->rq_xprt; + rdma_req_t *req = rpcr_to_rdmar(rqst); + unsigned char *base; + unsigned int hdrlen, rpclen, padlen, buflen; + chunktype rtype, wtype; + struct rdma_msg_no_chunks *headerp; + + if (xprt->stream) { /* trim the stream marker and total len */ + RPC_SEND_SEG0(rqst) += 4; + RPC_SEND_LEN0(rqst) -= 4; + rqst->rq_slen -= 4; + } + + /* + * rpclen gets amount of data in first buffer, which is the + * pre-registered buffer. + */ + base = RPC_SEND_SEG0(rqst); + rpclen = RPC_SEND_LEN0(rqst); + buflen = ((unsigned char *)(req + 1) + req->rl_size) - base; + + /* sanity check -- these should match */ + RDMA_ASSERT(buflen <= req->rl_size, + "rdma_marshal_req cannot match request buffers"); + + /* build RDMA header in private area at front */ + headerp = (struct rdma_msg_no_chunks *) req->rl_base; + /* don't htonl XID, it's already done in request */ + headerp->rdma_xid = rqst->rq_xid; + headerp->rdma_vers = xdr_one; + headerp->rdma_credit = htonl(req->rl_buffer->rb_max_requests); + headerp->rdma_type = __constant_htonl(RDMA_MSG); + + /* + * Chunks needed for results? + * + * o If the expected result is under the inline threshold, all ops + * return as inline (but see later). + * o Large non-read ops return as a single reply chunk. + * o Large read ops return data as write chunk(s), header as inline. + * + * Note: the NFS code sending down multiple result segments implies + * the op is one of read, readdir[plus] or readlink. + */ + if (rqst->rq_rcv_buf.len <= RDMA_INLINE_READ_THRESHOLD(rqst)) { + wtype = noch; + } else if (RPC_RECV_VECS(rqst) == 1) { + wtype = replych; + } else { + /* TBD fix this intimacy with a better test! */ + #include + if (rqst->rq_task->tk_client->cl_prog == NFS_PROGRAM && + rqst->rq_task->tk_client->cl_vers == 3 && + rqst->rq_task->tk_msg.rpc_proc->p_proc == NFS3PROC_READ) { + wtype = writech; + } else { + wtype = replych; + } + } + + /* + * Chunks needed for arguments? + * + * o If the total request is under the inline threshold, all ops + * are sent as inline. + * o Large non-write ops are sent with the entire message as a + * single read chunk (protocol 0-position special case). + * o Large write ops transmit data as read chunk(s), header as + * inline. + * + * Note: the NFS code sending down multiple argument segments + * implies the op is a write. + */ + if (rqst->rq_slen <= RDMA_INLINE_WRITE_THRESHOLD(rqst)) { + rtype = noch; + } else if (RPC_SEND_VECS(rqst) == 1) { + rtype = areadch; + } else { + rtype = readch; + } + + /* The following simplification is not true forever */ + RDMA_ASSERT(rtype == noch || wtype == noch, + "rdma_marshal_req read and write chunks needed"); + + if (rpcx_to_rdmax(xprt)->rx_ia->ri_memreg_strategy == 0 && + (rtype != noch || wtype != noch)) { + /* forced to "pure inline"? */ + Dprintk(0, ("rdma_marshal_req: too much data for inline\n")); + return -1; + } + + hdrlen = sizeof *headerp; + padlen = 0; + + /* + * Pull up any extra send data into the preregistered buffer. + * When padding is in use and applies to the transfer, insert + * it and change the message type. + */ + if (rtype == noch) { + + padlen = rdma_inline_pullup(rqst, RDMA_INLINE_PAD_VALUE(rqst)); + + if (padlen) { + struct rdma_msg_padded *pheaderp = + (struct rdma_msg_padded *) headerp; + pheaderp->rdma_type = __constant_htonl(RDMA_MSGP); + pheaderp->rdma_align = htonl(RDMA_INLINE_PAD_VALUE(rqst)); + pheaderp->rdma_thresh = __constant_htonl(RDMA_INLINE_PAD_THRESH); + pheaderp->rdma_nochunks[0] = xdr_zero; + pheaderp->rdma_nochunks[1] = xdr_zero; + pheaderp->rdma_nochunks[2] = xdr_zero; + hdrlen = sizeof *pheaderp; + RDMA_ASSERT(wtype == noch, + "rdma_marshal_req padded write chunk"); + + } else { + headerp->rdma_nochunks[0] = xdr_zero; + headerp->rdma_nochunks[1] = xdr_zero; + headerp->rdma_nochunks[2] = xdr_zero; + /* new length after pullup */ + rpclen = RPC_SEND_LEN0(rqst); + /* + * Currently we try to not actually use read inline. + * Reply chunks have the desirable property that + * they land, packed, directly in the target buffers + * without headers, so they require no fixup. The + * additional RDMA Write op sends the same amount + * of data, streams on-the-wire and adds no overhead + * on receive. Therefore, we request a reply chunk + * for non-writes wherever feasible and efficient. + */ + if (wtype == noch && + rpcx_to_rdmax(xprt)->rx_ia->ri_memreg_strategy > 1) + wtype = replych; + } + } + + /* + * Marshal chunks. This routine will return the header length + * consumed by marshaling. + */ + if (rtype != noch) { + hdrlen = rdma_create_chunks(rqst, &rqst->rq_snd_buf, + (struct rdma_msg *)headerp, rtype); + wtype = rtype; /* simplify Dprintk */ + + } else if (wtype != noch) { + hdrlen = rdma_create_chunks(rqst, &rqst->rq_rcv_buf, + (struct rdma_msg *)headerp, wtype); + } + + if (hdrlen == 0) + return -1; + + Dprintk(1, ("rdma_marshal_req: %s: " + "hdrlen %d rpclen %d padlen %d buflen %d " + "headerp 0x%p base 0x%p lmr 0x%x\n", + transfertypes[wtype], hdrlen, rpclen, padlen, buflen, + headerp, base, req->rl_iov.lmr_context)); + + /* + * initialize send_iov's - normally only two: rdma chunk header and + * single preregistered header buffer, but if padding is present, + * then use a preregistered (and zeroed) pad buffer between the RPC + * header and any write data. In all non-rdma cases, any following + * data has been copied into the header buffer. + * + * Be sure to account for any trimmed 4-byte length marker. + */ + req->rl_send_iov[0].virtual_address = req->rl_iov.virtual_address; + req->rl_send_iov[0].segment_length = hdrlen; + req->rl_send_iov[0].lmr_context = req->rl_iov.lmr_context; + + req->rl_send_iov[1].virtual_address = + req->rl_iov.virtual_address + MAX_RDMAHDR; + if (xprt->stream) + req->rl_send_iov[1].virtual_address += 4; + req->rl_send_iov[1].segment_length = rpclen; + req->rl_send_iov[1].lmr_context = req->rl_iov.lmr_context; + + req->rl_niovs = 2; + + if (padlen) { + rdma_ep_t *ep = &rpcx_to_rdmax(xprt)->rx_ep; + + req->rl_send_iov[2].virtual_address = ep->rep_pad.virtual_address; + req->rl_send_iov[2].segment_length = padlen; + req->rl_send_iov[2].lmr_context = ep->rep_pad.lmr_context; + + req->rl_send_iov[3].virtual_address = + req->rl_send_iov[1].virtual_address + rpclen; + req->rl_send_iov[3].segment_length = rqst->rq_slen - rpclen; + req->rl_send_iov[3].lmr_context = req->rl_iov.lmr_context; + + req->rl_niovs = 4; + } + + if (xprt->stream) { /* restore the stream marker and total len */ + RPC_SEND_SEG0(rqst) -= 4; + RPC_SEND_LEN0(rqst) += 4; + rqst->rq_slen += 4; + } + + return 0; +} + +/* + * Create read/write chunk lists, and reply chunks, for RDMA + * + * Assume check against THRESHOLD has been done, and chunks are required. + * Assume only encoding one list entry for read|write chunks. The NFSv3 + * protocol is simple enough to allow this as it only has a single "bulk + * result" in each procedure - complicated NFSv4 COMPOUNDs are not. (The + * RDMA/Sessions NFSv4 proposal addresses this for future v4 revs.) + * + * Algorithm: + * For each entry in the recv vector: + * o attempt to coalesce with next entry to create fewer ranges + * o when no longer contiguous, create a chunk: + * register memory + * marshal chunk + * Save lmr_handle in embedded array in rdma_req_t + * + * When used for a single reply chunk (which is a special write + * chunk used for the entire reply, rather than just the data), it + * is used primarily for READDIR and READLINK which would otherwise + * be severely size-limited by a small rdma inline read max. The server + * response will come back as an RDMA Write, followed by a message + * of type RDMA_NOMSG carrying the xid and length. As a result, reply + * chunks do not provide data alignment. + * + * Encoding key for single-list chunks (HLOO = Handle32 Length32 Offset64): + * + * Read chunklist (a linked list): + * N elements, position P (same P for all chunks of same arg!): + * 1 - PHLOO - 1 - PHLOO - ... - 1 - PHLOO - 0 + * + * Write chunklist (a list of (one) counted array): + * N elements: + * 1 - N - HLOO - HLOO - ... - HLOO - 0 + * + * Reply chunk (a counted array): + * N elements: + * 1 - N - HLOO - HLOO - ... - HLOO + * + * Changes for 2.4.20: + * + * This function will need to change for 2.4.20, as the extra + * data is passed as a page list, and not struct iovec. + * (The page lists in 2.4.20 are not kmap'd, which in the RDMA + * case is a big advantage, since we don't need them mapped.) + */ + +static unsigned int +rdma_create_chunks(struct rpc_rqst *rqst, XDR_TARGET *target, + struct rdma_msg *headerp, chunktype type) +{ + rdma_req_t *req = rpcr_to_rdmar(rqst); + rdma_xprt_t *r_xprt = rpcx_to_rdmax(rqst->rq_xprt); + int nsegs = 0; + int niovs, pos; + rdma_mr_iov segs[RDMA_MAX_SEGS], *seg = segs; + DAT_RMR_CONTEXT rmr_context; + struct xdr_read_chunk *cur_rchunk = NULL; + struct xdr_write_array *warray = NULL; + struct xdr_write_chunk *cur_wchunk = NULL; + unsigned int *iptr = (unsigned int *)(headerp + 1); + + pos = (type == replych || type == areadch ? 0 : pos0(target)); + /* Check for coalesce, optionally convert to physical for RMR bind */ + if (r_xprt->rx_ia->ri_memreg_strategy > 1) { + niovs = rdma_convert_phys(target, pos != 0, seg, RDMA_MAX_SEGS); + } else { + niovs = rdma_convert_virt(target, pos != 0, seg, RDMA_MAX_SEGS); + } + if (niovs == 0) { + return 0; + } + + if (type == readch || type == areadch) { + /* a read chunk - server will RDMA Read our memory */ + cur_rchunk = (struct xdr_read_chunk *) iptr; + } else { + /* a write or reply chunk - server will RDMA Write our memory */ + *iptr++ = xdr_zero; /* encode a NULL read chunk list */ + if (type == replych) { + *iptr++ = xdr_zero; /* a NULL write chunk list */ + } + warray = (struct xdr_write_array *) iptr; + cur_wchunk = (struct xdr_write_chunk *) (warray + 1); + } + + while (niovs--) { + /* bind/register the memory, then build chunk. */ + if (rdma_register_external(seg, + &req->rl_seg_handles[nsegs], + &rmr_context, + cur_wchunk != NULL, + r_xprt)) { + goto out; + } + if (cur_rchunk) { /* read */ + cur_rchunk->discrim = xdr_one; + /* all read chunks have the same "position" */ + cur_rchunk->position = htonl(pos); + cur_rchunk->target.handle = htonl(rmr_context); + cur_rchunk->target.length = htonl(seg->mr_len); + xdr_encode_hyper((u32 *)&cur_rchunk->target.offset, + seg->mr_base); + Dprintk(1, ("rdma_create_chunks: read chunk " + "elem %lld at 0x%llx:0x%x pos %d (%s)\n", + seg->mr_len, seg->mr_base, rmr_context, + pos, niovs ? "more" : "last")); + cur_rchunk++; + } else { /* write/reply */ + cur_wchunk->target.handle = htonl(rmr_context); + cur_wchunk->target.length = htonl(seg->mr_len); + xdr_encode_hyper((u32 *)&cur_wchunk->target.offset, + seg->mr_base); + Dprintk(1, ("rdma_create_chunks: %s chunk " + "elem %lld at 0x%llx:0x%x (%s)\n", + (type == replych) ? "reply" : "write", + seg->mr_len, seg->mr_base, rmr_context, + niovs ? "more" : "last")); + cur_wchunk++; + } + seg++; + nsegs++; + } + + /* success. all failures return above */ + req->rl_nsegs = nsegs; + + RDMA_ASSERT(nsegs > 0, "rdma_create_chunks internal error"); + + /* + * finish off header. If write, marshal discrim and nchunks. + */ + if (cur_rchunk) { + iptr = (unsigned int *) cur_rchunk; + *iptr++ = xdr_zero; /* finish the read chunk list */ + *iptr++ = xdr_zero; /* encode a NULL write chunk list */ + *iptr++ = xdr_zero; /* encode a NULL reply chunk */ + } else { + warray->discrim = xdr_one; + warray->nchunks = htonl(nsegs); + iptr = (unsigned int *) cur_wchunk; + if (type == writech) { + *iptr++ = xdr_zero; /* finish the write chunk list */ + *iptr++ = xdr_zero; /* encode a NULL reply chunk */ + } + } + + /* + * Return header size. + */ + return (unsigned char *)iptr - (unsigned char *)headerp; + +out: + while (--nsegs >= 0) { + rdma_deregister_external(&req->rl_seg_handles[nsegs], r_xprt, NULL); + } + return 0; +} + +/* + * Copy write data inline. + * This function is used for "small" requests. Data which is passed + * to RPC via iovecs (or page list) is copied directly into the + * pre-registered memory buffer for this request. For small amounts + * of data, this is efficient. The cutoff value is tunable. + */ +static int +rdma_inline_pullup(struct rpc_rqst *rqst, int pad) +{ + int i, n, curlen; + unsigned char *destp; + + destp = RPC_SEND_SEG0(rqst); + curlen = RPC_SEND_LEN0(rqst); + destp += curlen; + /* + * Do optional padding where it makes sense. Alignment of write + * payload can help the server, if our setting is accurate. + */ + pad -= (curlen + sizeof(struct rdma_msg_padded)); + if (pad < 0 || rqst->rq_slen - curlen < RDMA_INLINE_PAD_THRESH) { + pad = 0; /* don't pad this request */ + } + + n = RPC_SEND_VECS(rqst); + for (i = 1; i < n; i++) { + curlen = RPC_SEND_COPY(rqst, i, 65536, destp); + RPC_SEND_LEN0(rqst) += curlen; + destp += curlen; + } + /* header now contains entire send message */ + return pad; +} + +/* + * Coalesce chunks and/or convert a kernel-virtual iov list to physical. + * + * It's worth it to coalesce adjacent regions because each one + * requires its own RDMA op. + */ +static int +rdma_convert_physiov(struct kvec *iov, int niovs, + rdma_mr_iov *seg, int nsegs) +{ + unsigned char *base = iov->iov_base, *next; + unsigned long more; + u64 phys; + u64 len = iov->iov_len; + struct page *pg; + int n = 0, off, m; + + for (;;) { + ++iov; + --niovs; + more = 0; + next = 0; + off = (unsigned long)base & ~PAGE_MASK; + m = PAGE_SIZE; + + next = (unsigned char *)PAGE_ALIGN((unsigned long)(base + 1)); + pg = vmalloc_to_page(base); + phys = (pg ? page_to_phys(pg) : __pa(base) - off); + + /* check for coalesce */ + while (m - off < len) { + pg = vmalloc_to_page(next); + if (phys + m != (pg ? page_to_phys(pg) : __pa(next))) { + --iov; /* back up the truck */ + ++niovs; + goto out; + } + m += PAGE_SIZE; /* coalesce physical */ + next += PAGE_SIZE; + } + /* check for coalesce across iov's too */ + if (niovs > 0) { + /* TBD */ + } + out: + phys += off; + if (len > m - off) { + more = len - (m - off); + len = m - off; + } else { + more = 0; + } + seg->mr_base = phys; + seg->mr_len = len; + ++seg; + ++n; + if (niovs == 0) /* done */ + return n; + if (n >= nsegs) /* overflow */ + return 0; + if (more) { + base = next; + len = more; + } else { + base = iov->iov_base; + len = iov->iov_len; + } + } + /* notreached */ +} + +#if LINUX_VERSION_CODE <= KERNEL_VERSION(2,4,19) + +/* + * Coalesce chunks in a kernel-virtual iov list. + */ +static int +rdma_convert_virt(struct rpc_iov *target, int first, + rdma_mr_iov *seg, int nsegs) +{ + struct iovec *iov = target->io_vec + first; + int niovs = target->io_nr - first; + unsigned char *base = iov->iov_base; + int len = iov->iov_len; + int n = 0; + + for (;;) { + ++iov; + --niovs; + if (niovs > 0 && iov->iov_base == base + len) { + len += iov->iov_len; /* coalesce */ + continue; + } + seg->mr_base = (unsigned long) base; + seg->mr_len = len; + ++seg; + ++n; + if (niovs == 0) /* done */ + return n; + if (n >= nsegs) /* overflow */ + return 0; + base = iov->iov_base; + len = iov->iov_len; + } + /* notreached */ +} + +static int +rdma_convert_phys(struct rpc_iov *target, int first, + rdma_mr_iov *seg, int nsegs) +{ + return rdma_convert_physiov(target->io_vec + first, + target->io_nr - first, seg, nsegs); +} + +#else /* >= 2.4.20 */ + +static int +rdma_convert_virt(struct xdr_buf *xdrbuf, int first, rdma_mr_iov *seg, int nsegs) +{ +/* + * Need to kmap, return vaddrs, and find a place to kunmap. + */ + printk("rdma_convert_virt: Memreg=1 not yet supported on this Linux version\n"); + return 0; +} + +static int +rdma_convert_phys(struct xdr_buf *xdrbuf, int first, rdma_mr_iov *seg, int nsegs) +{ + int len, n = 0, p; + + if (first == 0) { + n = rdma_convert_physiov(xdrbuf->head, 1, seg, nsegs); + if (n == 0) + return 0; + } + + if (xdrbuf->page_len) { + /* This code can and should be improved to check for + * coalescing opportunities, like the code above. */ + if (n == nsegs) + return 0; + seg[n].mr_base = page_to_phys(xdrbuf->pages[0]) + xdrbuf->page_base; + seg[n].mr_len = PAGE_SIZE - xdrbuf->page_base; + len = xdrbuf->page_len - seg[n].mr_len; + ++n; + p = 1; + while (len > 0) { + if (n == nsegs) + return 0; + seg[n].mr_base = page_to_phys(xdrbuf->pages[p]); + if (len > PAGE_SIZE) + seg[n].mr_len = PAGE_SIZE; + else + seg[n].mr_len = len; + ++n; + ++p; + len -= PAGE_SIZE; + } + } + if (xdrbuf->tail[0].iov_len) { + if (n == nsegs) + return 0; + p = rdma_convert_physiov(xdrbuf->tail, 1, &seg[n], nsegs - n); + if (p == 0) + return 0; + n += p; + } + return n; +} +#endif + +/* + * Chase down a received write or reply chunklist to get length + * RDMA'd by server. See map at rdma_create_chunks()! :-) + */ +int +rdma_count_chunks(rdma_rep_t *rep, int max, int wrchunk, unsigned int **iptrp) +{ + unsigned int i, total_len; + struct xdr_write_chunk *cur_wchunk; + + i = ntohl(**iptrp); /* get array count */ + if (i > max) { + return -1; + } + cur_wchunk = (struct xdr_write_chunk *) (*iptrp + 1); + total_len = 0; + while (i--) { +#if RPCRDMA_DEBUG + unsigned long long off; + xdr_decode_hyper((u32 *)&cur_wchunk->target.offset, &off); +#endif + Dprintk(1, ("rdma_count_chunks: chunk %d at 0x%llx:0x%x\n", + ntohl(cur_wchunk->target.length), off, + ntohl(cur_wchunk->target.handle))); + total_len += ntohl(cur_wchunk->target.length); + ++cur_wchunk; + } + /* check and adjust for properly terminated write chunk */ +#if 0 /* XXX wtf? */ + if (wrchunk && *((unsigned int *) cur_wchunk)++ != xdr_zero) { + return -1; + } +#endif + if ((unsigned char *) cur_wchunk > rep->rr_base + rep->rr_len) { + return -1; + } + *iptrp = (unsigned int *) cur_wchunk; + return total_len; +} + +/* + * Scatter inline received data back into provided iov's. + * + * TBD keep a statistic fixup += len + */ +void +rdma_inline_fixup(struct rpc_rqst *rqst, void *srcp, int len) +{ + int i, j, n; + + if (len > 0) { + j = RPC_RECV_LEN0(rqst); + if (j >= len) { + j = len; + RPC_RECV_LEN0(rqst) = j; /* write chunk header fixup */ + } + /* Shift pointer for first receive segment only */ + RPC_RECV_SEG0(rqst) = srcp; + srcp = (char *) srcp + j; + len -= j; + } + if (len > 0) { + n = RPC_RECV_VECS(rqst); + for (i = 1; i < n; i++) { + j = RPC_RECV_COPY(rqst, i, len, srcp); + srcp = (char *) srcp + j; + if ((len -= j) == 0) + break; + } + } +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,4,19) + /* TBD avoid a warning from call_decode() */ + rqst->rq_private_buf = rqst->rq_rcv_buf; +#endif + + RDMA_ASSERT(len == 0, "rdma_inline_fixup too much inline data"); +} Index: drivers/infiniband/ulp/nfsrdma/rdma_proto.h =================================================================== --- drivers/infiniband/ulp/nfsrdma/rdma_proto.h (revision 0) +++ drivers/infiniband/ulp/nfsrdma/rdma_proto.h (revision 0) @@ -0,0 +1,101 @@ +/* + * Copyright (c) 2003, 2004, Network Appliance, Inc. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * Neither the name of the Network Appliance, Inc. nor the names of + * its contributors may be used to endorse or promote products + * derived from this software without specific prior written + * permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef _RDMA_PROTO_H +#define _RDMA_PROTO_H + +struct xdr_rdma_segment { + uint32_t handle; /* Registered memory handle */ + uint32_t length; /* Length of the chunk in bytes */ + uint64_t offset; /* Chunk virtual address or offset */ +}; + +/* + * read chunk(s), encoded as a linked list. + */ +struct xdr_read_chunk { + uint32_t discrim; /* 1 indicates presence */ + uint32_t position; /* Position in XDR stream */ + struct xdr_rdma_segment target; +}; + +/* + * write chunk(s), encoded as a counted array. + */ +struct xdr_write_array { + uint32_t discrim; /* 1 indicates presence */ + uint32_t nchunks; /* Array count */ + /* struct xdr_write_chunk array<> */ +}; + +/* + * actual write chunk, and reply chunk. + */ +struct xdr_write_chunk { + struct xdr_rdma_segment target; +}; + +struct rdma_msg { + uint32_t rdma_xid; /* Mirrors the RPC header xid */ + uint32_t rdma_vers; /* Version of this protocol */ + uint32_t rdma_credit; /* Buffers requested/granted */ + uint32_t rdma_type; /* Type of message (enum rdma_proc) */ + /* rdma_body rdma_body; */ +}; + +struct rdma_msg_no_chunks { + uint32_t rdma_xid; /* Mirrors the RPC header xid */ + uint32_t rdma_vers; /* Version of this protocol */ + uint32_t rdma_credit; /* Buffers requested/granted */ + uint32_t rdma_type; /* Type of message (enum rdma_proc) */ + uint32_t rdma_nochunks[3]; /* 3 empty chunk lists */ +}; + +struct rdma_msg_padded { + uint32_t rdma_xid; /* Mirrors the RPC header xid */ + uint32_t rdma_vers; /* Version of this protocol */ + uint32_t rdma_credit; /* Buffers requested/granted */ + uint32_t rdma_type; /* Type of message (enum rdma_proc) */ + uint32_t rdma_align; /* Padding alignment */ + uint32_t rdma_thresh; /* Padding threshold */ + uint32_t rdma_nochunks[3]; /* 3 empty chunk lists */ +}; + +enum rdma_proc { + RDMA_MSG = 0, /* An RPC call or reply msg */ + RDMA_NOMSG = 1, /* An RPC call or reply msg - separate body */ + RDMA_MSGP = 2, /* An RPC call or reply msg with padding */ + RDMA_DONE = 3 /* Client signals reply completion */ +}; + +#endif /* _RDMA_PROTO_H */ Index: drivers/infiniband/ulp/nfsrdma/Makefile =================================================================== --- drivers/infiniband/ulp/nfsrdma/Makefile (revision 0) +++ drivers/infiniband/ulp/nfsrdma/Makefile (revision 0) @@ -0,0 +1,5 @@ +EXTRA_CFLAGS += -DRPCRDMA_DEBUG -Idrivers/dat + +obj-$(CONFIG_NFSRDMA) = rdma_xprt.o + +rdma_xprt-objs := rdma_kdapl.o rdma_transport.o rdma_marshal.o Index: drivers/infiniband/Kconfig =================================================================== --- drivers/infiniband/Kconfig (revision 2723) +++ drivers/infiniband/Kconfig (working copy) @@ -23,4 +23,8 @@ source "drivers/infiniband/ulp/ipoib/Kco source "drivers/infiniband/ulp/sdp/Kconfig" +source "drivers/infiniband/ulp/dat-provider/Kconfig" + +source "drivers/infiniband/ulp/nfsrdma/Kconfig" + endmenu Index: drivers/infiniband/Makefile =================================================================== --- drivers/infiniband/Makefile (revision 2723) +++ drivers/infiniband/Makefile (working copy) @@ -2,3 +2,5 @@ obj-$(CONFIG_INFINIBAND) += core/ obj-$(CONFIG_INFINIBAND_MTHCA) += hw/mthca/ obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/ obj-$(CONFIG_INFINIBAND_SDP) += ulp/sdp/ +obj-$(CONFIG_INFINIBAND_DAT_PROVIDER) += ulp/dat-provider/ +obj-$(CONFIG_INFINIBAND_DAT_PROVIDER) += ulp/nfsrdma/ From ardavis at ichips.intel.com Mon Jun 27 15:31:09 2005 From: ardavis at ichips.intel.com (Arlin Davis) Date: Mon, 27 Jun 2005 15:31:09 -0700 Subject: [openib-general] Re: [PATCH] [udapl] a simple uDAPL test Message-ID: <42C07E2D.3040301@ichips.intel.com> James, Here is a simple uDAPL test that you can add to your userspace directory. -arlin Signed-off-by: Arlin Davis Index: dtest/dtest.c =================================================================== --- dtest/dtest.c (revision 0) +++ dtest/dtest.c (revision 0) @@ -0,0 +1,1700 @@ +/* + * Copyright (c) 2005 Intel Corporation. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id: $ + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef DAPL_PROVIDER +#define DAPL_PROVIDER "OpenIB1_2" +#endif + +#define MAX_POLLING_CNT 50000 + +/* Header files needed for DAT/uDAPL */ +#include "dat/udat.h" + +/* definitions */ +#define SERVER_CONN_QUAL 71123 +#define DTO_TIMEOUT (1000*1000*5) +#define DTO_FLUSH_TIMEOUT (1000*1000*2) +#define CONN_TIMEOUT (1000*1000*10) +#define SERVER_TIMEOUT (1000*1000*20) +#define RDMA_BUFFER_SIZE (64) + +/* Global DAT vars */ +static DAT_IA_HANDLE h_ia = DAT_HANDLE_NULL; +static DAT_PZ_HANDLE h_pz = DAT_HANDLE_NULL; +static DAT_EP_HANDLE h_ep = DAT_HANDLE_NULL; +static DAT_PSP_HANDLE h_psp = DAT_HANDLE_NULL; +static DAT_CR_HANDLE h_cr = DAT_HANDLE_NULL; + +static DAT_EVD_HANDLE h_async_evd = DAT_HANDLE_NULL; +static DAT_EVD_HANDLE h_dto_evd = DAT_HANDLE_NULL; +static DAT_EVD_HANDLE h_cr_evd = DAT_HANDLE_NULL; +static DAT_EVD_HANDLE h_conn_evd = DAT_HANDLE_NULL; +static DAT_CNO_HANDLE h_dto_cno = DAT_HANDLE_NULL; + +/* RDMA buffers */ +static DAT_LMR_HANDLE h_lmr_send = DAT_HANDLE_NULL; +static DAT_LMR_HANDLE h_lmr_recv = DAT_HANDLE_NULL; +static DAT_LMR_CONTEXT lmr_context_send; +static DAT_LMR_CONTEXT lmr_context_recv; +static DAT_RMR_CONTEXT rmr_context_send; +static DAT_RMR_CONTEXT rmr_context_recv; +static DAT_VLEN registered_size_send; +static DAT_VLEN registered_size_recv; +static DAT_VADDR registered_addr_send; +static DAT_VADDR registered_addr_recv; + +/* Initial msg receive buf, RMR exchange, and Rdma-write notification */ +#define MSG_BUF_COUNT 3 +static DAT_RMR_TRIPLET rmr_recv_msg[ MSG_BUF_COUNT ]; +static DAT_LMR_HANDLE h_lmr_recv_msg = DAT_HANDLE_NULL; +static DAT_LMR_CONTEXT lmr_context_recv_msg; +static DAT_RMR_CONTEXT rmr_context_recv_msg; +static DAT_VLEN registered_size_recv_msg; +static DAT_VADDR registered_addr_recv_msg; + +/* message send buffer */ +static DAT_RMR_TRIPLET rmr_send_msg; +static DAT_LMR_HANDLE h_lmr_send_msg = DAT_HANDLE_NULL; +static DAT_LMR_CONTEXT lmr_context_send_msg; +static DAT_RMR_CONTEXT rmr_context_send_msg; +static DAT_VLEN registered_size_send_msg; +static DAT_VADDR registered_addr_send_msg; +static DAT_EP_ATTR ep_attr; +char hostname[256] = {0}; + +/* rdma pointers */ +char *rbuf = NULL; +char *sbuf = NULL; +int status; + +/* timers */ +double start,stop,total_us,total_sec; +struct { + double total; + double open; + double reg; + double unreg; + double pzc; + double pzf; + double evdc; + double evdf; + double cnoc; + double cnof; + double epc; + double epf; + double rdma_wr; + double rdma_rd; + double rtt; + double close; +} time; + +/* defaults */ +static int parent=1; +static int connected=0; +static int burst=10; +static int server=1; +static int verbose=0; +static int polling=1; +static int poll_count=0; +static int rdma_wr_poll_count=0; +static int rdma_rd_poll_count=0; +static int pin_memory=0; +static int delay=0; +static int buf_len=RDMA_BUFFER_SIZE; +static int use_cno=0; +static int post_recv_count=MSG_BUF_COUNT; +static int recv_msg_index=0; +static int burst_msg_posted=0; +static int burst_msg_index=0; + +#define MAX_RDMA_RD 4 +#define MAX_PROCS 1000 + +static pid_t child[MAX_PROCS+1]; + +/* forward prototypes */ +const char * DT_RetToString (DAT_RETURN ret_value); +const char * DT_EventToSTr (DAT_EVENT_NUMBER event_code); +void print_usage(); +double get_time(); +void init_data(); + +DAT_RETURN send_msg( void *data, + DAT_COUNT size, + DAT_LMR_CONTEXT context, + DAT_DTO_COOKIE cookie, + DAT_COMPLETION_FLAGS flags ); + +DAT_RETURN connect_ep( char *hostname, int conn_id ); +void disconnect_ep( void ); +DAT_RETURN register_rdma_memory( void ); +DAT_RETURN unregister_rdma_memory( void ); +DAT_RETURN create_events( void ); +DAT_RETURN destroy_events(void); +DAT_RETURN do_rdma_write_with_msg( void ); +DAT_RETURN do_rdma_read_with_msg( void ); +DAT_RETURN do_ping_pong_msg( void ); + +#define LOGPRINTF(_format, _aa...) \ + if (verbose) \ + printf(_format, ##_aa) + +main(int argc, char **argv) +{ + int c; + DAT_RETURN ret; + + /* parse arguments */ + while ((c = getopt(argc, argv, "scvpb:d:B:h:")) != -1) + { + switch(c) + { + case 's': + server = 1; + printf("%d Running as server\n",getpid()); + fflush(stdout); + break; + case 'c': + use_cno = 1; + printf("%d Creating CNO for DTO EVD's\n",getpid()); + fflush(stdout); + break; + case 'v': + verbose = 1; + printf("%d Verbose\n",getpid()); + fflush(stdout); + break; + case 'p': + polling = 1; + printf("%d Polling\n",getpid()); + fflush(stdout); + break; + case 'B': + burst = atoi(optarg); + break; + case 'd': + delay = atoi(optarg); + break; + case 'b': + buf_len = atoi(optarg); + break; + case 'h': + server = 0; + strcpy (hostname, optarg); + break; + default: + print_usage(); + exit(-12); + } + } + + if (!server) { + printf("%d Running as client\n",getpid()); fflush(stdout); + } else { + printf("%d Running as server\n",getpid()); fflush(stdout); + } + + /* allocate send and receive buffers */ + if (((rbuf = malloc(buf_len*burst)) == NULL) || + ((sbuf = malloc(buf_len*burst)) == NULL)) { + perror("malloc"); + exit(1); + } + memset( &time, sizeof(time), 0); + LOGPRINTF("%d Allocated RDMA buffers (r:%p,s:%p) len %d \n", + getpid(), rbuf, sbuf, buf_len); + + /* dat_ia_open, dat_pz_create */ + h_async_evd = DAT_HANDLE_NULL; + start = get_time(); + ret = dat_ia_open( DAPL_PROVIDER, 8, &h_async_evd, &h_ia ); + stop = get_time(); + time.open += ((stop - start)*1.0e6); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d: Error Adaptor open: %s\n", + getpid(),DT_RetToString(ret)); + exit(1); + } else + LOGPRINTF("%d Opened Interface Adaptor\n",getpid()); + + /* Create Protection Zone */ + start = get_time(); + LOGPRINTF("%d Create Protection Zone\n",getpid()); + ret = dat_pz_create(h_ia, &h_pz); + stop = get_time(); + time.pzc += ((stop - start)*1.0e6); + if(ret != DAT_SUCCESS) { + fprintf(stderr, + "%d Error creating Protection Zone: %s\n", + getpid(),DT_RetToString(ret)); + exit(1); + } else + LOGPRINTF("%d Created Protection Zone\n",getpid()); + + /* Register memory */ + LOGPRINTF("%d Register RDMA memory\n", getpid()); + ret = register_rdma_memory(); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error creating events: %s\n", + getpid(),DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF("%d Register RDMA memory done\n", getpid()); + + LOGPRINTF("%d Create events\n", getpid()); + ret = create_events(); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error creating events: %s\n", + getpid(),DT_RetToString(ret)); + goto cleanup; + } else { + LOGPRINTF("%d Create events done\n", getpid()); + } + + /* create EP */ + memset( &ep_attr, 0, sizeof(ep_attr) ); + ep_attr.service_type = DAT_SERVICE_TYPE_RC; + ep_attr.max_rdma_size = 0x10000; + ep_attr.qos = 0; + ep_attr.recv_completion_flags = 0; + ep_attr.max_recv_dtos = MSG_BUF_COUNT + (burst*3); + ep_attr.max_request_dtos = MSG_BUF_COUNT + (burst*3) + MAX_RDMA_RD; + ep_attr.max_recv_iov = 1; + ep_attr.max_request_iov = 1; + ep_attr.max_rdma_read_in = MAX_RDMA_RD; + ep_attr.max_rdma_read_out = MAX_RDMA_RD; + ep_attr.request_completion_flags = DAT_COMPLETION_DEFAULT_FLAG; + ep_attr.ep_transport_specific_count = 0; + ep_attr.ep_transport_specific = NULL; + ep_attr.ep_provider_specific_count = 0; + ep_attr.ep_provider_specific = NULL; + + start = get_time(); + ret = dat_ep_create( h_ia, h_pz, h_dto_evd, h_dto_evd, h_conn_evd, &ep_attr, &h_ep ); + stop = get_time(); + time.epc += ((stop - start)*1.0e6); + time.total += time.epc; + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error dat_ep_create: %s\n", + getpid(),DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF("%d EP created %p \n", getpid(), h_ep); + + /* + * register message buffers, establish connection, and + * exchange DMA RMR information info via messages + */ + ret = connect_ep( hostname, SERVER_CONN_QUAL ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error connect_ep: %s\n", + getpid(),DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF("%d connect_ep complete\n", getpid()); + + /*********** RDMA write data *************/ + ret = do_rdma_write_with_msg(); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error do_rdma_write_with_msg: %s\n", + getpid(),DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF("%d do_rdma_write_with_msg complete\n", getpid()); + + /*********** RDMA read data *************/ + ret = do_rdma_read_with_msg(); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error do_rdma_read_with_msg: %s\n", + getpid(),DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF("%d do_rdma_read_with_msg complete\n", getpid()); + + /*********** PING PING messages ************/ + ret = do_ping_pong_msg(); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error do_ping_pong_msg: %s\n", + getpid(),DT_RetToString(ret)); + goto cleanup; + } else + LOGPRINTF("%d do_ping_pong_msg complete\n", getpid()); + +cleanup: + /* disconnect and free EP resources */ + if ( h_ep != DAT_HANDLE_NULL ) { + /* unregister message buffers and tear down connection */ + LOGPRINTF("%d Disconnect and Free EP %p \n",getpid(),h_ep); + disconnect_ep(); + } + + /* free EP */ + LOGPRINTF("%d Free EP %p \n",getpid(),h_ep); + start = get_time(); + ret = dat_ep_free( h_ep ); + stop = get_time(); + time.epf += ((stop - start)*1.0e6); + time.total += time.epf; + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error freeing EP: %s\n", + getpid(), DT_RetToString(ret)); + } else { + LOGPRINTF("%d Freed EP\n",getpid()); + h_ep = DAT_HANDLE_NULL; + } + + /* free EVDs */ + LOGPRINTF("%d destroy events\n", getpid()); + ret = destroy_events(); + if(ret != DAT_SUCCESS) + fprintf(stderr, "%d Error destroy_events: %s\n", + getpid(),DT_RetToString(ret)); + else + LOGPRINTF("%d destroy events done\n", getpid()); + + + ret = unregister_rdma_memory(); + LOGPRINTF("%d unregister_rdma_memory \n", getpid()); + if(ret != DAT_SUCCESS) + fprintf(stderr, "%d Error unregister_rdma_memory: %s\n", + getpid(),DT_RetToString(ret)); + else + LOGPRINTF("%d unregister_rdma_memory done\n", getpid()); + + + if (delay) sleep(delay); + + /* Free protection domain */ + LOGPRINTF("%d Freeing pz\n",getpid()); + start = get_time(); + ret = dat_pz_free( h_pz ); + stop = get_time(); + time.pzf += ((stop - start)*1.0e6); + if (ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error freeing PZ: %s\n", + getpid(), DT_RetToString(ret)); + } else { + LOGPRINTF("%d Freed pz\n",getpid()); + h_pz = NULL; + } + + /* close the device */ + LOGPRINTF("%d Closing Interface Adaptor\n",getpid()); + start = get_time(); + ret = dat_ia_close( h_ia, DAT_CLOSE_ABRUPT_FLAG ); + stop = get_time(); + time.close += ((stop - start)*1.0e6); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d: Error Adaptor close: %s\n", + getpid(),DT_RetToString(ret)); + exit(1); + } else + LOGPRINTF("%d Closed Interface Adaptor\n",getpid()); + + printf( "\n%d: DAPL Test Complete.\n\n",getpid()); + printf( "%d: RDMA write: Total=%10.2lf usec, %d bursts, itime=%10.2lf usec, pc=%d\n", + getpid(), time.rdma_wr, burst, time.rdma_wr/burst, rdma_wr_poll_count ); + printf( "%d: RDMA read: Total=%10.2lf usec, %d bursts, itime=%10.2lf usec, pc=%d\n", + getpid(), time.rdma_rd, MAX_RDMA_RD, time.rdma_rd/MAX_RDMA_RD, rdma_rd_poll_count ); + printf( "%d: Message RTT: Total=%10.2lf usec, %d bursts, itime=%10.2lf usec, pc=%d\n\n", + getpid(), time.rtt, burst, time.rtt/burst, poll_count ); + + printf( "%d: open: %10.2lf usec\n", getpid(), time.open ); + printf( "%d: close: %10.2lf usec\n", getpid(), time.close ); + printf( "%d: PZ create: %10.2lf usec\n", getpid(), time.pzc ); + printf( "%d: PZ free: %10.2lf usec\n", getpid(), time.pzf ); + printf( "%d: LMR create:%10.2lf usec\n", getpid(), time.reg ); + printf( "%d: LMR free: %10.2lf usec\n", getpid(), time.unreg ); + printf( "%d: EVD create:%10.2lf usec\n", getpid(), time.evdc ); + printf( "%d: EVD free: %10.2lf usec\n", getpid(), time.evdf ); + if (use_cno) { + printf( "%d: CNO create: %10.2lf usec\n", getpid(), time.cnoc ); + printf( "%d: CNO free: %10.2lf usec\n", getpid(), time.cnof ); + } + printf( "%d: EP create: %10.2lf usec\n",getpid(), time.epc ); + printf( "%d: EP free: %10.2lf usec\n",getpid(), time.epf ); + printf( "%d: TOTAL: %10.2lf usec\n",getpid(), time.total ); + + /* free rdma buffers */ + free(rbuf); + free(sbuf); +} + + +double get_time() +{ + struct timeval tp; + + gettimeofday(&tp, NULL); + return ((double) tp.tv_sec + (double) tp.tv_usec * 1e-6); +} + +void init_data() +{ + memset(rbuf, 'a', buf_len); + memset(sbuf, 'b', buf_len); +} + + +DAT_RETURN +send_msg( void *data, + DAT_COUNT size, + DAT_LMR_CONTEXT context, + DAT_DTO_COOKIE cookie, + DAT_COMPLETION_FLAGS flags ) +{ + DAT_LMR_TRIPLET iov; + DAT_EVENT event; + DAT_COUNT nmore; + DAT_RETURN ret; + + iov.lmr_context = context; + iov.pad = 0; + iov.virtual_address = (DAT_VADDR)(unsigned long)data; + iov.segment_length = size; + LOGPRINTF("%d calling post_send\n", getpid()); + cookie.as_64 = 0xaaaa; + ret = dat_ep_post_send( h_ep, + 1, + &iov, + cookie, + flags ); + + if (ret != DAT_SUCCESS) { + fprintf(stderr, "%d: ERROR: dat_ep_post_send() %s\n", + getpid(),DT_RetToString(ret)); + return ret; + } + + if (!(flags & DAT_COMPLETION_SUPPRESS_FLAG)) { + if ( polling ) { + printf("%d Polling post send completion...\n",getpid()); + while ( dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY ); + } + else { + LOGPRINTF("%d waiting for post_send completion event\n", getpid()); + if (use_cno) { + DAT_EVD_HANDLE evd = DAT_HANDLE_NULL; + ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd ); + LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd); + if ( evd != h_dto_evd ) { + fprintf(stderr, + "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", + getpid()); + return( DAT_ABORT ); + } + } + /* use wait to dequeue */ + ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore ); + if (ret != DAT_SUCCESS) { + fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", + getpid(),DT_RetToString(ret)); + return ret; + } + } + + /* validate event number, len, cookie, and status */ + if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) { + fprintf(stderr, "%d: ERROR: DTO event number %s\n", + getpid(),DT_EventToSTr(event.event_number)); + return( DAT_ABORT ); + } + + if ((event.event_data.dto_completion_event_data.transfered_length != size ) || + (event.event_data.dto_completion_event_data.user_cookie.as_64 != 0xaaaa )) { + fprintf(stderr, "%d: ERROR: DTO len %d or cookie %x\n", + getpid(), + event.event_data.dto_completion_event_data.transfered_length, + event.event_data.dto_completion_event_data.user_cookie.as_64 ); + return( DAT_ABORT ); + + } + if (event.event_data.dto_completion_event_data.status != DAT_SUCCESS) { + fprintf(stderr, "%d: ERROR: DTO event status %s\n", + getpid(),DT_RetToString(ret)); + return( DAT_ABORT ); + } + } + + return DAT_SUCCESS; +} + + +DAT_RETURN +connect_ep( char *hostname, int conn_id ) +{ + DAT_SOCK_ADDR remote_addr; + DAT_EP_ATTR ep_attr; + DAT_RETURN ret; + DAT_REGION_DESCRIPTION region; + DAT_EVENT event; + DAT_COUNT nmore; + DAT_LMR_TRIPLET l_iov; + DAT_RMR_TRIPLET r_iov; + DAT_DTO_COOKIE cookie; + int i; + + /* Register send message buffer */ + LOGPRINTF("%d Registering send Message Buffer %p, len %d\n", + getpid(), &rmr_send_msg, sizeof(DAT_RMR_TRIPLET) ); + region.for_va = &rmr_send_msg; + ret = dat_lmr_create( h_ia, + DAT_MEM_TYPE_VIRTUAL, + region, + sizeof(DAT_RMR_TRIPLET), + h_pz, + DAT_MEM_PRIV_LOCAL_WRITE_FLAG, + &h_lmr_send_msg, + &lmr_context_send_msg, + &rmr_context_send_msg, + ®istered_size_send_msg, + ®istered_addr_send_msg ); + + if (ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error registering send msg buffer: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d Registered send Message Buffer %p \n", + getpid(),region.for_va ); + + /* Register Receive buffers */ + LOGPRINTF("%d Registering Receive Message Buffer %p\n", + getpid(), rmr_recv_msg ); + region.for_va = rmr_recv_msg; + ret = dat_lmr_create( h_ia, + DAT_MEM_TYPE_VIRTUAL, + region, + sizeof(DAT_RMR_TRIPLET)*MSG_BUF_COUNT, + h_pz, + DAT_MEM_PRIV_LOCAL_WRITE_FLAG, + &h_lmr_recv_msg, + &lmr_context_recv_msg, + &rmr_context_recv_msg, + ®istered_size_recv_msg, + ®istered_addr_recv_msg ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error registering recv msg buffer: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d Registered Receive Message Buffer %p\n", + getpid(),region.for_va); + + for ( i = 0; i < MSG_BUF_COUNT; i++ ) { + cookie.as_64 = i; + l_iov.lmr_context = lmr_context_recv_msg; + l_iov.pad = 0; + l_iov.virtual_address = (DAT_VADDR)(unsigned long)&rmr_recv_msg[ i ]; + l_iov.segment_length = sizeof(DAT_RMR_TRIPLET); + + LOGPRINTF("%d Posting Receive Message Buffer %p\n", + getpid(), &rmr_recv_msg[ i ]); + ret = dat_ep_post_recv( h_ep, + 1, + &l_iov, + cookie, + DAT_COMPLETION_DEFAULT_FLAG ); + + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error registering recv msg buffer: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d Registered Receive Message Buffer %p\n", + getpid(),region.for_va); + + } + + /* setup receive rdma buffer to initial string to be overwritten */ + strcpy( (char*)rbuf, "blah, blah, blah\n" ); + + if ( server ) { /* SERVER */ + + /* create the service point for server listen */ + LOGPRINTF("%d Creating service point for listen\n",getpid()); + ret = dat_psp_create( h_ia, + conn_id, + h_cr_evd, + DAT_PSP_CONSUMER_FLAG, + &h_psp ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error dat_psp_create: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d dat_psp_created for server listen\n", getpid()); + + printf("%d Server waiting for connect request..\n", getpid()); + ret = dat_evd_wait( h_cr_evd, SERVER_TIMEOUT, 1, &event, &nmore ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error dat_evd_wait: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d dat_evd_wait for cr_evd completed\n", getpid()); + + if ( event.event_number != DAT_CONNECTION_REQUEST_EVENT ) { + fprintf(stderr, "%d Error unexpected cr event : %s\n", + getpid(),DT_EventToSTr(event.event_number)); + return( DAT_ABORT ); + } + if ( (event.event_data.cr_arrival_event_data.conn_qual != SERVER_CONN_QUAL) || + (event.event_data.cr_arrival_event_data.sp_handle.psp_handle != h_psp) ) { + fprintf(stderr, "%d Error wrong cr event data : %s\n", + getpid(),DT_EventToSTr(event.event_number)); + return( DAT_ABORT ); + } + + /* accept connect request from client */ + h_cr = event.event_data.cr_arrival_event_data.cr_handle; + LOGPRINTF("%d Accepting connect request from client\n",getpid()); + ret = dat_cr_accept( h_cr, h_ep, 0, (DAT_PVOID)0 ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error dat_cr_accept: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d dat_cr_accept completed\n", getpid()); + } + else { /* CLIENT */ + struct addrinfo *target; + int rval; + + if (getaddrinfo (hostname, NULL, NULL, &target) != 0) { + printf("\n remote name resolution failed!\n"); + exit ( 1 ); + } + + rval = ((struct sockaddr_in *)target->ai_addr)->sin_addr.s_addr; + printf ("%d Server Name: %s \n", getpid(), hostname); + printf ("%d Server Net Address: %d.%d.%d.%d\n", getpid(), + (rval >> 0) & 0xff, + (rval >> 8) & 0xff, + (rval >> 16) & 0xff, + (rval >> 24) & 0xff); + + remote_addr = *((DAT_IA_ADDRESS_PTR)target->ai_addr); + + LOGPRINTF("%d Connecting to server\n",getpid()); + ret = dat_ep_connect( h_ep, + &remote_addr, + conn_id, + CONN_TIMEOUT, + 0, + (DAT_PVOID)0, + 0, + DAT_CONNECT_DEFAULT_FLAG ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error dat_ep_connect: %s\n", + getpid(), DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d dat_ep_connect completed\n", getpid()); + } + + printf("%d Waiting for connect response\n",getpid()); + + ret = dat_evd_wait( h_conn_evd, DAT_TIMEOUT_INFINITE, 1, &event, &nmore ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error dat_evd_wait: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d dat_evd_wait for h_conn_evd completed\n", getpid()); + + if ( event.event_number != DAT_CONNECTION_EVENT_ESTABLISHED ) { + fprintf(stderr, "%d Error unexpected conn event : %s\n", + getpid(),DT_EventToSTr(event.event_number)); + return( DAT_ABORT ); + } + printf("\n%d CONNECTED!\n\n",getpid()); + connected = 1; + + /* + * Setup our remote memory and tell the other side about it + */ + rmr_send_msg.rmr_context = rmr_context_recv; + rmr_send_msg.pad = 0; + rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf; + rmr_send_msg.segment_length = RDMA_BUFFER_SIZE; + + printf("%d Send RMR to remote: snd_msg: r_key_ctx=%x,pad=%x,va=%llx,len=0x%x\n", + getpid(), rmr_send_msg.rmr_context, rmr_send_msg.pad, + rmr_send_msg.target_address, rmr_send_msg.segment_length ); + + ret = send_msg( &rmr_send_msg, + sizeof( DAT_RMR_TRIPLET ), + lmr_context_send_msg, + cookie, + DAT_COMPLETION_SUPPRESS_FLAG ); + + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error send_msg: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else + LOGPRINTF("%d send_msg completed\n", getpid()); + + /* + * Wait for remote RMR information for RDMA + */ + if ( polling ) { + printf("%d Polling for remote to send RMR data\n",getpid()); + while ( dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY ); + } + else { + printf("%d Waiting for remote to send RMR data\n",getpid()); + if (use_cno) + { + DAT_EVD_HANDLE evd = DAT_HANDLE_NULL; + ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd ); + LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd); + if ( evd != h_dto_evd ) { + fprintf(stderr, + "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", + getpid()); + return( DAT_ABORT ); + } + } + /* use wait to dequeue */ + ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error waiting on h_dto_evd: %s\n", + getpid(),DT_RetToString(ret)); + return(ret); + } + else { + LOGPRINTF("%d dat_evd_wait h_dto_evd completed\n", getpid()); + } + } + + printf("%d remote RMR data arrived!\n",getpid()); + + if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) { + fprintf(stderr, "%d Error unexpected DTO event : %s\n", + getpid(),DT_EventToSTr(event.event_number)); + return( DAT_ABORT ); + } + if ((event.event_data.dto_completion_event_data.transfered_length != + sizeof( DAT_RMR_TRIPLET )) || + (event.event_data.dto_completion_event_data.user_cookie.as_64 != + recv_msg_index) ) { + fprintf(stderr,"ERR recv event: len=%d cookie=%d expected %d/%d\n", + (int)event.event_data.dto_completion_event_data.transfered_length, + (int)event.event_data.dto_completion_event_data.user_cookie.as_64, + sizeof(DAT_RMR_TRIPLET), recv_msg_index ); + return( DAT_ABORT ); + } + + r_iov = rmr_recv_msg[ recv_msg_index ]; + + printf("%d Received RMR from remote: r_iov: r_key_ctx=%x,pad=%x,va=%llx,len=0x%x\n", + getpid(), r_iov.rmr_context, r_iov.pad, + r_iov.target_address, r_iov.segment_length ); + + recv_msg_index++; + + return ( DAT_SUCCESS ); +} + + +void +disconnect_ep() +{ + DAT_RETURN ret; + DAT_EVENT event; + DAT_COUNT nmore; + int i,flush_cnt; + + if (connected) { + + LOGPRINTF("%d dat_ep_disconnect\n", getpid()); + ret = dat_ep_disconnect( h_ep, DAT_CLOSE_DEFAULT ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error dat_ep_disconnect: %s\n", + getpid(),DT_RetToString(ret)); + } + else { + LOGPRINTF("%d dat_ep_disconnect completed\n", getpid()); + } + } + + /* destroy service point */ + if (( server ) && ( h_psp != DAT_HANDLE_NULL )) { + ret = dat_psp_free( h_psp ); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error dat_psp_free: %s\n", + getpid(),DT_RetToString(ret)); + } + else { + LOGPRINTF("%d dat_psp_free completed\n", getpid()); + } + } + + /* Unregister Send message Buffer */ + if ( h_lmr_send_msg != DAT_HANDLE_NULL ) { + LOGPRINTF("%d Unregister send message h_lmr %p \n",getpid(),h_lmr_send_msg); + ret = dat_lmr_free(h_lmr_send_msg); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error deregistering send msg mr: %s\n", + getpid(), DT_RetToString(ret)); + } else { + LOGPRINTF("%d Unregistered send message Buffer\n",getpid()); + h_lmr_send_msg = NULL; + } + } + + /* Unregister recv message Buffer */ + if ( h_lmr_recv_msg != DAT_HANDLE_NULL ) { + LOGPRINTF("%d Unregister recv message h_lmr %p \n",getpid(),h_lmr_recv_msg); + ret = dat_lmr_free(h_lmr_recv_msg); + if(ret != DAT_SUCCESS) { + fprintf(stderr, "%d Error deregistering recv msg mr: %s\n", + getpid(), DT_RetToString(ret)); + } else { + LOGPRINTF("%d Unregistered recv message Buffer\n",getpid()); + h_lmr_recv_msg = NULL; + } + } + return; +} + + +DAT_RETURN +do_rdma_write_with_msg( ) +{ + DAT_REGION_DESCRIPTION region; + DAT_EVENT event; + DAT_COUNT nmore; + DAT_LMR_TRIPLET l_iov; + DAT_RMR_TRIPLET r_iov; + DAT_DTO_COOKIE cookie; + DAT_RMR_CONTEXT their_context; + DAT_RETURN ret; + int i; + + printf("\n %d RDMA WRITE DATA with SEND MSG\n\n",getpid()); + + cookie.as_64 = 0x5555; + + if ( recv_msg_index >= MSG_BUF_COUNT ) + return( DAT_ABORT ); + + /* get RMR information from previously received message */ + r_iov = rmr_recv_msg[ recv_msg_index-1 ]; + + if ( server ) + strcpy( (char*)sbuf, "server written data..." ); + else + strcpy( (char*)sbuf, "client written data..." ); + + l_iov.lmr_context = lmr_context_send; + l_iov.pad = 0; + l_iov.virtual_address = (DAT_VADDR)(unsigned long)sbuf; + l_iov.segment_length = buf_len; + + start = get_time(); + for (i=0;i= MSG_BUF_COUNT ) + return( DAT_ABORT ); + + /* get RMR information from previously received message */ + r_iov = rmr_recv_msg[ recv_msg_index-1 ]; + + /* setup rdma read buffer to initial string to be overwritten */ + strcpy( (char*)sbuf, "blah, blah, blah\n" ); + + if ( server ) + strcpy( (char*)rbuf, "server read data..." ); + else + strcpy( (char*)rbuf, "client read data..." ); + + l_iov.lmr_context = lmr_context_send; + l_iov.pad = 0; + l_iov.virtual_address = (DAT_VADDR)(unsigned long)sbuf; + l_iov.segment_length = buf_len; + + start = get_time(); + for (i=0;i \ +# +# +# Example for openib using the first Mellanox adapter, port 1 and port 2 + +IB1 u1.2 nonthreadsafe default /home/ardavis/Proj/openib/gen2/users/jlentini/userspace/dapl/udapl/Target/libdapl.so r.1.2 "mthca0 1" "" Index: dtest/README =================================================================== --- dtest/README (revision 0) +++ dtest/README (revision 0) @@ -0,0 +1,19 @@ +simple dapl test just for initial openIB uDAPL testing... + + dtest/dtest.c + dtest/makefile + dtest/dat.conf + +to build (default uDAPL name == IB1, ib device == mthca0, port == 1) + edit makefile and change path (DAT_LIB) to appropriate libdat.so + edit dat.conf and change path to appropriate libdapl.so + cp dat.conf to /etc/dat.conf + +to run: + server: dtest + client: dtest -h hostname + +for verbose uDAPL and uDAT debug: + + export DAPL_DBG_TYPE=0xffff + export DAT_DBG_TYPE=0xffff Index: dtest/makefile =================================================================== --- dtest/makefile (revision 0) +++ dtest/makefile (revision 0) @@ -0,0 +1,16 @@ +CC = gcc +CFLAGS = -O2 -g + +DAT_INC = ../dat/include +DAT_LIB = /usr/lib64 + +all: dtest + +clean: + rm -f *.o;touch *.c;rm -f dtest + +dtest: ./dtest.c + $(CC) $(CFLAGS) ./dtest.c -o dtest \ + -DDAPL_PROVIDER='"IB1"' \ + -I $(DAT_INC) -L $(DAT_LIB) -ldat + From libor at topspin.com Mon Jun 27 16:06:11 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 27 Jun 2005 16:06:11 -0700 Subject: [openib-general] Re: SDP: still getting sk_alloc() panic, any ideas? In-Reply-To: <1119907675.6148.7.camel@duffman>; from tduffy@sun.com on Mon, Jun 27, 2005 at 02:27:54PM -0700 References: <9d3b7de7050623130658aa92@mail.gmail.com> <20050627111721.A26240@topspin.com> <1119907675.6148.7.camel@duffman> Message-ID: <20050627160611.C26240@topspin.com> On Mon, Jun 27, 2005 at 02:27:54PM -0700, Tom Duffy wrote: > On Mon, 2005-06-27 at 11:17 -0700, Libor Michalek wrote: > > The problem is that each call to sk_alloc() is grabbing a reference to > > the module, but it checks to make sure that there already is at least one > > reference, if not the top BUG is triggered. In the case of the passive > > connection there are no other references to the module. You can see that > > the problem goes away if you open just one socket, even if you don't > > listen on it, and then try the failing passive connect. When a socket is > > created it actually grabs two references to the module, one at the sock > > level and one at the sk level. The first reference at the sock level does > > not trigger the BUG since it's through another code path. (try_module_get > > vs. __module_get) This is why we only hit this during passive connect > > to a system that has no active SDP sockets. > > > > Not sure the right way to fix this, maybe check to see if the socket > > table size (dev_root_s.sk_entry) is greater then 0 in sdp_cm_req_handler() > > before even performing the alloc... > > Hrm. That seems ugly. How about a patch to upstream changing > sk_alloc() to use try_module_get(). I'm thinking the listen_lookup needs to be moved earlier in the req_handler ahead of the sk_alloc, since it makes no sense to do the alloc if we are not going to queue the new incomming connection, since it just leads to a destroy in the same function. -Libor From xma at us.ibm.com Mon Jun 27 16:20:59 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 27 Jun 2005 16:20:59 -0700 Subject: [openib-general] ibv_open_device() broken in r2720? Message-ID: I am running netpipe test for ibv, the error is Couldn't create InfiniBand context, seems like ibv_open_device() return NULL for r2720. Is it broken, or I did something wrong? Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rolandd at cisco.com Mon Jun 27 16:26:29 2005 From: rolandd at cisco.com (Roland Dreier) Date: Mon, 27 Jun 2005 16:26:29 -0700 Subject: [openib-general] ibv_open_device() broken in r2720? In-Reply-To: (Shirley Ma's message of "Mon, 27 Jun 2005 16:20:59 -0700") References: Message-ID: <52wtofz8ii.fsf@topspin.com> Shirley> I am running netpipe test for ibv, the error is Couldn't Shirley> create InfiniBand context, seems like ibv_open_device() Shirley> return NULL for r2720. Is it broken, or I did something Shirley> wrong? Are you using udev, or creating /dev/ entries by hand? If you are not using udev then you need to recreate the /dev/infiniband/uverbsN files, since the base minor has moved from 128 or 192. If that's not it, please post strace output for your program. - R. From xma at us.ibm.com Mon Jun 27 16:29:50 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 27 Jun 2005 16:29:50 -0700 Subject: [openib-general] ibv_open_device() broken in r2720? In-Reply-To: <52wtofz8ii.fsf@topspin.com> Message-ID: >since the base minor has moved from 128 to 192. That could be the reason. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rolandd at cisco.com Mon Jun 27 16:59:54 2005 From: rolandd at cisco.com (Roland Dreier) Date: Mon, 27 Jun 2005 16:59:54 -0700 Subject: [openib-general] [PATCH] process locked in D state. In-Reply-To: <20050627082745.GJ15312@minantech.com> (Gleb Natapov's message of "Mon, 27 Jun 2005 11:27:45 +0300") References: <20050627082745.GJ15312@minantech.com> Message-ID: <52fyv3z6yt.fsf@topspin.com> Something like this should work... - R. --- infiniband/core/uverbs_mem.c (revision 2710) +++ infiniband/core/uverbs_mem.c (working copy) @@ -37,6 +37,13 @@ #include "uverbs.h" +struct ib_umem_account_work { + struct work_struct work; + struct mm_struct *mm; + unsigned long diff; +}; + + static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty) { struct ib_umem_chunk *chunk, *tmp; @@ -160,21 +167,53 @@ out: return ret; } -void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) +static void ib_umem_account(void *work_ptr) { - struct mm_struct *mm; + struct ib_umem_account_work *work = work_ptr; - mm = get_task_mm(current); + down_write(&work->mm->mmap_sem); + work->mm->locked_vm -= work->diff; + up_write(&work->mm->mmap_sem); + mmput(work->mm); + kfree(work); +} - if (mm) { - down_write(&mm->mmap_sem); - mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; - } +void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) +{ + struct mm_struct *mm; __ib_umem_release(dev, umem, 1); + mm = get_task_mm(current); if (mm) { - up_write(&mm->mmap_sem); - mmput(mm); + /* + * We may be called with the mm's mmap_sem already + * held. This can happen when a userspace munmap() is + * the call that drops the last reference to our file + * and calls our release method. If there are memory + * regions to destroy, we'll end up here and not be + * able to take the mmap_sem. + * + * To handle this, we try to grab the mmap_sem, and if + * we can't get it immediately, we defer the + * accounting to the system workqueue. + */ + if (down_write_trylock(&mm->mmap_sem)) { + mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; + up_write(&mm->mmap_sem); + mmput(mm); + } else { + struct ib_umem_account_work *work; + + work = kmalloc(sizeof *work, GFP_KERNEL); + if (!work) + return; + + INIT_WORK(&work->work, ib_umem_account, work); + work->mm = mm; + work->diff = PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; + + schedule_work(&work->work); + } } } From sean.hefty at intel.com Mon Jun 27 17:05:39 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 27 Jun 2005 17:05:39 -0700 Subject: [openib-general] [PATCH] updates to copyright Message-ID: The following patch adds copyright statements for Mellanox, Sun, and Intel. I'm not the usual maintainer for all of the files listed, so I wanted to verify that there are no issues applying this. Note that for Mellanox I copied an existing copyright statement, rather than using Michael's patch. (Tom, this should include your patch.) Signed-off-by: Sean Hefty Index: include/ib_cm.h =================================================================== --- include/ib_cm.h (revision 2723) +++ include/ib_cm.h (working copy) @@ -2,6 +2,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: include/ib_verbs.h =================================================================== --- include/ib_verbs.h (revision 2723) +++ include/ib_verbs.h (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: include/ib_cache.h =================================================================== --- include/ib_cache.h (revision 2723) +++ include/ib_cache.h (working copy) @@ -1,5 +1,7 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Intel Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: include/ib_fmr_pool.h =================================================================== --- include/ib_fmr_pool.h (revision 2723) +++ include/ib_fmr_pool.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/agent.c =================================================================== --- core/agent.c (revision 2723) +++ core/agent.c (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004, 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004, 2005 Topspin Corporation. All rights reserved. * Copyright (c) 2004, 2005 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/device.c =================================================================== --- core/device.c (revision 2723) +++ core/device.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/user_mad.c =================================================================== --- core/user_mad.c (revision 2723) +++ core/user_mad.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. * Copyright (c) 2005 Voltaire, Inc. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/cm.c =================================================================== --- core/cm.c (revision 2723) +++ core/cm.c (working copy) @@ -2,6 +2,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/mad.c =================================================================== --- core/mad.c (revision 2723) +++ core/mad.c (working copy) @@ -1,5 +1,7 @@ /* * Copyright (c) 2004, 2005 Voltaire, Inc. All rights reserved. + * Copyright (c) 2005 Intel Corporation. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/cache.c =================================================================== --- core/cache.c (revision 2723) +++ core/cache.c (working copy) @@ -1,5 +1,7 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Intel Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/packer.c =================================================================== --- core/packer.c (revision 2723) +++ core/packer.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/mad_priv.h =================================================================== --- core/mad_priv.h (revision 2723) +++ core/mad_priv.h (working copy) @@ -1,5 +1,7 @@ /* * Copyright (c) 2004, 2005, Voltaire, Inc. All rights reserved. + * Copyright (c) 2005 Intel Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/sysfs.c =================================================================== --- core/sysfs.c (revision 2723) +++ core/sysfs.c (working copy) @@ -1,5 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/fmr_pool.c =================================================================== --- core/fmr_pool.c (revision 2723) +++ core/fmr_pool.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/ud_header.c =================================================================== --- core/ud_header.c (revision 2723) +++ core/ud_header.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/verbs.c =================================================================== --- core/verbs.c (revision 2723) +++ core/verbs.c (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/smi.c =================================================================== --- core/smi.c (revision 2723) +++ core/smi.c (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004, 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004, 2005 Topspin Corporation. All rights reserved. * Copyright (c) 2004, 2005 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU Index: core/ping.c =================================================================== --- core/ping.c (revision 2723) +++ core/ping.c (working copy) @@ -4,6 +4,7 @@ * Copyright (c) 2004, 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004, 2005 Topspin Corporation. All rights reserved. * Copyright (c) 2004, 2005 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU From jcarr at linuxmachines.com Mon Jun 27 17:36:21 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 27 Jun 2005 17:36:21 -0700 Subject: [openib-general] Re: A new simple ulp (SPTS) In-Reply-To: <20050627203245.GB26909@mellanox.co.il> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <42C060DB.4040409@linuxmachines.com> <20050627203245.GB26909@mellanox.co.il> Message-ID: <42C09B85.4040904@linuxmachines.com> On 06/27/05 13:32, Michael S. Tsirkin wrote: > Quoting r. Jeff Carr : > >>Here is an updated version and a simple perl script that tests it's >>performance. With 2K messages, these were the performance numbers >>between 2 systems (3.6ghz Xeon/w 133mhz/64bit pci). >> >>root at delta:~# ./fast_test.pl 20 >>starting sends >>0 messages/sec (0 Mb/sec) >>131072 messages/sec (2047 Mb/sec) >>131072 messages/sec (2047 Mb/sec) >>131072 messages/sec (2047 Mb/sec) >>174762 messages/sec (2730 Mb/sec) >>163840 messages/sec (2559 Mb/sec) >>196608 messages/sec (3071 Mb/sec) >>183500 messages/sec (2867 Mb/sec) >>174762 messages/sec (2730 Mb/sec) >>196618 messages/sec (3072 Mb/sec) >>187254 messages/sec (2925 Mb/sec) >>180232 messages/sec (2816 Mb/sec) >>196616 messages/sec (3072 Mb/sec) >>189333 messages/sec (2958 Mb/sec) >>183507 messages/sec (2867 Mb/sec) >>196614 messages/sec (3072 Mb/sec) >>190656 messages/sec (2978 Mb/sec) >>202571 messages/sec (3165 Mb/sec) >>138785 messages/sec (2168 Mb/sec) >>131075 messages/sec (2048 Mb/sec) >> >>Jeff >> > > > Does this mean the bandwidth is 200-300 MByte/sec? Well, 3000 Mb/sec / 8 == 375 MB/sec :) yes. It's not as fast as I would hope. I'm not sure what the cause is yet. I think it's more to do with the small 2K message size. If I run this test at 16/64KB message size it's likely to have better performance. The PCI bus has theoretical throughput around 700MB/sec. > What is the CPU utilization number (easy to sample with e.g. vstat)? The CPU is maxed out in this code because it's set to spin while polling. Jeff From jcarr at linuxmachines.com Mon Jun 27 17:58:33 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 27 Jun 2005 17:58:33 -0700 Subject: [openib-general] Re: A new simple ulp (SPTS) In-Reply-To: <20050627203245.GB26909@mellanox.co.il> References: <42B86EEE.4020804@linuxmachines.com> <527jgnv6ac.fsf@topspin.com> <42B99546.2070703@linuxmachines.com> <42C060DB.4040409@linuxmachines.com> <20050627203245.GB26909@mellanox.co.il> Message-ID: <42C0A0B9.5090307@linuxmachines.com> On 06/27/05 13:32, Michael S. Tsirkin wrote: > Does this mean the bandwidth is 200-300 MByte/sec? root at delta:~# ./fast_test.pl 10 starting sends 0 messages/sec (0 megabits/sec) 21845 messages/sec (2730 megabits/sec) 26214 messages/sec (3276 megabits/sec) 30247 messages/sec (3780 megabits/sec) 32768 messages/sec (4095 megabits/sec) 32768 messages/sec (4095 megabits/sec) 34192 messages/sec (4274 megabits/sec) 35288 messages/sec (4411 megabits/sec) 36157 messages/sec (4519 megabits/sec) 35746 messages/sec (4468 megabits/sec) These are the results with 16KB messages; so the other speed problem is likely due to the smaller 2KB message size or deficiencies in my code. At any rate, this is good enough for what I need it for. Jeff From eitan at mellanox.co.il Mon Jun 27 22:29:25 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Tue, 28 Jun 2005 08:29:25 +0300 Subject: [openib-general] OpenSM on multiple HCA machine Message-ID: <506C3D7B14CDD411A52C00025558DED607C303CB@mtlex01.yok.mtl.com> I was not aware of an issue with multiple HCAs in the OpenIB (gen2) stack. The Gen1 stack had this issue and it was resolved. I hope to be able to focus on OpenIB stack in the coming months such that I can help Hal in fixing these kind of issues too. Eitan Zahavi > -----Original Message----- > From: Bernhard Fischer [mailto:rep.nop at aon.at] > Sent: Tuesday, June 28, 2005 12:14 AM > To: Hal Rosenstock > Cc: Eitan Zahavi; openib-windows at openib.org; 'openib-general at openib.org' > Subject: Re: [openib-general] Re: IB Diagnositic Tools > > On Mon, Jun 27, 2005 at 02:04:16PM -0400, Hal Rosenstock wrote: > >Hi Eitan, > > > >On Sat, 2005-06-25 at 15:25, Eitan Zahavi wrote: > >> Following the discussion about the debug tools, I would like to > >> propose using OpenSM Vendor layer as a common layer for developing the > >> debug tools. > > Hal, > > This is kinda offtopic, but (iirc) i once stumbled over the issue of > "port" vs. "mgmt port" [back then i had access to two 2-port cards] > where you may have said something along the lines of > \"There is clearly a bug for multi HCAs in osm_vendor_get_all_port_attr > which is in the vendor layer. This needs to be fixed and is our problem. > So I am close to being able to commit what I now have for this and fix > this later (as there are other multi HCA issues).\" > Just curious.. did somebody already have a chance to touch those or not? > > On a related note (just the same thing, i tend to think) > /* "local ports" is(?) phys, shouldn't this exclude port 0 then ? */ > > I'm not too familiar with these kindof questions, which might be define > in a spec, so any hint on this would be well received, at least from my > part. > > anyone? Eitan? > -- > thank you, > Bernhard -------------- next part -------------- An HTML attachment was scrubbed... URL: From gleb at minantech.com Mon Jun 27 23:43:10 2005 From: gleb at minantech.com (Gleb Natapov) Date: Tue, 28 Jun 2005 09:43:10 +0300 Subject: [openib-general] [PATCH] process locked in D state. In-Reply-To: <523br3938y.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <523br3938y.fsf@topspin.com> Message-ID: <20050628064310.GK15312@minantech.com> On Mon, Jun 27, 2005 at 09:24:45AM -0700, Roland Dreier wrote: > Gleb> This is what happens: ibv_close_device() close cmd_fd and > Gleb> then calls free_context(). free_context() calls munmap to > Gleb> unmap doorbell registers. In kernel sys_munmap gets > Gleb> mm->mmap_sem semaphore and calls do_munmap. do_munmap is > Gleb> the last user of the file so it calls release method of the > Gleb> file (ib_uverbs_close() in our case). ib_uverbs_close() > Gleb> calls ib_dealloc_ucontext(). ib_dealloc_ucontext() notices > Gleb> that there is unregistered memory on the file and calls > Gleb> ib_umem_release(). And there we are trying to acquire > Gleb> mm->mmap_sem on more time. > > Thanks for the good debugging work. > > Gleb> In attached patch I use down_write_trylock() instead of > Gleb> down_write() in ib_umem_release(). If semaphore is already > Gleb> locked we will not update locked_vm statistics. This way > Gleb> malicious user can only cause harm to itself. > > I don't like this solution -- as you point out, down_write_trylock() > may fail if there is even momentary contention on the mmap_sem. So > for example a different malicious user could poll on /proc//maps > and cause our locked_vm to continue to grow. > You are right. For a moment I forgot that /proc//maps readable by the world. > How about if we use schedule_work() to defer the modification of > locked_vm? It seems this is the only sane way to do it. -- Gleb. From glebn at voltaire.com Tue Jun 28 00:04:41 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Tue, 28 Jun 2005 10:04:41 +0300 Subject: [openib-general] [PATCH] process locked in D state. In-Reply-To: <52fyv3z6yt.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> Message-ID: <20050628070441.GL15312@minantech.com> On Mon, Jun 27, 2005 at 04:59:54PM -0700, Roland Dreier wrote: > Something like this should work... Yes, but don't forget the first part of my patch (for libibverbs/src/device.c) I think it should be applied to. [snip] > > -void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) > +static void ib_umem_account(void *work_ptr) > { > - struct mm_struct *mm; > + struct ib_umem_account_work *work = work_ptr; > > - mm = get_task_mm(current); Are you sure that ib_umem_account will run on behalf the process that schedules it? Anyway you are correctly using work->mm in the rest of the function so this line should be dropped I think. > + down_write(&work->mm->mmap_sem); > + work->mm->locked_vm -= work->diff; > + up_write(&work->mm->mmap_sem); > + mmput(work->mm); > + kfree(work); > +} > > - if (mm) { > - down_write(&mm->mmap_sem); > - mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; > - } > +void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) > +{ > + struct mm_struct *mm; > > __ib_umem_release(dev, umem, 1); > > + mm = get_task_mm(current); > if (mm) { > - up_write(&mm->mmap_sem); > - mmput(mm); > + /* > + * We may be called with the mm's mmap_sem already > + * held. This can happen when a userspace munmap() is > + * the call that drops the last reference to our file > + * and calls our release method. If there are memory > + * regions to destroy, we'll end up here and not be > + * able to take the mmap_sem. > + * > + * To handle this, we try to grab the mmap_sem, and if > + * we can't get it immediately, we defer the > + * accounting to the system workqueue. > + */ > + if (down_write_trylock(&mm->mmap_sem)) { > + mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; > + up_write(&mm->mmap_sem); > + mmput(mm); > + } else { > + struct ib_umem_account_work *work; > + > + work = kmalloc(sizeof *work, GFP_KERNEL); > + if (!work) > + return; > + > + INIT_WORK(&work->work, ib_umem_account, work); > + work->mm = mm; > + work->diff = PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; > + > + schedule_work(&work->work); > + } > } > } -- Gleb. From arne.redlich at xiranet.com Tue Jun 28 00:15:21 2005 From: arne.redlich at xiranet.com (Arne Redlich) Date: Tue, 28 Jun 2005 09:15:21 +0200 Subject: [openib-general] SDP sk_data_ready() callback In-Reply-To: <20050627113521.B26240@topspin.com> References: <1119429583.8208.28.camel@confield> <20050627113521.B26240@topspin.com> Message-ID: <1119942921.8348.21.camel@confield> Am Montag, den 27.06.2005, 11:35 -0700 schrieb Libor Michalek: > On Wed, Jun 22, 2005 at 10:39:43AM +0200, Arne Redlich wrote: > > Hi, > > > > I'm trying to use SDP from within the kernel. My problem is that the > > code relies on sk_data_ready() (this callback is modified to wake up a > > Rx thread before executing the original function), but sk_data_ready() > > apparently never gets called. Is there any way to fix this? > > You mean that you replace sk->sk_data_ready, with a similar but > slightly modified version and so rely on sk->sk_data_ready() being > called? Yes, that's right. Actually the modifications are just minor as I already pointed out: the replacement function just wakes the thread and then calls the original sk_data_ready(). > You're right it's currently not being called, instead we're calling > the function to which it's pointing directly, and for no real reason. > Also, the function that's being called is a duplicate of the function > sock_def_readable() in net/sock.c. I'll look at correcting these > issues. Thanks a lot! Arne -- Arne Redlich Xiranet Communications GmbH From mst at mellanox.co.il Tue Jun 28 00:22:22 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 10:22:22 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <52fyv3z6yt.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> Message-ID: <20050628072222.GE25304@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] process locked in D state. > > Something like this should work... Note that there's a window when close() has returned but vm_locked isnt updated yet. Actually, a process that doesnt close memory regions before closing the device will leak (virtual) memory anyway. The only interesting case is of the process exiting, in which case mm is NULL. So I propose to kill all this code. Signed-off-by: Michael S. Tsirkin Index: core/uverbs_mem.c =================================================================== --- core/uverbs_mem.c (revision 2724) +++ core/uverbs_mem.c (working copy) @@ -180,40 +180,7 @@ static void ib_umem_account(void *work_p void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) { - struct mm_struct *mm; - + /* A well-behaved application will close regions before + * closing the file. Thus we dont have to update mm->locked_vm here. */ __ib_umem_release(dev, umem, 1); - - mm = get_task_mm(current); - if (mm) { - /* - * We may be called with the mm's mmap_sem already - * held. This can happen when a userspace munmap() is - * the call that drops the last reference to our file - * and calls our release method. If there are memory - * regions to destroy, we'll end up here and not be - * able to take the mmap_sem. - * - * To handle this, we try to grab the mmap_sem, and if - * we can't get it immediately, we defer the - * accounting to the system workqueue. - */ - if (down_write_trylock(&mm->mmap_sem)) { - mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; - up_write(&mm->mmap_sem); - mmput(mm); - } else { - struct ib_umem_account_work *work; - - work = kmalloc(sizeof *work, GFP_KERNEL); - if (!work) - return; - - INIT_WORK(&work->work, ib_umem_account, work); - work->mm = mm; - work->diff = PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; - - schedule_work(&work->work); - } - } } -- MST From glebn at voltaire.com Tue Jun 28 00:50:57 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Tue, 28 Jun 2005 10:50:57 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <20050628072222.GE25304@mellanox.co.il> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> Message-ID: <20050628075057.GM15312@minantech.com> On Tue, Jun 28, 2005 at 10:22:22AM +0300, Michael S. Tsirkin wrote: > Quoting r. Roland Dreier : > > Subject: Re: [PATCH] process locked in D state. > > > > Something like this should work... > > Note that there's a window when close() has returned but > vm_locked isnt updated yet. This is the case only if close() can return before workqueues are ran. Is this possible? If yes perhaps it is better to use tascklets. > Actually, a process that doesnt close memory regions before > closing the device will leak (virtual) memory anyway. This is not the kernel problem. > > The only interesting case is of the process exiting, in which case mm is > NULL. > > So I propose to kill all this code. By your patch you completely removed locked_vm accounting on deregister path. I don't think this was your intention. I think kernel should cleanup everything it can after close(). If user call ibv_close_device() he expects that the process is in the state such as ibv_open_device() was never called in the first place. > > Signed-off-by: Michael S. Tsirkin > > Index: core/uverbs_mem.c > =================================================================== > --- core/uverbs_mem.c (revision 2724) > +++ core/uverbs_mem.c (working copy) > @@ -180,40 +180,7 @@ static void ib_umem_account(void *work_p > > void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) > { > - struct mm_struct *mm; > - > + /* A well-behaved application will close regions before > + * closing the file. Thus we dont have to update mm->locked_vm here. */ > __ib_umem_release(dev, umem, 1); > - > - mm = get_task_mm(current); > - if (mm) { > - /* > - * We may be called with the mm's mmap_sem already > - * held. This can happen when a userspace munmap() is > - * the call that drops the last reference to our file > - * and calls our release method. If there are memory > - * regions to destroy, we'll end up here and not be > - * able to take the mmap_sem. > - * > - * To handle this, we try to grab the mmap_sem, and if > - * we can't get it immediately, we defer the > - * accounting to the system workqueue. > - */ > - if (down_write_trylock(&mm->mmap_sem)) { > - mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; > - up_write(&mm->mmap_sem); > - mmput(mm); > - } else { > - struct ib_umem_account_work *work; > - > - work = kmalloc(sizeof *work, GFP_KERNEL); > - if (!work) > - return; > - > - INIT_WORK(&work->work, ib_umem_account, work); > - work->mm = mm; > - work->diff = PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; > - > - schedule_work(&work->work); > - } > - } > } > > -- > MST -- Gleb. From mst at mellanox.co.il Tue Jun 28 02:25:22 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 12:25:22 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <20050628075057.GM15312@minantech.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> Message-ID: <20050628092521.GG25304@mellanox.co.il> > By your patch you completely removed locked_vm accounting on deregister > path. I don't think this was your intention. Of course. Closing the file should be handled separately from regular deregistration. A new patch (below) fixes that. --- Dont touch locked_vm when file is being closed, since a well-behaved user is either exiting or has freed all mrs already, and a malicious user is only hurting himself by making locked_vm seem higher than the actual usage. Note that current code doesnt seem to handle the later case anyway, since there's a window after close() returns where locked_vm isnt updated yet. Signed-off-by: Michael S. Tsirkin Index: core/uverbs_mem.c =================================================================== --- core/uverbs_mem.c (revision 2726) +++ core/uverbs_mem.c (working copy) @@ -37,13 +37,6 @@ #include "uverbs.h" -struct ib_umem_account_work { - struct work_struct work; - struct mm_struct *mm; - unsigned long diff; -}; - - static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty) { struct ib_umem_chunk *chunk, *tmp; @@ -167,15 +160,9 @@ out: return ret; } -static void ib_umem_account(void *work_ptr) +void ib_umem_put(struct ib_device *dev, struct ib_umem *umem) { - struct ib_umem_account_work *work = work_ptr; - - down_write(&work->mm->mmap_sem); - work->mm->locked_vm -= work->diff; - up_write(&work->mm->mmap_sem); - mmput(work->mm); - kfree(work); + __ib_umem_release(dev, umem, 1); } void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) @@ -185,35 +172,11 @@ void ib_umem_release(struct ib_device *d __ib_umem_release(dev, umem, 1); mm = get_task_mm(current); - if (mm) { - /* - * We may be called with the mm's mmap_sem already - * held. This can happen when a userspace munmap() is - * the call that drops the last reference to our file - * and calls our release method. If there are memory - * regions to destroy, we'll end up here and not be - * able to take the mmap_sem. - * - * To handle this, we try to grab the mmap_sem, and if - * we can't get it immediately, we defer the - * accounting to the system workqueue. - */ - if (down_write_trylock(&mm->mmap_sem)) { - mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; - up_write(&mm->mmap_sem); - mmput(mm); - } else { - struct ib_umem_account_work *work; - - work = kmalloc(sizeof *work, GFP_KERNEL); - if (!work) - return; - - INIT_WORK(&work->work, ib_umem_account, work); - work->mm = mm; - work->diff = PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; + if (!mm) + return; - schedule_work(&work->work); - } - } + down_write(&mm->mmap_sem); + mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; + up_write(&mm->mmap_sem); + mmput(mm); } Index: core/uverbs_main.c =================================================================== --- core/uverbs_main.c (revision 2726) +++ core/uverbs_main.c (working copy) @@ -133,7 +133,7 @@ static int ib_dealloc_ucontext(struct ib struct ib_umem_object *memobj; memobj = container_of(uobj, struct ib_umem_object, uobject); - ib_umem_release(mr->device, &memobj->umem); + ib_umem_put(mr->device, &memobj->umem); idr_remove(&ib_uverbs_mr_idr, uobj->id); ib_dereg_mr(mr); Index: core/uverbs.h =================================================================== --- core/uverbs.h (revision 2726) +++ core/uverbs.h (working copy) @@ -104,6 +104,7 @@ void ib_uverbs_qp_event_handler(struct i int ib_umem_get(struct ib_device *dev, struct ib_umem *mem, void *addr, size_t size, int write); void ib_umem_release(struct ib_device *dev, struct ib_umem *umem); +void ib_umem_put(struct ib_device *dev, struct ib_umem *umem); #define IB_UVERBS_DECLARE_CMD(name) \ ssize_t ib_uverbs_##name(struct ib_uverbs_file *file, \ -- MST From halr at voltaire.com Tue Jun 28 03:45:47 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 06:45:47 -0400 Subject: [openib-general] OpenSM on multiple HCA machine In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C303CB@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C303CB@mtlex01.yok.mtl.com> Message-ID: <1119955423.4848.2250.camel@hal.voltaire.com> On Tue, 2005-06-28 at 01:29, Eitan Zahavi wrote: > I was not aware of an issue with multiple HCAs in the OpenIB (gen2) > stack. > The Gen1 stack had this issue and it was resolved. I hope to be able > to focus on OpenIB stack in the coming months such that I can help Hal > in fixing these kind of issues too. OpenSM works fine when multiple HCAs are present. It can work on any port on any HCA. I need to refresh what the issue with multiple HCAs was exactly. -- Hal > > Eitan Zahavi > > -----Original Message----- > > From: Bernhard Fischer [mailto:rep.nop at aon.at] > > Sent: Tuesday, June 28, 2005 12:14 AM > > To: Hal Rosenstock > > Cc: Eitan Zahavi; openib-windows at openib.org; > 'openib-general at openib.org' > > Subject: Re: [openib-general] Re: IB Diagnositic Tools > > > > On Mon, Jun 27, 2005 at 02:04:16PM -0400, Hal Rosenstock wrote: > > >Hi Eitan, > > > > > >On Sat, 2005-06-25 at 15:25, Eitan Zahavi wrote: > > >> Following the discussion about the debug tools, I would like to > > >> propose using OpenSM Vendor layer as a common layer for > developing the > > >> debug tools. > > > > Hal, > > > > This is kinda offtopic, but (iirc) i once stumbled over the issue of > > "port" vs. "mgmt port" [back then i had access to two 2-port cards] > > where you may have said something along the lines of > > \"There is clearly a bug for multi HCAs in > osm_vendor_get_all_port_attr > > which is in the vendor layer. This needs to be fixed and is our > problem. > > So I am close to being able to commit what I now have for this and > fix > > this later (as there are other multi HCA issues).\" > > Just curious.. did somebody already have a chance to touch those or > not? > > > > On a related note (just the same thing, i tend to think) > > /* "local ports" is(?) phys, shouldn't this exclude port 0 then ? */ > > > > I'm not too familiar with these kindof questions, which might be > define > > in a spec, so any hint on this would be well received, at least from > my > > part. > > > > anyone? Eitan? > > -- > > thank you, > > Bernhard > From halr at voltaire.com Tue Jun 28 04:07:07 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 07:07:07 -0400 Subject: [openib-general] RE: IB Diagnositic Tools In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C303C9@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C303C9@mtlex01.yok.mtl.com> Message-ID: <1119956826.4848.2291.camel@hal.voltaire.com> On Mon, 2005-06-27 at 15:27, Eitan Zahavi wrote: > > The current OpenIB topology file has a place where these annotations > can > > be made (and displayed). > [EZ] How would you define the internal structure of a 288port switch > in the existing topology file? > Would it support writing code that is able to report something like > "board spine2 of system mySwitch is missing"? I think it could. > The code that supports all that is part of the simulator code I have > posted long ago. > Please give it a look. Especially the Fabric.h, SysDef.h, > ibnl_parser.yy in > https://openib.org/svn/gen2/utils/src/linux-user/ibdm/datamodel Why part of the simulator ? Will these be part of the to be released diagnostics ? > > This brings in more things that are not currently ported to OpenIB > and > > also there are some issues with some of these tools. > [EZ] Never heard of any specific issue. Can you describe these issues? I'm pretty sure they were discussed on this list before perhaps quite a while ago. -- Hal From eitan at mellanox.co.il Tue Jun 28 04:30:12 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Tue, 28 Jun 2005 14:30:12 +0300 Subject: [openib-general] RE: IB Diagnositic Tools Message-ID: <506C3D7B14CDD411A52C00025558DED607C303D3@mtlex01.yok.mtl.com> > > [EZ] How would you define the internal structure of a 288port switch > > in the existing topology file? > > Would it support writing code that is able to report something like > > "board spine2 of system mySwitch is missing"? > > I think it could. [EZ] Anyway this capability already exists in IBDM. So why re-develop it again? Also you did not answer the first question. Is the implementation open to support definition of hierarchical internal IB network like the one embedded in a 288port switch? > > > The code that supports all that is part of the simulator code I have > > posted long ago. > > Please give it a look. Especially the Fabric.h, SysDef.h, > > ibnl_parser.yy in > > https://openib.org/svn/gen2/utils/src/linux-user/ibdm/datamodel > > Why part of the simulator ? Will these be part of the to be released > diagnostics ? [EZ] It was uploaded with the simulator. It is a separate library: IBDM. The simulator uses it to define the topology that is simulated. > > > > This brings in more things that are not currently ported to OpenIB > > and > > > also there are some issues with some of these tools. > > [EZ] Never heard of any specific issue. Can you describe these issues? > > I'm pretty sure they were discussed on this list before perhaps quite a > while ago. [EZ] If nobody remembers what they were maybe they are not so subtle and critical? > > -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Tue Jun 28 05:08:54 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 15:08:54 +0300 Subject: [openib-general] Re: IB Diagnositic Tools In-Reply-To: <1119895456.4848.353.camel@hal.voltaire.com> References: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com> <1119895456.4848.353.camel@hal.voltaire.com> Message-ID: <20050628120854.GI25304@mellanox.co.il> Quoting r. Hal Rosenstock : > Will the Linux distros take it this way (with #ifdef OS) I never looked at opensm vendor layer, but generally in userspace code, it is not always worth it the effort to get rid of all os-dependent code. "If you want an application to be portable, you don't necessarily create an abstraction layer like a microkernel so much as you program intelligently." -- Linus Torvalds on Microkernels (Open Sources, 1999 O'Reilly and Associates) -- MST From mst at mellanox.co.il Tue Jun 28 05:39:32 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 15:39:32 +0300 Subject: [openib-general] backport patch for 2.6.11 Message-ID: <20050628123932.GK25304@mellanox.co.il> Hi! I have uploaded the patches to make svn trunk build on 2.6.11. They can be found in https://openib.org/svn/gen2/branches/backport-to-2.6.11 Enjoy! -- MST From halr at voltaire.com Tue Jun 28 05:41:18 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 08:41:18 -0400 Subject: [openib-general] backport patch for 2.6.11 In-Reply-To: <20050628123932.GK25304@mellanox.co.il> References: <20050628123932.GK25304@mellanox.co.il> Message-ID: <1119962478.4848.2320.camel@hal.voltaire.com> On Tue, 2005-06-28 at 08:39, Michael S. Tsirkin wrote: > Hi! > I have uploaded the patches to make svn trunk build on 2.6.11. > They can be found in > > https://openib.org/svn/gen2/branches/backport-to-2.6.11 > > Enjoy! Great. Will this be periodically updated ? Any idea for how long this would be maintained ? Thanks. -- Hal From halr at voltaire.com Tue Jun 28 06:10:34 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 09:10:34 -0400 Subject: [openib-general] RE: IB Diagnositic Tools In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C303D3@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C303D3@mtlex01.yok.mtl.com> Message-ID: <1119964233.4848.2363.camel@hal.voltaire.com> On Tue, 2005-06-28 at 07:30, Eitan Zahavi wrote: > > > [EZ] How would you define the internal structure of a 288port > switch > > > in the existing topology file? > > > Would it support writing code that is able to report something > like > > > "board spine2 of system mySwitch is missing"? > > > > I think it could. > [EZ] Anyway this capability already exists in IBDM. So why re-develop > it again? IBDM is not available for OpenIB as yet and the OpenIB diagnostics have been in place for a while now. Also, as far as I know, no one outside of Mellanox has looked at this. So that would be a next step. Is there any documentation on IBDM ? Also, are there updates to this or is the version in the tree what should be looked at (and commented on) ? > Also you did not answer the first question. By using the comment fields appropriately with semantics for logical hierarchy. > Is the implementation open to support definition of hierarchical > internal IB network like the one embedded in a 288port switch? Yes. > > > The code that supports all that is part of the simulator code I > have > > > posted long ago. > > > Please give it a look. Especially the Fabric.h, SysDef.h, > > > ibnl_parser.yy in > > > https://openib.org/svn/gen2/utils/src/linux-user/ibdm/datamodel > > > > Why part of the simulator ? Will these be part of the to be released > > diagnostics ? > [EZ] It was uploaded with the simulator. It is a separate library: > IBDM. > The simulator uses it to define the topology that is simulated. OK. > > > > This brings in more things that are not currently ported to > OpenIB > > > and > > > > also there are some issues with some of these tools. > > > [EZ] Never heard of any specific issue. Can you describe these > issues? > > > > I'm pretty sure they were discussed on this list before perhaps > quite a > > while ago. > [EZ] If nobody remembers what they were maybe they are not so subtle > and critical? IMO that was not the case. I don't have the time to go dig these back out right now. -- Hal From mst at mellanox.co.il Tue Jun 28 06:26:47 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 16:26:47 +0300 Subject: [openib-general] backport patch for 2.6.11 In-Reply-To: <1119962478.4848.2320.camel@hal.voltaire.com> References: <20050628123932.GK25304@mellanox.co.il> <1119962478.4848.2320.camel@hal.voltaire.com> Message-ID: <20050628132647.GL25304@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: Re: [openib-general] backport patch for 2.6.11 > > On Tue, 2005-06-28 at 08:39, Michael S. Tsirkin wrote: > > Hi! > > I have uploaded the patches to make svn trunk build on 2.6.11. > > They can be found in > > > > https://openib.org/svn/gen2/branches/backport-to-2.6.11 > > > > Enjoy! > > Great. Will this be periodically updated ? Thats the idea. > Any idea for how long this > would be maintained ? Thanks. > > -- Hal > As long as I need it. -- MST From halr at voltaire.com Tue Jun 28 06:22:00 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 09:22:00 -0400 Subject: [Openib-windows] Re: [openib-general] Re: IB Diagnositic Tools In-Reply-To: <1119907826.4848.486.camel@hal.voltaire.com> References: <506C3D7B14CDD411A52C00025558DED607C30394@mtlex01.yok.mtl.com> <1119895456.4848.353.camel@hal.voltaire.com> <20050627211346.GA17011@aon.at> <1119907826.4848.486.camel@hal.voltaire.com> Message-ID: <1119964919.4848.2387.camel@hal.voltaire.com> On Mon, 2005-06-27 at 17:30, Hal Rosenstock wrote: > On Mon, 2005-06-27 at 17:13, Bernhard Fischer wrote: > > Hal, > > > > This is kinda offtopic, > > Yes, this is different topic(s). > > > but (iirc) i once stumbled over the issue of > > "port" vs. "mgmt port" [back then i had access to two 2-port cards] > > where you may have said something along the lines of > > \"There is clearly a bug for multi HCAs in osm_vendor_get_all_port_attr > > which is in the vendor layer. This needs to be fixed and is our problem. > > So I am close to being able to commit what I now have for this and fix > > this later (as there are other multi HCA issues).\" > > Just curious.. did somebody already have a chance to touch those or not? > > I don't think it has been fixed :-( I will need to refresh myself on > this again. Just tried the diags and OpenSM on a multi HCA machine and they appear to run fine. I will need to dig out from my notes as that comment came from back on 2/16. -- Hal From eitan at mellanox.co.il Tue Jun 28 06:34:16 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Tue, 28 Jun 2005 16:34:16 +0300 Subject: [openib-general] RE: IB Diagnositic Tools Message-ID: <506C3D7B14CDD411A52C00025558DED607C303D5@mtlex01.yok.mtl.com> > > On Tue, 2005-06-28 at 07:30, Eitan Zahavi wrote: > > > > [EZ] How would you define the internal structure of a 288port > > switch > > > > in the existing topology file? > > > > Would it support writing code that is able to report something > > like > > > > "board spine2 of system mySwitch is missing"? > > > > > > I think it could. > > [EZ] Anyway this capability already exists in IBDM. So why re-develop > > it again? > > IBDM is not available for OpenIB as yet and the OpenIB diagnostics have > been in place for a while now. [EZ] IBDM is available since I have uploaded it when I uploaded the simulator code. I'm not talking about IBADM here. IBDM= IB Data Model. > > Also, as far as I know, no one outside of Mellanox has looked at this. > So that would be a next step. > > Is there any documentation on IBDM ? [EZ] Not much. The attached tar file holds: IBDM Overview - a diagram showing IBDM objects and their relations (maybe I should UML this one day). Ibdm_wrap.html - the TCL API to the IBDM utilities and datamodel IBNL Specification - describing the format for defining arbitrary systems IB netlist MTS14400-48.ibnl - example IBNL for 144 ports switch. > > Also, are there updates to this or is the version in the tree what > should be looked at (and commented on) ? > > > Also you did not answer the first question. > > By using the comment fields appropriately with semantics for logical > hierarchy. [EZ] So the user is supposed to annotate the entire cluster at the node level with hierarchy names? Is there a tool to create these netlists? > > > Is the implementation open to support definition of hierarchical > > internal IB network like the one embedded in a 288port switch? > > Yes. [EZ] Can you provide an example system definition file? > > > > > The code that supports all that is part of the simulator code I > > have > > > > posted long ago. > > > > Please give it a look. Especially the Fabric.h, SysDef.h, > > > > ibnl_parser.yy in > > > > https://openib.org/svn/gen2/utils/src/linux-user/ibdm/datamodel > > > > > > Why part of the simulator ? Will these be part of the to be released > > > diagnostics ? > > [EZ] It was uploaded with the simulator. It is a separate library: > > IBDM. > > The simulator uses it to define the topology that is simulated. > > OK. > > > > > > This brings in more things that are not currently ported to > > OpenIB > > > > and > > > > > also there are some issues with some of these tools. > > > > [EZ] Never heard of any specific issue. Can you describe these > > issues? > > > > > > I'm pretty sure they were discussed on this list before perhaps > > quite a > > > while ago. > > [EZ] If nobody remembers what they were maybe they are not so subtle > > and critical? > > IMO that was not the case. I don't have the time to go dig these back > out right now. > > -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: IBDM Tutorial.zip Type: application/octet-stream Size: 83007 bytes Desc: not available URL: From Thomas.Talpey at netapp.com Tue Jun 28 06:38:16 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Tue, 28 Jun 2005 09:38:16 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <523br34h6n.fsf@topspin.com> References: <91DB792C7985D411BEC300B40080D29CC35D0E@mtvex01.mtv.mtl.com> <6.2.1.2.2.20050627151532.04a0a480@exnane01.nane.netapp.com> <523br34h6n.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050628090215.04e2ceb0@exnane01.nane.netapp.com> At 05:34 PM 6/27/2005, Roland Dreier wrote: >I'm not sure I understand this. At best, ATS can give you back a list >of IPs. How do you decide which one to check against the exports? Any or all of them. Exports is a fairly simple access list, and membership by the client is all that's required. It supports wildcards as well as single address entries. Here's the example from the Linux manpage: # sample /etc/exports file / master(rw) trusty(rw,no_root_squash) /projects proj*.local.domain(rw) /usr *.local.domain(ro) @trusted(rw) /home/joe pc001(rw,all_squash,anonuid=150,anongid=100) /pub (ro,insecure,all_squash) See the wildcards? If any of the machine's IPs matches one, that line yields true. Also of course, even the non-wildcards can expand to a list of addresses; in the first line "master" is a single host, any of its IP addresses is eligible for a match. >In a pure IP world, every packet from a multihomed client carries a >source IP address. So a server can use getpeername() to determine >which address a client is connecting from. This is fundamentally >different from ATS. I don't understand. ATS allows each incoming connection to map to one or more IP addresses, effectively supporting getpeername() on the IB QP. DAPL passes this address up to the consumer in the connection indication via the cr_param's ia_address_ptr. The consumer doesn't invoke ATS directly, nor would it want to. In the NFS server case, it just needs to run this address down the exports list, same way it would for a TCP connection or UDP datagram. Tom. From jlentini at netapp.com Tue Jun 28 06:42:46 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 09:42:46 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [udapl] fix build for x86_64 In-Reply-To: <42C07A5D.5000201@ichips.intel.com> References: <42C07A5D.5000201@ichips.intel.com> Message-ID: Arlin, Would this be a more standard way to do the cast: On Mon, 27 Jun 2005, Arlin Davis wrote: > Signed-off-by: Arlin Davis > > Index: dapl/openib/dapl_ib_dto.h > =================================================================== > --- dapl/openib/dapl_ib_dto.h (revision 2720) > +++ dapl/openib/dapl_ib_dto.h (working copy) > @@ -88,7 +88,7 @@ > total_len = 0; > wr.next = 0; > wr.num_sge = 0; > - wr.wr_id = (uint64_t)cookie; > + wr.wr_id = (uint64_t)(unsigned long)cookie; wr.wr_id = (uint64_t)(uintptr_t)cookie; > wr.sg_list = ds_array_p; > > for (i = 0; i < segments; i++ ) { > @@ -162,7 +162,7 @@ > wr.opcode = op_type; > wr.num_sge = 0; > wr.send_flags = 0; > - wr.wr_id = (uint64_t)cookie; > + wr.wr_id = (uint64_t)(unsigned long)cookie; wr.wr_id = (uint64_t)(uintptr_t)cookie; > wr.sg_list = ds_array_p; > total_len = 0; > From halr at voltaire.com Tue Jun 28 06:39:54 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 09:39:54 -0400 Subject: [openib-general] RE: IB Diagnositic Tools In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C303D5@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C303D5@mtlex01.yok.mtl.com> Message-ID: <1119965993.4848.2409.camel@hal.voltaire.com> On Tue, 2005-06-28 at 09:34, Eitan Zahavi wrote: > > > > On Tue, 2005-06-28 at 07:30, Eitan Zahavi wrote: > > > > > [EZ] How would you define the internal structure of a 288port > > > switch > > > > > in the existing topology file? > > > > > Would it support writing code that is able to report something > > > like > > > > > "board spine2 of system mySwitch is missing"? > > > > > > > > I think it could. > > > [EZ] Anyway this capability already exists in IBDM. So why > re-develop > > > it again? > > > > IBDM is not available for OpenIB as yet and the OpenIB diagnostics > have > > been in place for a while now. > [EZ] IBDM is available since I have uploaded it when I uploaded the > simulator code. > I'm not talking about IBADM here. IBDM= IB Data Model. Understood. > > Also, as far as I know, no one outside of Mellanox has looked at > this. > > So that would be a next step. > > > > Is there any documentation on IBDM ? > [EZ] Not much. The attached tar file holds: > IBDM Overview - a diagram showing IBDM objects and their relations > (maybe I should UML this one day). > Ibdm_wrap.html - the TCL API to the IBDM utilities and datamodel > IBNL Specification - describing the format for defining arbitrary > systems IB netlist > MTS14400-48.ibnl - example IBNL for 144 ports switch. Why not put this in ibdm/doc ? > > Also, are there updates to this or is the version in the tree what > > should be looked at (and commented on) ? Is IBDM still version 1.0 (with no updates from what is in the tree) ? > > > Also you did not answer the first question. > > > > By using the comment fields appropriately with semantics for logical > > hierarchy. > [EZ] So the user is supposed to annotate the entire cluster at the > node level with hierarchy names? Is there a tool to create these > netlists? > > > > > > Is the implementation open to support definition of hierarchical > > > internal IB network like the one embedded in a 288port switch? > > > > Yes. > [EZ] Can you provide an example system definition file? Yes but not right now. When attempting to build ibdm per instructions in INSTALL, I get the following error: make install cp .libs/libibdm.so.1.1.1 /usr/local/lib/ibadm/ibdm1.0/libibdm.so.1.0 cp: cannot stat `.libs/libibdm.so.1.1.1': No such file or directory -- Hal From halr at voltaire.com Tue Jun 28 06:48:13 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 09:48:13 -0400 Subject: [openib-general] Simulator building Message-ID: <1119966492.4848.2420.camel@hal.voltaire.com> Hi Eitan, First, a nit. Should IBMgtSim/src/.autosave be removed from the tree ? Also, have you gone through the instructions in README on a fresh machine ? It fails for me on the first step as follows: ./config/bootstrap /home/hal/openib/utils/src/linux-user/IBMgtSim/config/libtoolize: line 1: cd: /home/eitan/SW/autotools/share/libtool: No such file or directory libtoolize: cannot list files in `/home/eitan/SW/autotools/share/libtool' configure.in: installing `config/install-sh' configure.in: installing `config/mkinstalldirs' configure.in: installing `config/missing' Use of uninitialized value in concatenation (.) or string at /usr/bin/automake line 8449. : installing `config/config.guess' Use of uninitialized value in concatenation (.) or string at /usr/bin/automake line 8449. : installing `config/config.sub' aclocal.m4:983: required file `config/ltmain.sh' not found automake: src/Makefile.am: object `client.$(OBJEXT)' created both with libtool and without ... It shouldn't be going after /home/eitan/... In INSTALL, it states: Software Dependencies ===================== IBMgtSim depends on: tcl8.3/8.4, OpenSM 1.7.0 and ibdm 1.0 What are the specific OpenSM 1.7.0 dependencies ? Also, can you update the instructions (README, INSTALL, etc.) for OpenIB ? Thanks. -- Hal From Thomas.Talpey at netapp.com Tue Jun 28 06:58:16 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Tue, 28 Jun 2005 09:58:16 -0400 Subject: [openib-general] [PATCH][RFC] nfsordma: initial port of nfsrdma to 2.6 and james'sss kdapl In-Reply-To: <1119911140.25388.6.camel@duffman> References: <1119911140.25388.6.camel@duffman> Message-ID: <6.2.1.2.2.20050628094824.04e0dc10@exnane01.nane.netapp.com> At 06:25 PM 6/27/2005, Tom Duffy wrote: >I have done some initial work to port nfsrdma to 2.6 and to James's >kDAPL. This builds now inside the kernel. Tom, thanks for starting this and I'll take a look at your approach. In fact we already have a version working on 2.6.11, the main change merging up to 2.6.12 is having to merge with Chuck's new RPC transport switch. We are definitely willing to GPL the code, I am in the process of getting that approved and putting the result in the pipeline. I am not sure about using the OpenIB repository just yet, because of the dependency on the RPC transport. We don't want to inject the changes in multiple places. Let's caucus offline to figure out how to handle the repository question. It shouldn't be a major issue once we figure out the dependencies. I'll get back to you tomorrow (after taking a look at these patches too). Tom. > >You will need to follow the kDAPL directions first to put that in your >kernel tree, then slap this patch over top of that. > >So you have 2.6.12 + svn drivers/infiniband + kdapl from james's tree + >this patch to get it to build. Oh you will also need to patch the rpc >header to get it to build. > >I think it is time to open up a tree in openib repository. Tom, is >netapp willing to GPL this code? > From eitan at mellanox.co.il Tue Jun 28 07:10:38 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Tue, 28 Jun 2005 17:10:38 +0300 Subject: [openib-general] RE: Simulator building Message-ID: <506C3D7B14CDD411A52C00025558DED607C303D6@mtlex01.yok.mtl.com> This is all good input. I will go through and revise the material. Also I will upload the newest and greatest code. This should be done during the weekend and be there for Monday. Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Tuesday, June 28, 2005 4:48 PM > To: Eitan Zahavi > Cc: openib-general at openib.org > Subject: Simulator building > > Hi Eitan, > > First, a nit. Should IBMgtSim/src/.autosave be removed from the tree ? > > Also, have you gone through the instructions in README on a fresh > machine ? It fails for me on the first step as follows: > > ./config/bootstrap > /home/hal/openib/utils/src/linux-user/IBMgtSim/config/libtoolize: line 1: cd: > /home/eitan/SW/autotools/share/libtool: No such file or directory > libtoolize: cannot list files in `/home/eitan/SW/autotools/share/libtool' > configure.in: installing `config/install-sh' > configure.in: installing `config/mkinstalldirs' > configure.in: installing `config/missing' > Use of uninitialized value in concatenation (.) or string at /usr/bin/automake line 8449. > : installing `config/config.guess' > Use of uninitialized value in concatenation (.) or string at /usr/bin/automake line 8449. > : installing `config/config.sub' > aclocal.m4:983: required file `config/ltmain.sh' not found > automake: src/Makefile.am: object `client.$(OBJEXT)' created both with libtool and > without > ... > > It shouldn't be going after /home/eitan/... > > In INSTALL, it states: > Software Dependencies > ===================== > IBMgtSim depends on: tcl8.3/8.4, OpenSM 1.7.0 and ibdm 1.0 > > What are the specific OpenSM 1.7.0 dependencies ? > > Also, can you update the instructions (README, INSTALL, etc.) for OpenIB > ? > > Thanks. > > -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: From rolandd at cisco.com Tue Jun 28 07:10:13 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 07:10:13 -0700 Subject: [openib-general] [PATCH] process locked in D state. In-Reply-To: <20050628070441.GL15312@minantech.com> (Gleb Natapov's message of "Tue, 28 Jun 2005 10:04:41 +0300") References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628070441.GL15312@minantech.com> Message-ID: <52aclay3lm.fsf@topspin.com> Gleb> Yes, but don't forget the first part of my patch (for Gleb> libibverbs/src/device.c) I think it should be applied to. Yes, I guess it makes sense. > - mm = get_task_mm(current); Gleb> Are you sure that ib_umem_account will run on behalf the Gleb> process that schedules it? Anyway you are correctly using Gleb> work->mm in the rest of the function so this line should be Gleb> dropped I think. The '-' at the beginning of the line means it is being deleted in the patch! Of course schedule_work() delegates the work to a different process so the get_task_mm() must be done in the original process. - R. From rolandd at cisco.com Tue Jun 28 07:11:23 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 07:11:23 -0700 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <20050628075057.GM15312@minantech.com> (Gleb Natapov's message of "Tue, 28 Jun 2005 10:50:57 +0300") References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> Message-ID: <521x6my3jo.fsf@topspin.com> Gleb> This is the case only if close() can return before Gleb> workqueues are ran. Is this possible? If yes perhaps it is Gleb> better to use tascklets. Yes, it's possible. Unfortunately a tasklet can't be used because tasklet context can't do down_write(). - R. From rolandd at cisco.com Tue Jun 28 07:11:54 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 07:11:54 -0700 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <20050628092521.GG25304@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 28 Jun 2005 12:25:22 +0300") References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> Message-ID: <52wtoewoyd.fsf@topspin.com> Michael> Of course. Closing the file should be handled separately Michael> from regular deregistration. A new patch (below) fixes Michael> that. This looks better than my approach. Let me make sure it covers everything, but it definitely seems simpler. - R. From tduffy at sun.com Tue Jun 28 08:13:55 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 28 Jun 2005 08:13:55 -0700 Subject: [openib-general] [PATCH][RFC] nfsordma: initial port of nfsrdma to 2.6 and james'sss kdapl In-Reply-To: <6.2.1.2.2.20050628094824.04e0dc10@exnane01.nane.netapp.com> References: <1119911140.25388.6.camel@duffman> <6.2.1.2.2.20050628094824.04e0dc10@exnane01.nane.netapp.com> Message-ID: <1119971635.31370.3.camel@duffman> On Tue, 2005-06-28 at 09:58 -0400, Talpey, Thomas wrote: > At 06:25 PM 6/27/2005, Tom Duffy wrote: > >I have done some initial work to port nfsrdma to 2.6 and to James's > >kDAPL. This builds now inside the kernel. > > Tom, thanks for starting this and I'll take a look at your approach. > In fact we already have a version working on 2.6.11, the main change > merging up to 2.6.12 is having to merge with Chuck's new RPC > transport switch. I am sure I got the RPC stuff wrong. I just wanted to make it compile against James's kDAPL and inside the drivers/infiniband directory. Where can I find the 2.6.11 version of the patch? > We are definitely willing to GPL the code, I am in the process of > getting that approved and putting the result in the pipeline. I am > not sure about using the OpenIB repository just yet, because of > the dependency on the RPC transport. We don't want to inject > the changes in multiple places. Well, there is going to be an openib branch because of the *DAPL changes. Better sooner than later. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From Thomas.Talpey at netapp.com Tue Jun 28 08:28:32 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Tue, 28 Jun 2005 11:28:32 -0400 Subject: [openib-general] [PATCH][RFC] nfsordma: initial port of nfsrdma to 2.6 and james'sss kdapl In-Reply-To: <1119971635.31370.3.camel@duffman> References: <1119911140.25388.6.camel@duffman> <6.2.1.2.2.20050628094824.04e0dc10@exnane01.nane.netapp.com> <1119971635.31370.3.camel@duffman> Message-ID: <6.2.1.2.2.20050628111852.0685fc60@exnane01.nane.netapp.com> At 11:13 AM 6/28/2005, Tom Duffy wrote: >I am sure I got the RPC stuff wrong. I just wanted to make it compile >against James's kDAPL and inside the drivers/infiniband directory. > >Where can I find the 2.6.11 version of the patch? Here: This is kind of involved, you need to apply some prepatches as well as postpatches. The 2.6.12 is significantly cleaner but it implements a new API, due to Trond's requests. In other words, moving target alert. I have a working tarball against 2.6.11. The two files rdma_transport.c and rdma_marshal.c plus what you're doing in rdma_kdapl.c might just work. But it's not GPL yet. Can you wait a couple of days? Tom. From glebn at voltaire.com Tue Jun 28 08:33:45 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Tue, 28 Jun 2005 18:33:45 +0300 Subject: [openib-general] [PATCH] process locked in D state. In-Reply-To: <52aclay3lm.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628070441.GL15312@minantech.com> <52aclay3lm.fsf@topspin.com> Message-ID: <20050628153345.GN15312@minantech.com> On Tue, Jun 28, 2005 at 07:10:13AM -0700, Roland Dreier wrote: > Gleb> Yes, but don't forget the first part of my patch (for > Gleb> libibverbs/src/device.c) I think it should be applied to. > > Yes, I guess it makes sense. > > > - mm = get_task_mm(current); > > Gleb> Are you sure that ib_umem_account will run on behalf the > Gleb> process that schedules it? Anyway you are correctly using > Gleb> work->mm in the rest of the function so this line should be > Gleb> dropped I think. > > The '-' at the beginning of the line means it is being deleted in the > patch! Of course schedule_work() delegates the work to a different > process so the get_task_mm() must be done in the original process. > Oops. I should be more careful :) -- Gleb. From halr at voltaire.com Tue Jun 28 08:30:46 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 11:30:46 -0400 Subject: [openib-general] [PATCH] [uDAPL] Fix compile of dapl_bpool.c on x86_64 Message-ID: <1119972645.4848.2434.camel@hal.voltaire.com> I was able to build udapltest for both x86 and x86-64 with the following change: uDAPL: Fix compile of dapl_bpool.c on x86_64 Signed-off-by: Hal Rosenstock --- userspace/test/dapltest/test/dapl_bpool.c.orig 2005-01-18 13:20:31.000000000 -0500 +++ userspace/test/dapltest/test/dapl_bpool.c 2005-06-24 23:47:43.000000000 -0400 @@ -356,8 +356,8 @@ "BPOOL alloc_size %x\n", (int) bpool_ptr->alloc_size); DT_Tdep_PT_Printf (phead, - "BPOOL pz_handle %x\n", - (int) bpool_ptr->pz_handle); + "BPOOL pz_handle %p\n", + bpool_ptr->pz_handle); DT_Tdep_PT_Printf (phead, "BPOOL num_segs %x\n", (int) bpool_ptr->num_segs); From glebn at voltaire.com Tue Jun 28 08:42:04 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Tue, 28 Jun 2005 18:42:04 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <521x6my3jo.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <521x6my3jo.fsf@topspin.com> Message-ID: <20050628154204.GO15312@minantech.com> On Tue, Jun 28, 2005 at 07:11:23AM -0700, Roland Dreier wrote: > Gleb> This is the case only if close() can return before > Gleb> workqueues are ran. Is this possible? If yes perhaps it is > Gleb> better to use tascklets. > > Yes, it's possible. Unfortunately a tasklet can't be used because > tasklet context can't do down_write(). > Then workqueues not good enough. And you are right about tascklets of cause. It seems there is no 100% right and simple solution then. -- Gleb. From glebn at voltaire.com Tue Jun 28 08:48:12 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Tue, 28 Jun 2005 18:48:12 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <52wtoewoyd.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> <52wtoewoyd.fsf@topspin.com> Message-ID: <20050628154812.GP15312@minantech.com> On Tue, Jun 28, 2005 at 07:11:54AM -0700, Roland Dreier wrote: > Michael> Of course. Closing the file should be handled separately > Michael> from regular deregistration. A new patch (below) fixes > Michael> that. > > This looks better than my approach. Let me make sure it covers > everything, but it definitely seems simpler. Perhaps it is worth to add trylock in ib_umem_put()? At least it will work like it should if user do munmap before close (and this is will be done by ibv_close_device() automatically). -- Gleb. From mshefty at ichips.intel.com Tue Jun 28 09:38:02 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 28 Jun 2005 09:38:02 -0700 Subject: [openib-general] [PATCH] cm: fix for service_id/service_mask hole In-Reply-To: <5D78D28F88822E4D8702BB9EEF1A4367140273@mercury.infiniconsys.com> References: <5D78D28F88822E4D8702BB9EEF1A4367140273@mercury.infiniconsys.com> Message-ID: <42C17CEA.109@ichips.intel.com> Jordan, Bill wrote: > Sean, > > Here is a to fix a bug in service_id/service_mask > handling in CM. > > Previously, if you tried to listen on service_id: > service_mask of 0x00:0x0F, followed by a listen > on 0xF0:0xF0, the second listen would fail because > the two ranges overlap (0xF0 appears in both ranges). > But, if you tried the listens in the reverse order, > both would succeed leading to an ambigious listen. > > This patch forces both service_ids in the compare to be > masked against both mask values first. Thanks - committed. - Sean From David.Brean at Sun.COM Tue Jun 28 10:24:34 2005 From: David.Brean at Sun.COM (David M. Brean) Date: Tue, 28 Jun 2005 13:24:34 -0400 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <42BC443E.6040802@hp.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <42BC3CBB.6030909@hp.com> <6.2.1.2.2.20050624131145.060437b0@exnane01.nane.netapp.com> <42BC443E.6040802@hp.com> Message-ID: <42C187D2.2090805@sun.com> The Sun implementation uses a Service ID from the "Local OS Administrative" range when installing an entry in the SA. So, the first byte should be 0x2 in the ServiceID component in the SA record. The Service Name in the SA record contained the IP address with the NFS port number appended at the end. So, this component of the SA record would appear as :2049 where IPaddress is provided by the NFS server. So, the well-known TCP port number for NFS is used in the Service Name. [Note, the information that I have is from a discussion with the Sun NFS team a year ago. I haven't used an analyzer to examine the data crossing the IB links.] I think that a well-known Service ID has more value when using the CM private-data mechanism. [I have some comments about ATS vs CM private-data mechanisms that I'll send in seperate mail.] -David Jay Rosser wrote: > >>> On the subject of NFS/RDMA, what is the IB ServiceID space that is >>> used? If I recall correctly, I have seen simply the value 2049 (i.e. >>> the standard TCP/UDP port number) used in some implementations (i.e. >>> 00 00 00 00 00 00 20 49). Is there a mapping onto an IB ServiceID >>> defined? >>> >> >> >> We aren't currently using the portmapper to discover the serviceid that >> the NFS/RDMA server is listening on. Brent Callaghan chose serviceid >> 2049 >> as a convenience in Sun's first implementation, and so far it has stuck. >> >> Theoretically the server can listen on any endpoint it chooses, this is >> how NFS/TCP and NFS/UDP work. But typically all servers use the well >> known port. It's probably a good idea to define a better default >> mapping. >> >> > Using a well-known port seems okay to me. > > However, if I understand correctly, the 2049 ServiceId is in the > IBTA's ServiceID space (i.e. for IBTA-defined well-known ServiceIDs - > see IB1.2 Vol1 pg 1181 A3.2.3.1). It would probably make sense to use > the IETF ServiceID space (A3.2.3.2) but that would require someone to > administer it. > > Thanks, > > Jay > >> Tom. >> >> >> > > From halr at voltaire.com Tue Jun 28 10:34:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 13:34:51 -0400 Subject: [openib-general] RMPP Message-ID: <1119980090.4848.2512.camel@hal.voltaire.com> Hi Sean, I'm in the process of enabling the receive side RMPP from user space and this is what I'm seeing in terms of RMPP right now. I have a question about the OpenSM side. SA client OpenSM SA GetTable (PortInfoRecord) --> <-- SA GetTableResp (PortInfoRecord) RMPP active, first payload length 0x44C retries is set to 4 so I see 4 responses (at 2 sec intervals) as the client is not currently ACKing. All is fine up to that point. At that point, OpenSM sees a large receive which appears to be that send timing out (nothing was sent nor observed on the IB wire). Could a timed out RMPP send end up as a receive somehow ? Thanks. -- Hal From mshefty at ichips.intel.com Tue Jun 28 10:44:57 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 28 Jun 2005 10:44:57 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1119980090.4848.2512.camel@hal.voltaire.com> References: <1119980090.4848.2512.camel@hal.voltaire.com> Message-ID: <42C18C99.6060102@ichips.intel.com> Hal Rosenstock wrote: > Hi Sean, > > I'm in the process of enabling the receive side RMPP from user space and > this is what I'm seeing in terms of RMPP right now. I have a question > about the OpenSM side. > > SA client OpenSM > SA GetTable (PortInfoRecord) --> > <-- SA GetTableResp (PortInfoRecord) > RMPP active, first > payload length 0x44C > > retries is set to 4 so I see 4 responses (at 2 sec intervals) as the > client is not currently ACKing. All is fine up to that point. > > At that point, OpenSM sees a large receive which appears to be that send > timing out (nothing was sent nor observed on the IB wire). > > Could a timed out RMPP send end up as a receive somehow ? On the side that sent the MAD? That should be no. On the remote side, a send can timeout, but still be received, since the ACK is unreliable. But I don't think that this is the situation that you're describing. - Sean From halr at voltaire.com Tue Jun 28 10:48:24 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 13:48:24 -0400 Subject: [openib-general] Re: RMPP In-Reply-To: <42C18C99.6060102@ichips.intel.com> References: <1119980090.4848.2512.camel@hal.voltaire.com> <42C18C99.6060102@ichips.intel.com> Message-ID: <1119980903.4848.2518.camel@hal.voltaire.com> On Tue, 2005-06-28 at 13:44, Sean Hefty wrote: > Hal Rosenstock wrote: > > Hi Sean, > > > > I'm in the process of enabling the receive side RMPP from user space and > > this is what I'm seeing in terms of RMPP right now. I have a question > > about the OpenSM side. > > > > SA client OpenSM > > SA GetTable (PortInfoRecord) --> > > <-- SA GetTableResp (PortInfoRecord) > > RMPP active, first > > payload length 0x44C > > > > retries is set to 4 so I see 4 responses (at 2 sec intervals) as the > > client is not currently ACKing. All is fine up to that point. > > > > At that point, OpenSM sees a large receive which appears to be that send > > timing out (nothing was sent nor observed on the IB wire). > > > > Could a timed out RMPP send end up as a receive somehow ? > > On the side that sent the MAD? The side that sent the RMPP MAD response (e.g. OpenSM). > That should be no. That's what I thought. I'm not sure where the problem is but will start to try to narrow it down. > On the remote side, a send can timeout, but still be received, since the ACK > is unreliable. But I don't think that this is the situation that you're > describing. No. The remote side appears to be behaving as I would expect (at least as far as I have gone). -- Hal From libor at topspin.com Tue Jun 28 10:55:25 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 28 Jun 2005 10:55:25 -0700 Subject: [openib-general] Re: [PATCH (repost)] error values cleanup In-Reply-To: <20050619073336.GN20041@mellanox.co.il>; from mst@mellanox.co.il on Sun, Jun 19, 2005 at 10:33:36AM +0300 References: <20050615094747.GX21081@mellanox.co.il> <20050619073336.GN20041@mellanox.co.il> Message-ID: <20050628105525.D26240@topspin.com> On Sun, Jun 19, 2005 at 10:33:36AM +0300, Michael S. Tsirkin wrote: > Here's an update to the latest bits: Libor, does this make sense? > > Make sdp_cm_actv_error and sdp_conn_inet_error accept standard, negative > error values (esp. for sdp_cm_actv_error it was sufficiently confusing to > be passing a positive value to warrant a comment). > There are still a couple of functions which return positive error > values but these are static and these positive errors never propagate > outside a single .c file. > > As a side effect, "(0 - result)" in a couple of places gets replaced with > "result" which is also good. The patch does a couple of cosmetic cleanups > as well. Michael, I expanded your patch to remove SDP_CONN_{GET,SET}_ERR, conn->error, and sdp_conn_error() and just use the already available sk->sk_err and sock_error() instead. -Libor Index: sdp_inet.c =================================================================== --- sdp_inet.c (revision 2731) +++ sdp_inet.c (working copy) @@ -214,10 +214,12 @@ */ static int sdp_inet_disconnect(struct sdp_opt *conn) { + struct sock *sk; int result = 0; /* * close buffered data transmission space */ + sk = sk_sdp(conn); conn->send_buf = 0; /* * Generate a Disconnect message, and mark self as disconnecting. @@ -230,7 +232,7 @@ * completions needs to complete the closing. */ SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); - SDP_CONN_SET_ERR(conn, ECONNRESET); + sk->sk_err = ECONNRESET; break; case SDP_CONN_ST_REQ_RECV: case SDP_CONN_ST_REP_RECV: @@ -279,7 +281,7 @@ /* * abortive close. */ - sdp_conn_inet_error(conn, ECONNRESET); + sdp_conn_inet_error(conn, -ECONNRESET); (void)ib_send_cm_dreq(conn->cm_id, NULL, 0); return result; @@ -563,7 +565,7 @@ inet_sk(sk)->sport = htons(conn->src_port); } - SDP_CONN_SET_ERR(conn, 0); + sk->sk_err = 0; sock->state = SS_CONNECTING; @@ -658,7 +660,7 @@ break; case SDP_CONN_ST_CLOSED: case SDP_CONN_ST_ERROR: - result = sdp_conn_error(conn) ? : -ECONNABORTED; + result = sock_error(sk) ? : -ECONNABORTED; sock->state = SS_UNCONNECTED; break; default: @@ -1266,7 +1268,7 @@ * completions needs to complete the closing. */ SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); - SDP_CONN_SET_ERR(conn, ECONNRESET); + sock->sk->sk_err = ECONNRESET; break; case SDP_CONN_ST_LISTEN: if (flag & RCV_SHUTDOWN) { Index: sdp_send.c =================================================================== --- sdp_send.c (revision 2731) +++ sdp_send.c (working copy) @@ -2022,8 +2022,8 @@ timeout = sock_sndtimeo(sk, (MSG_DONTWAIT & msg->msg_flags)); - if (SDP_CONN_GET_ERR(conn)) { - result = (copied > 0) ? 0 : sdp_conn_error(conn); + if (sk->sk_err) { + result = (copied > 0) ? 0 : sock_error(sk); break; } Index: sdp_actv.c =================================================================== --- sdp_actv.c (revision 2731) +++ sdp_actv.c (working copy) @@ -43,8 +43,6 @@ int result; struct sock *sk; /* - * error value is positive error. - * * Handle errors within active connections stream. * First generate appropriate response, REJ, DREQ or nothing. * Second the socket must be notified of the error. @@ -90,15 +88,16 @@ break; } - SDP_CONN_SET_ERR(conn, error); conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; sk = sk_sdp(conn); + sk->sk_err = -error; + if (sk->sk_socket) sk->sk_socket->state = SS_UNCONNECTED; - sdp_iocb_q_cancel_all(conn, (0 - error)); + sdp_iocb_q_cancel_all(conn, error); sdp_inet_wake_error(sk); } @@ -322,7 +321,7 @@ return 0; error: - sdp_cm_actv_error(conn, (0 - result)); + sdp_cm_actv_error(conn, result); if (conn->state == SDP_CONN_ST_CLOSED) { conn->cm_id = NULL; @@ -504,7 +503,7 @@ goto done; failed: - sdp_cm_actv_error(conn, (0 - status)); + sdp_cm_actv_error(conn, status); done: sdp_conn_unlock(conn); sdp_conn_put(conn); /* address resolution reference */ Index: sdp_conn.c =================================================================== --- sdp_conn.c (revision 2731) +++ sdp_conn.c (working copy) @@ -87,22 +87,23 @@ (void)sdp_inet_accept_q_remove(conn); SDP_CONN_ST_SET(conn, SDP_CONN_ST_ERROR); - SDP_CONN_SET_ERR(conn, error); conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; sk = sk_sdp(conn); + sk->sk_err = -error; + if (sk->sk_socket) sk->sk_socket->state = SS_UNCONNECTED; - sdp_iocb_q_cancel_all(conn, -error); + sdp_iocb_q_cancel_all(conn, error); sdp_inet_wake_error(sk); } void sdp_conn_abort(struct sdp_opt *conn) { int result; - int error = ECONNRESET; + int error = -ECONNRESET; sdp_dbg_ctrl(conn, "Abort send. src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -137,7 +138,7 @@ conn->flags &= ~SDP_CONN_F_DIS_PEND; case SDP_CONN_ST_DIS_RECV_1: - error = EPIPE; + error = -EPIPE; case SDP_CONN_ST_ESTABLISHED: /* * abortive close. @@ -154,7 +155,7 @@ * outstanding CM request. Mark it in error, and CM * completion needs to complete the closing. */ - error = ECONNREFUSED; + error = -ECONNREFUSED; break; case SDP_CONN_ST_ERROR: case SDP_CONN_ST_CLOSED: Index: sdp_recv.c =================================================================== --- sdp_recv.c (revision 2731) +++ sdp_recv.c (working copy) @@ -1278,8 +1278,8 @@ * check connection errors, and then wait for more data. * check status. POSIX 1003.1g order. */ - if (SDP_CONN_GET_ERR(conn)) { - result = (copied > 0) ? 0 : sdp_conn_error(conn); + if (sk->sk_err) { + result = (copied > 0) ? 0 : sock_error(sk); break; } Index: sdp_conn.h =================================================================== --- sdp_conn.h (revision 2731) +++ sdp_conn.h (working copy) @@ -119,11 +119,6 @@ return (struct sock *)conn; } -#define SDP_CONN_SET_ERR(conn, val) \ - ((conn)->error = sk_sdp(conn)->sk_err = (val)) -#define SDP_CONN_GET_ERR(conn) \ - ((conn)->error) - /* * state transition information recording */ @@ -264,8 +259,6 @@ */ u32 nond_recv; /* non discarded buffers received. */ u32 nond_send; /* non discarded buffers sent */ - - s32 error; /* error value on connection. */ /* * OOB/URG data transfer. */ @@ -491,22 +484,6 @@ sdp_conn_destruct(conn); } -/* - * sdp_conn_error - get the connections error value destructively - */ -static inline int sdp_conn_error(struct sdp_opt *conn) -{ - /* - * The connection error parameter is set and read under the connection - * lock, however the linux socket error, needs to be xchg'd since the - * SO_ERROR getsockopt happens outside of the connection lock. - */ - int error = xchg(&sk_sdp(conn)->sk_err, 0); - SDP_CONN_SET_ERR(conn, 0); - - return -error; -} - static inline void *hashent_arg(s32 hashent) { return (void *)(unsigned long)hashent; Index: sdp_pass.c =================================================================== --- sdp_pass.c (revision 2731) +++ sdp_pass.c (working copy) @@ -42,7 +42,7 @@ { struct ib_qp_attr *qp_attr; int attr_mask = 0; - int result = 0; + int result; sdp_dbg_ctrl(conn, "Passive Establish src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -63,27 +63,26 @@ result = ib_cm_init_qp_attr(conn->cm_id, qp_attr, &attr_mask); if (result) { - sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS", - result); + sdp_dbg_warn(conn, "Error <%d> QP attributes for RTS", result); goto error; } result = ib_modify_qp(conn->qp, qp_attr, attr_mask); if (result) { - sdp_dbg_warn(conn, "Error <%d> modifiying QP to RTS", result); + sdp_dbg_warn(conn, "Error <%d> modifying QP to RTS", result); goto error; } conn->send_buf = SDP_INET_SEND_SIZE; result = sdp_send_flush(conn); - if (0 > result) { + if (result < 0) { sdp_dbg_warn(conn, "Error <%d> flushing sends.", result); goto error; } result = sdp_recv_flush(conn); - if (0 > result) { + if (result < 0) { sdp_dbg_warn(conn, "Error <%d> flushing receives.", result); goto error; } @@ -95,7 +94,7 @@ error: kfree(qp_attr); done: - sdp_conn_inet_error(conn, -result); + sdp_conn_inet_error(conn, result); return result; } /* Index: sdp_event.c =================================================================== --- sdp_event.c (revision 2731) +++ sdp_event.c (working copy) @@ -185,7 +185,7 @@ static void sdp_cm_to_error(struct sdp_opt *conn) { - sdp_conn_inet_error(conn, ECONNRESET); + sdp_conn_inet_error(conn, -ECONNRESET); conn->cm_id = NULL; sdp_conn_put(conn); /* CM reference */ } @@ -203,11 +203,11 @@ */ switch (conn->state) { case SDP_CONN_ST_REQ_SENT: - sdp_cm_actv_error(conn, ECONNREFUSED); + sdp_cm_actv_error(conn, -ECONNREFUSED); break; case SDP_CONN_ST_REQ_RECV: case SDP_CONN_ST_ESTABLISHED: - sdp_conn_inet_error(conn, ECONNREFUSED); + sdp_conn_inet_error(conn, -ECONNREFUSED); break; case SDP_CONN_ST_TIME_WAIT_1: sdp_dbg_warn(conn, "Unexpected connection state"); From rolandd at cisco.com Tue Jun 28 11:05:20 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 11:05:20 -0700 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <20050628092521.GG25304@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 28 Jun 2005 12:25:22 +0300") References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> Message-ID: <52psu6uzkv.fsf@topspin.com> Michael> Of course. Closing the file should be handled separately Michael> from regular deregistration. A new patch (below) fixes Michael> that. Thinking about this some more, I really don't like having such a large hole where vm_locked accounting can be wrong. I would prefer to stick with the current code, which has at worst a short window where vm_locked is too high. I don't like making it possible for a buggy userspace process to leak vm_locked (even though it is technically the process's own fault). - R. From jlentini at netapp.com Tue Jun 28 11:11:56 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 14:11:56 -0400 (EDT) Subject: [openib-general] Proposal for mapping DAT_RETURN values to ERRNO values Message-ID: The attached PDF, prepared by Itamar Rabenstein of Mellanox, contains a proposed mapping from DAT_RETURN values to ERRNO values. Please send me suggestions for alternative mappings by the end of the week. james -------------- next part -------------- A non-text attachment was scrubbed... Name: errno.pdf Type: application/pdf Size: 18436 bytes Desc: errno.pdf URL: From halr at voltaire.com Tue Jun 28 11:07:14 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 14:07:14 -0400 Subject: [openib-general] Re: RMPP In-Reply-To: <1119980903.4848.2518.camel@hal.voltaire.com> References: <1119980090.4848.2512.camel@hal.voltaire.com> <42C18C99.6060102@ichips.intel.com> <1119980903.4848.2518.camel@hal.voltaire.com> Message-ID: <1119982033.4848.2522.camel@hal.voltaire.com> On Tue, 2005-06-28 at 13:48, Hal Rosenstock wrote: > On Tue, 2005-06-28 at 13:44, Sean Hefty wrote: > > Hal Rosenstock wrote: > > > Hi Sean, > > > > > > I'm in the process of enabling the receive side RMPP from user space and > > > this is what I'm seeing in terms of RMPP right now. I have a question > > > about the OpenSM side. > > > > > > SA client OpenSM > > > SA GetTable (PortInfoRecord) --> > > > <-- SA GetTableResp (PortInfoRecord) > > > RMPP active, first > > > payload length 0x44C > > > > > > retries is set to 4 so I see 4 responses (at 2 sec intervals) as the > > > client is not currently ACKing. All is fine up to that point. > > > > > > At that point, OpenSM sees a large receive which appears to be that send > > > timing out (nothing was sent nor observed on the IB wire). > > > > > > Could a timed out RMPP send end up as a receive somehow ? > > > > On the side that sent the MAD? > > The side that sent the RMPP MAD response (e.g. OpenSM). > > > That should be no. > > That's what I thought. I'm not sure where the problem is but will start > to try to narrow it down. I do get EINVAL from user_mad.c::ib_umad_read as follows: if (count < packet->length + sizeof (struct ib_user_mad)) ret = -EINVAL; as the packet->length is larger than a single MAD (and looks like the user MAD that was sent by OpenSM). -- Hal From libor at topspin.com Tue Jun 28 11:19:13 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 28 Jun 2005 11:19:13 -0700 Subject: [openib-general] Re: SDP: still getting sk_alloc() panic, any ideas? In-Reply-To: <20050627160611.C26240@topspin.com>; from libor@topspin.com on Mon, Jun 27, 2005 at 04:06:11PM -0700 References: <9d3b7de7050623130658aa92@mail.gmail.com> <20050627111721.A26240@topspin.com> <1119907675.6148.7.camel@duffman> <20050627160611.C26240@topspin.com> Message-ID: <20050628111913.E26240@topspin.com> On Mon, Jun 27, 2005 at 04:06:11PM -0700, Libor Michalek wrote: > On Mon, Jun 27, 2005 at 02:27:54PM -0700, Tom Duffy wrote: > > On Mon, 2005-06-27 at 11:17 -0700, Libor Michalek wrote: > > > The problem is that each call to sk_alloc() is grabbing a reference to > > > the module, but it checks to make sure that there already is at least one > > > reference, if not the top BUG is triggered. In the case of the passive > > > connection there are no other references to the module. You can see that > > > the problem goes away if you open just one socket, even if you don't > > > listen on it, and then try the failing passive connect. When a socket is > > > created it actually grabs two references to the module, one at the sock > > > level and one at the sk level. The first reference at the sock level does > > > not trigger the BUG since it's through another code path. (try_module_get > > > vs. __module_get) This is why we only hit this during passive connect > > > to a system that has no active SDP sockets. > > > > Hrm. That seems ugly. How about a patch to upstream changing > > sk_alloc() to use try_module_get(). > > I'm thinking the listen_lookup needs to be moved earlier in the > req_handler ahead of the sk_alloc, since it makes no sense to do > the alloc if we are not going to queue the new incomming connection, > since it just leads to a destroy in the same function. Here's a patch to reorder listen_lookup() in req_handler() and a few other fixups, such as making a couple int return functions void. -Libor Index: sdp_proto.h =================================================================== --- sdp_proto.h (revision 2731) +++ sdp_proto.h (working copy) @@ -282,8 +282,8 @@ /* * port/queue managment */ -int sdp_inet_accept_q_put(struct sdp_opt *listen_conn, - struct sdp_opt *accept_conn); +void sdp_inet_accept_q_put(struct sdp_opt *listen_conn, + struct sdp_opt *accept_conn); struct sdp_opt *sdp_inet_accept_q_get(struct sdp_opt *listen_conn); @@ -299,7 +299,7 @@ int sdp_inet_port_put(struct sdp_opt *conn); -int sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child); +void sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child); /* * active connect functions Index: sdp_conn.c =================================================================== --- sdp_conn.c (revision 2732) +++ sdp_conn.c (working copy) @@ -176,16 +176,14 @@ /* * sdp_inet_accept_q_put - put a conn into a listen conn's accept Q. */ -int sdp_inet_accept_q_put(struct sdp_opt *listen_conn, - struct sdp_opt *accept_conn) +void sdp_inet_accept_q_put(struct sdp_opt *listen_conn, + struct sdp_opt *accept_conn) { struct sdp_opt *next_conn; - if (listen_conn->parent || - accept_conn->parent || - !listen_conn->accept_next || - !listen_conn->accept_prev) - return -EFAULT; + BUG_ON(listen_conn->parent); + BUG_ON(accept_conn->parent); + BUG_ON(!listen_conn->accept_next || !listen_conn->accept_prev); next_conn = listen_conn->accept_next; @@ -200,8 +198,6 @@ * up ref until we release. One ref for GW and one for INET. */ sdp_conn_hold(accept_conn); /* AcceptQueue INET reference */ - - return 0; } /* @@ -545,9 +541,8 @@ /* * sdp_inet_port_inherit - inherit a port from another socket (accept) */ -int sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child) +void sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child) { - int result; unsigned long flags; /* @@ -555,13 +550,8 @@ */ spin_lock_irqsave(&dev_root_s.bind_lock, flags); - if (child->bind_p_next || - child->src_port != parent->src_port) { - sdp_dbg_warn(child, "child already bound. <%d:%d>", - parent->src_port, child->src_port); - result = -EADDRNOTAVAIL; - goto done; - } + BUG_ON(child->bind_p_next); + BUG_ON(child->src_port != parent->src_port); /* * insert into listening list. */ @@ -572,10 +562,7 @@ if (child->bind_next) child->bind_next->bind_p_next = &child->bind_next; - result = 0; -done: spin_unlock_irqrestore(&dev_root_s.bind_lock, flags); - return result; } /* Index: sdp_pass.c =================================================================== --- sdp_pass.c (revision 2732) +++ sdp_pass.c (working copy) @@ -229,121 +229,54 @@ return result; } -static int sdp_cm_listen_lookup(struct sdp_opt *conn) +static void sdp_cm_listen_inherit(struct sdp_opt *parent, + struct sdp_opt *child) { - struct sdp_opt *listen_conn; - struct sock *listen_sk; - struct sock *sk; - int result; - /* - * match a listener with an incoming conn, and generate - * accept message and state on success. - */ - sdp_dbg_ctrl(conn, - "listen match for conn. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, - conn->dst_addr, conn->dst_port); - /* - * first find a listening connection - */ - listen_conn = sdp_inet_listen_lookup(conn->src_addr, conn->src_port); - if (!listen_conn) { - /* - * no connection, reject - */ - sdp_dbg_warn(conn, "no listener for connection. <%08x:%04x>", - conn->src_addr, conn->src_port); - result = -ECONNREFUSED; - goto lookup_err; - } + struct sock *psk; + struct sock *csk; - sdp_conn_lock(listen_conn); - /* - * check backlog - */ - listen_sk = sk_sdp(listen_conn); - sk = sk_sdp(conn); + sdp_inet_port_inherit(parent, child); - if (listen_conn->backlog_cnt > listen_conn->backlog_max) { - sdp_dbg_warn(listen_conn, - "Listen backlog <%d> too big to accept new conn", - listen_conn->backlog_cnt); - result = -ECONNREFUSED; - goto locked_err; - } - - result = sdp_inet_port_inherit(listen_conn, conn); - if (result < 0) { - sdp_dbg_warn(listen_conn, "Error <%d> listen port inherit.", - result); - result = -EFAULT; - goto locked_err; - } + psk = sk_sdp(parent); + csk = sk_sdp(child); /* * insert accept socket into listen sockets list. * TODO: needs to be a FIFO not a LIFO, as is now. */ - inet_sk(sk)->num = conn->src_port; - inet_sk(sk)->sport = htons(conn->src_port); - inet_sk(sk)->rcv_saddr = htonl(conn->src_addr); - inet_sk(sk)->saddr = htonl(conn->src_addr); - inet_sk(sk)->daddr = htonl(conn->dst_addr); - inet_sk(sk)->dport = htons(conn->dst_port); + inet_sk(csk)->num = child->src_port; + inet_sk(csk)->sport = htons(child->src_port); + inet_sk(csk)->rcv_saddr = htonl(child->src_addr); + inet_sk(csk)->saddr = htonl(child->src_addr); + inet_sk(csk)->daddr = htonl(child->dst_addr); + inet_sk(csk)->dport = htons(child->dst_port); /* * relevant options, and others... TCP does a full copy, I'd like to * know what I'm inheriting. */ - sk->sk_lingertime = listen_sk->sk_lingertime; - sk->sk_rcvlowat = listen_sk->sk_rcvlowat; - if (sock_flag(listen_sk, SOCK_DBG)) - sock_set_flag(sk, SOCK_DBG); - if (sock_flag(listen_sk, SOCK_LOCALROUTE)) - sock_set_flag(sk, SOCK_LOCALROUTE); - sk->sk_sndbuf = listen_sk->sk_sndbuf; - sk->sk_rcvbuf = listen_sk->sk_rcvbuf; - sk->sk_no_check = listen_sk->sk_no_check; - sk->sk_priority = listen_sk->sk_priority; - if (sock_flag(listen_sk, SOCK_RCVTSTAMP)) - sock_set_flag(sk, SOCK_RCVTSTAMP); - sk->sk_rcvtimeo = listen_sk->sk_rcvtimeo; - sk->sk_sndtimeo = listen_sk->sk_sndtimeo; - sk->sk_reuse = listen_sk->sk_reuse; - sk->sk_bound_dev_if = listen_sk->sk_bound_dev_if; - sk->sk_userlocks |= (listen_sk->sk_userlocks & ~SOCK_BINDPORT_LOCK); - sk->sk_flags = ((SOCK_URGINLINE|SOCK_LINGER|SOCK_BROADCAST) & - listen_sk->sk_flags); + csk->sk_lingertime = psk->sk_lingertime; + csk->sk_rcvlowat = psk->sk_rcvlowat; + csk->sk_sndbuf = psk->sk_sndbuf; + csk->sk_rcvbuf = psk->sk_rcvbuf; + csk->sk_no_check = psk->sk_no_check; + csk->sk_priority = psk->sk_priority; + csk->sk_rcvtimeo = psk->sk_rcvtimeo; + csk->sk_sndtimeo = psk->sk_sndtimeo; + csk->sk_reuse = psk->sk_reuse; + csk->sk_bound_dev_if = psk->sk_bound_dev_if; + csk->sk_userlocks |= (psk->sk_userlocks & ~SOCK_BINDPORT_LOCK); + csk->sk_flags = ((SOCK_URGINLINE|SOCK_LINGER|SOCK_BROADCAST) & + psk->sk_flags); - conn->src_zthresh = listen_conn->src_zthresh; - conn->snk_zthresh = listen_conn->snk_zthresh; - conn->nodelay = listen_conn->nodelay; - /* - * initiate a CM response message. - */ - result = sdp_cm_accept(conn); - if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> CM connect accept", result); - goto locked_err; - } - /* - * place connection into the listen connections accept queue. - */ - result = sdp_inet_accept_q_put(listen_conn, conn); - if (result < 0) { - sdp_dbg_warn(conn, - "Error <%d> adding socket to accept queue", - result); - result = -EFAULT; - goto locked_err; - } + if (sock_flag(psk, SOCK_DBG)) + sock_set_flag(csk, SOCK_DBG); + if (sock_flag(psk, SOCK_LOCALROUTE)) + sock_set_flag(csk, SOCK_LOCALROUTE); + if (sock_flag(psk, SOCK_RCVTSTAMP)) + sock_set_flag(csk, SOCK_RCVTSTAMP); - sdp_inet_wake_recv(listen_sk, 0); - - result = 0; -locked_err: - sdp_conn_unlock(listen_conn); - sdp_conn_put(listen_conn); /* ListenLookup reference. */ -lookup_err: - return result; + child->src_zthresh = parent->src_zthresh; + child->snk_zthresh = parent->snk_zthresh; + child->nodelay = parent->nodelay; } static int sdp_cm_hello_check(struct sdp_msg_hello *msg_hello) @@ -415,8 +348,11 @@ int sdp_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) { struct sdp_msg_hello *msg_hello = event->private_data; + struct sdp_opt *listen_conn; struct sdp_opt *conn; int result; + u16 port; + u32 addr; sdp_dbg_ctrl(NULL, "CM REQ. comm <%08x> SID <%016llx> ca <%s> port <%d>", @@ -430,7 +366,36 @@ if (result < 0) { sdp_dbg_warn(NULL, "Error <%d> validating hello msg. <%08x>", result, cm_id->local_id); + goto empty; + } + + port = SDP_SID_TO_PORT(be64_to_cpu(cm_id->service_id)); + addr = msg_hello->hh.dst.ipv4.addr; + /* + * first find a listening connection, and check backlog + */ + result = -ECONNREFUSED; + + listen_conn = sdp_inet_listen_lookup(addr, port); + if (!listen_conn) { + /* + * no connection, reject + */ + sdp_dbg_ctrl(NULL, "no listener for connection. <%08x:%04x>", + addr, port); + goto empty; + } + + sdp_conn_lock(listen_conn); + + if (listen_conn->state != SDP_CONN_ST_LISTEN) goto done; + + if (listen_conn->backlog_cnt > listen_conn->backlog_max) { + sdp_dbg_ctrl(listen_conn, + "Listen backlog <%d> too big to accept new conn", + listen_conn->backlog_cnt); + goto done; } /* * Create a connection for this request. @@ -451,8 +416,8 @@ */ SDP_CONN_ST_SET(conn, SDP_CONN_ST_REQ_RECV); - conn->src_addr = msg_hello->hh.dst.ipv4.addr; - conn->src_port = SDP_SID_TO_PORT(be64_to_cpu(cm_id->service_id)); + conn->src_addr = addr; + conn->src_port = port; conn->dst_addr = msg_hello->hh.src.ipv4.addr; conn->dst_port = msg_hello->hh.port; @@ -473,7 +438,7 @@ &event->param.req_rcvd.remote_ca_guid, sizeof(conn->d_gid)); /* - * update CM context to refer to the connection. + * update CM context to refer to the connection, before alloc_ib() */ conn->cm_id = cm_id; conn->cm_id->context = hashent_arg(conn->hashent); @@ -490,7 +455,7 @@ goto error; } /* - * Save connect request info for QP modify. + * Save connect request info for QP modify in cm_accept(). */ conn->d_lid = event->param.req_rcvd.primary_path->dlid; conn->s_lid = event->param.req_rcvd.primary_path->slid; @@ -498,19 +463,29 @@ conn->path_mtu = event->param.req_rcvd.primary_path->mtu; /* - * Find a matching listening socket, and insert new connection - * into listeners accept queue. + * inherit listener properties */ - result = sdp_cm_listen_lookup(conn); + sdp_cm_listen_inherit(listen_conn, conn); + /* + * initiate a CM response message. + */ + result = sdp_cm_accept(conn); if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> matching listen socket queue", - result); + sdp_dbg_warn(conn, "Error <%d> CM connect accept", result); goto error; } /* + * place connection into the listen connections accept queue. + */ + sdp_inet_accept_q_put(listen_conn, conn); + sdp_inet_wake_recv(sk_sdp(listen_conn), 0); + /* * unlock */ sdp_conn_unlock(conn); + sdp_conn_unlock(listen_conn); + sdp_conn_put(listen_conn); /* ListenLookup reference. */ + return 0; error: SDP_CONN_ST_SET(conn, SDP_CONN_ST_CLOSED); @@ -518,6 +493,9 @@ sdp_conn_unlock(conn); sdp_conn_put(conn); /* CM reference */ done: + sdp_conn_unlock(listen_conn); + sdp_conn_put(listen_conn); /* ListenLookup reference. */ +empty: (void)ib_send_cm_rej(cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); From rolandd at cisco.com Tue Jun 28 11:19:58 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 11:19:58 -0700 Subject: [openib-general] [PATCH] mthca: report board id in sysfs In-Reply-To: <20050627150308.GD25304@mellanox.co.il> (Michael S. Tsirkin's message of "Mon, 27 Jun 2005 18:03:08 +0300") References: <20050627150308.GD25304@mellanox.co.il> Message-ID: <52ekamuywh.fsf@topspin.com> This looks good but I'm going to queue it up for a while. I want to get userspace verbs merged upstream soon so I'd rather keep changes to the core and mthca to a minimum for the next few days. - R. From mst at mellanox.co.il Tue Jun 28 11:50:22 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 21:50:22 +0300 Subject: [openib-general] [PATCH] uverbs: deregister memory before ib_umem_release Message-ID: <20050628185022.GA6442@mellanox.co.il> While all qps and cqs are closed first, so HCA doesnt have a reason to write to the mr, I think its still cleaner to first deregister the mr, and then release the memory. It might become more important if/when shared mrs are implemented. --- Deregister memory before ib_umem_release, to make sure HCA wont be writing there. Signed-off-by: Michael S. Tsirkin Index: core/uverbs_main.c =================================================================== --- core/uverbs_main.c (revision 2732) +++ core/uverbs_main.c (working copy) @@ -132,11 +132,10 @@ static int ib_dealloc_ucontext(struct ib struct ib_mr *mr = idr_find(&ib_uverbs_mr_idr, uobj->id); struct ib_umem_object *memobj; - memobj = container_of(uobj, struct ib_umem_object, uobject); - ib_umem_release(mr->device, &memobj->umem); - idr_remove(&ib_uverbs_mr_idr, uobj->id); ib_dereg_mr(mr); + memobj = container_of(uobj, struct ib_umem_object, uobject); + ib_umem_release(mr->device, &memobj->umem); list_del(&uobj->list); kfree(memobj); } -- MST From halr at voltaire.com Tue Jun 28 11:46:39 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 14:46:39 -0400 Subject: [openib-general] Re: RMPP In-Reply-To: <1119982033.4848.2522.camel@hal.voltaire.com> References: <1119980090.4848.2512.camel@hal.voltaire.com> <42C18C99.6060102@ichips.intel.com> <1119980903.4848.2518.camel@hal.voltaire.com> <1119982033.4848.2522.camel@hal.voltaire.com> Message-ID: <1119984398.4848.2549.camel@hal.voltaire.com> On Tue, 2005-06-28 at 14:07, Hal Rosenstock wrote: > On Tue, 2005-06-28 at 13:48, Hal Rosenstock wrote: > > On Tue, 2005-06-28 at 13:44, Sean Hefty wrote: > > > Hal Rosenstock wrote: > > > > Hi Sean, > > > > > > > > I'm in the process of enabling the receive side RMPP from user space and > > > > this is what I'm seeing in terms of RMPP right now. I have a question > > > > about the OpenSM side. > > > > > > > > SA client OpenSM > > > > SA GetTable (PortInfoRecord) --> > > > > <-- SA GetTableResp (PortInfoRecord) > > > > RMPP active, first > > > > payload length 0x44C > > > > > > > > retries is set to 4 so I see 4 responses (at 2 sec intervals) as the > > > > client is not currently ACKing. All is fine up to that point. > > > > > > > > At that point, OpenSM sees a large receive which appears to be that send > > > > timing out (nothing was sent nor observed on the IB wire). > > > > > > > > Could a timed out RMPP send end up as a receive somehow ? > > > > > > On the side that sent the MAD? > > > > The side that sent the RMPP MAD response (e.g. OpenSM). > > > > > That should be no. > > > > That's what I thought. I'm not sure where the problem is but will start > > to try to narrow it down. > > I do get EINVAL from user_mad.c::ib_umad_read as follows: > > if (count < packet->length + sizeof (struct ib_user_mad)) > ret = -EINVAL; > > as the packet->length is larger than a single MAD (and looks like the > user MAD that was sent by OpenSM). I think I see what is going on here... In user_mad.c::send_handler if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) { packet->mad.hdr.status = ETIMEDOUT; if (!queue_packet(file, agent, packet)) return; } That is what is causing the problem. I think the send side queues the packet on a timeout and simulates a receive so that a transaction can be terminated. RMPP sends appear to be a little different in that even non transactions get timeouts. -- Hal From mst at mellanox.co.il Tue Jun 28 11:53:26 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 21:53:26 +0300 Subject: [openib-general] Re: [PATCH] mthca: report board id in sysfs In-Reply-To: <52ekamuywh.fsf@topspin.com> References: <20050627150308.GD25304@mellanox.co.il> <52ekamuywh.fsf@topspin.com> Message-ID: <20050628185326.GB6442@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] mthca: report board id in sysfs > > This looks good but I'm going to queue it up for a while. I want to > get userspace verbs merged upstream soon so I'd rather keep changes to > the core and mthca to a minimum for the next few days. > > - R. > Apropos that, do you have the kernel patch implementing something like mprotect(...., PROT_DONTCOPY) (needed for fork support)? -- MST From rolandd at cisco.com Tue Jun 28 12:04:02 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 12:04:02 -0700 Subject: [openib-general] [PATCH] uverbs: deregister memory before ib_umem_release In-Reply-To: <20050628185022.GA6442@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 28 Jun 2005 21:50:22 +0300") References: <20050628185022.GA6442@mellanox.co.il> Message-ID: <523br2uwv1.fsf@topspin.com> Yes, good catch. Applied, thanks. - R. From rolandd at cisco.com Tue Jun 28 12:05:28 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 12:05:28 -0700 Subject: [openib-general] Re: [PATCH] mthca: report board id in sysfs In-Reply-To: <20050628185326.GB6442@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 28 Jun 2005 21:53:26 +0300") References: <20050627150308.GD25304@mellanox.co.il> <52ekamuywh.fsf@topspin.com> <20050628185326.GB6442@mellanox.co.il> Message-ID: <52y88uti87.fsf@topspin.com> Michael> Apropos that, do you have the kernel patch implementing Michael> something like mprotect(...., PROT_DONTCOPY) (needed for Michael> fork support)? Not yet -- I am deferring that a little. - R. From mst at mellanox.co.il Tue Jun 28 12:17:10 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 22:17:10 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <52psu6uzkv.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> <52psu6uzkv.fsf@topspin.com> Message-ID: <20050628191710.GC6442@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] process locked in D state. > > I would prefer to stick > with the current code, which has at worst a short window where > vm_locked is too high. Right, but it seems this introduces the race for apps doing deregister_mr properly, too, which used to work fine. > I don't like making it possible for a buggy > userspace process to leak vm_locked (even though it is technically the > process's own fault). > > - R. > Right, but at least lets not introduce the hole for the good case. When I call deregister_mr I have the right to expect that I can lock the memory immediately after that. So deregister mr should always wait till it has the semaphore. In the case of close, its the buggy app that suffers, and I'm ok with handling it in a work queue since you think its worth it - but lets not try to optimise that case, it should be ok to always use the workqueue. --- Split the code handling the well-behaved and the misbehaved app, to avoid races for the good case, and make behaviour more consistent for the bad one. Signed-off-by: Michael S. Tsirkin Index: core/uverbs_mem.c =================================================================== --- core/uverbs_mem.c (revision 2732) +++ core/uverbs_mem.c (working copy) @@ -185,35 +185,42 @@ void ib_umem_release(struct ib_device *d __ib_umem_release(dev, umem, 1); mm = get_task_mm(current); - if (mm) { - /* - * We may be called with the mm's mmap_sem already - * held. This can happen when a userspace munmap() is - * the call that drops the last reference to our file - * and calls our release method. If there are memory - * regions to destroy, we'll end up here and not be - * able to take the mmap_sem. - * - * To handle this, we try to grab the mmap_sem, and if - * we can't get it immediately, we defer the - * accounting to the system workqueue. - */ - if (down_write_trylock(&mm->mmap_sem)) { - mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; - up_write(&mm->mmap_sem); - mmput(mm); - } else { - struct ib_umem_account_work *work; - - work = kmalloc(sizeof *work, GFP_KERNEL); - if (!work) - return; - - INIT_WORK(&work->work, ib_umem_account, work); - work->mm = mm; - work->diff = PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; - - schedule_work(&work->work); - } - } + if (!mm) + return; + + down_write(&mm->mmap_sem); + mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; + up_write(&mm->mmap_sem); + mmput(mm); +} + +void ib_umem_put(struct ib_device *dev, struct ib_umem *umem) +{ + struct ib_umem_account_work *work; + struct mm_struct *mm; + + __ib_umem_release(dev, umem, 1); + + mm = get_task_mm(current); + if (!mm) + return; + + /* + * We may be called with the mm's mmap_sem already + * held. This can happen when a userspace munmap() is + * the call that drops the last reference to our file + * and calls our release method. If there are memory + * regions to destroy, we'll end up here and not be + * able to take the mmap_sem. + */ + + work = kmalloc(sizeof *work, GFP_KERNEL); + if (!work) + return; + + INIT_WORK(&work->work, ib_umem_account, work); + work->mm = mm; + work->diff = PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; + + schedule_work(&work->work); } Index: core/uverbs_main.c =================================================================== --- core/uverbs_main.c (revision 2732) +++ core/uverbs_main.c (working copy) @@ -133,7 +133,7 @@ static int ib_dealloc_ucontext(struct ib struct ib_umem_object *memobj; memobj = container_of(uobj, struct ib_umem_object, uobject); - ib_umem_release(mr->device, &memobj->umem); + ib_umem_put(mr->device, &memobj->umem); idr_remove(&ib_uverbs_mr_idr, uobj->id); ib_dereg_mr(mr); Index: core/uverbs.h =================================================================== --- core/uverbs.h (revision 2732) +++ core/uverbs.h (working copy) @@ -103,6 +103,7 @@ void ib_uverbs_qp_event_handler(struct i int ib_umem_get(struct ib_device *dev, struct ib_umem *mem, void *addr, size_t size, int write); +void ib_umem_put(struct ib_device *dev, struct ib_umem *umem); void ib_umem_release(struct ib_device *dev, struct ib_umem *umem); #define IB_UVERBS_DECLARE_CMD(name) \ -- MST From halr at voltaire.com Tue Jun 28 12:13:00 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 15:13:00 -0400 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response Message-ID: <1119985980.4848.2560.camel@hal.voltaire.com> user_mad: In send_handler, only queue_packet on timeout if not response Signed-off-by: Hal Rosenstock Index: user_mad.c =================================================================== --- user_mad.c (revision 2731) +++ user_mad.c (working copy) @@ -146,12 +146,14 @@ ib_free_send_mad(packet->msg); if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) { - packet->mad.hdr.status = ETIMEDOUT; - - if (!queue_packet(file, agent, packet)) - return; - } - + /* Only if not response */ + if (!((packet->msg->mad->mad_hdr.method == IB_MGMT_METHOD_TRAP_REPRESS) || + (packet->msg->mad->mad_hdr.method & IB_MGMT_METHOD_RESP))) { + packet->mad.hdr.status = ETIMEDOUT; + if (!queue_packet(file, agent, packet)) + return; + } + } kfree(packet); } From jlentini at netapp.com Tue Jun 28 12:24:35 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 15:24:35 -0400 (EDT) Subject: [openib-general] IP addressing on InfiniBand networks Message-ID: I'd like to summarize the discussion we've been having around addressing and start a new email thread with a more appropriate title. First off, here is there requirement we are trying to satisfy: kDAPL consumers use an Internet Protocol (IP) addresses to identify remote nodes in an interoperable way. On the active side of a connection, an InfiniBand kDAPL provider must determine the IP address corresponding IB address (see the dat_ep_connect() function). On the passive side of a connection, a InfiniBand kDAPL provider must determine a source IB address for an InfiniBand connection request. This information can be obtain by a kDAPL consumer either in the DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or dat_cr_query()'s dat_cr_param structure. By interoperable, we mean that the solution must not introduce a non-standard protocol or force ULPs using kDAPL to perform special operations when using an InfiniBand network. The first sentence of this requirement is the most important. This is to best support the existing practice of ULPs that use IP addresses: NFS, iSCSI, and and sockets applications. Administrators expect to be able to configure systems using IP addresses. Note that this feature will be used. There has been considerable discussion on how NFS-RDMA software would make use of this feature. Other ULPs may use this feature as well. Of the two mappings, the second (IB address to IP address) has proven the most difficult to implement. Here are the different implementations that have been proposed and a brief critique of each: + CM Private Data The active side of an IB connection could place its source IP address in the CM's private data. The passive side would retrieve the source IP from this location. Analysis: This approach introduces a new protocol that is not hidden from the ULP. The problem becomes where in the software stack this would be implemented. If it was implemented in the DAT provider, kDAPL consumers would not be able to communicate fully with non-kDAPL consumers (the non-kDAPL consumers would not be expecting the IP address and thus interpret the private data incorrectly). If this were implemented in a ULP, the ULP would not be able to use the same code for both an IB interface and an RNIC interface. The security of this is very week. An end node could easily present a false IP address. + Address Translation Service (ATS) Each IB node places a record containing its IP address and IB address in the SA database. kDAPL consults these records to map between addresses. Analysis: This requires all IB nodes to implement a new protocol/adhere to a new convention. The advantage is that ULPs do not need to be aware of it. Since a GID can have multiple IP address, there may be multiple records with the same GID. The passive side would need a convention for assuming which of these matches to the active side's IP. The security of this solution is also fairly weak. The end nodes must be trusted to place valid records in the SA. Since the configuration of IP addresses will occur on the end nodes, it would be difficult for the SA to validate the records. However, the SA is a central part of the network, so it may be possible to add additional security features if needed. + IPoIB IPoIB encapsulates IP packets in InfiniBand messages. There have been proposals to use the address resolution mechanisms in IPoIB to implement these features. IPv4 subnets use ARP and IPv6 subnets use Neighbor Discovery. Analysis: IPoIB is not free. All nodes would be need to implement it for this to work. The IB address -> IP address mapping on the passive side is problematic. If a reverse lookup were available, IPoIB would require both a GID and QP number as input. The passive side would know the GID but the QP number. Further more, reverse lookup is not well supported. On IPv4 subnets, RARP is quickly becoming (already?) obsolete. Neighbor Discovery doesn't support reverse lookup at all. [RFC 2461] In addition to all this, IPoIB restricts an IP subnet to the same scope as an IB subnet. If a kDAPL consumer desired to communicate between IB subnet's, IPoIB may not be sufficient. + GID as an IPv6 Address See the attachment to Caitlin Bestler's email: http://openib.org/pipermail/openib-general/2005-June/008104.html Analysis: This has been the least discussed option. One issue is that GIDs may not be easy to administer. GIDs can be specific to a particular channel adapter since they can contain EUI-64 identifiers. Administrators avoid using Ethernet MAC addresses in configuration files and they should be able to avoid using adapter specific IB addresses as well. Another issue is how dynamically assigned SM GIDs would be managed. Are there other implementations? Is there a way to more tightly integrate IP addressing with InfiniBand? From rolandd at cisco.com Tue Jun 28 12:26:27 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 12:26:27 -0700 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <1119985980.4848.2560.camel@hal.voltaire.com> (Hal Rosenstock's message of "28 Jun 2005 15:13:00 -0400") References: <1119985980.4848.2560.camel@hal.voltaire.com> Message-ID: <52oe9qth98.fsf@topspin.com> Hal> user_mad: In send_handler, only queue_packet on timeout if Hal> not response Why do we want to prevent userspace from seeing these events? What's special about trap repress and resp methods? - R. From hch at lst.de Tue Jun 28 12:26:49 2005 From: hch at lst.de (Christoph Hellwig) Date: Tue, 28 Jun 2005 21:26:49 +0200 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: References: Message-ID: <20050628192649.GA9340@lst.de> On Tue, Jun 28, 2005 at 03:24:35PM -0400, James Lentini wrote: > > I'd like to summarize the discussion we've been having around > addressing and start a new email thread with a more appropriate title. > > First off, here is there requirement we are trying to satisfy: > > kDAPL consumers use an Internet Protocol (IP) addresses to > identify remote nodes in an interoperable way. I don't think that does belong into the kernel. It's fine to do that in userspace if you want. From rolandd at cisco.com Tue Jun 28 12:31:51 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 12:31:51 -0700 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <20050628191710.GC6442@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 28 Jun 2005 22:17:10 +0300") References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> <52psu6uzkv.fsf@topspin.com> <20050628191710.GC6442@mellanox.co.il> Message-ID: <52hdfith08.fsf@topspin.com> Michael> Right, but it seems this introduces the race for apps Michael> doing deregister_mr properly, too, which used to work Michael> fine. True. Michael> Right, but at least lets not introduce the hole for the Michael> good case. When I call deregister_mr I have the right to Michael> expect that I can lock the memory immediately after that. Michael> So deregister mr should always wait till it has the Michael> semaphore. Michael> In the case of close, its the buggy app that suffers, and Michael> I'm ok with handling it in a work queue since you think Michael> its worth it - but lets not try to optimise that case, it Michael> should be ok to always use the workqueue. OK, this looks fine. Applied (with slight renames of function names). Thanks, Roland From jlentini at netapp.com Tue Jun 28 12:39:59 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 15:39:59 -0400 (EDT) Subject: [openib-general] Re: [PATCH][dapl] cleanup dapl_cookie In-Reply-To: <20050627173951.GB19279@aon.at> References: <20050627173951.GB19279@aon.at> Message-ID: Hi Bernhard, The changes look fine. Why the additional copyright? I need to be able to explain it to my legal department. james On Mon, 27 Jun 2005, Bernhard Fischer wrote: > Hi James, > > untested. > > - cleanup dapl_cookie.c: remove unneeded local variables and simplify > branches to be consistent with dapl_rmr_cookie_alloc(). > > Signed-off-by: Bernhard Fischer > > thank you, > Bernhard > From halr at voltaire.com Tue Jun 28 12:35:15 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 15:35:15 -0400 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <52oe9qth98.fsf@topspin.com> References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> Message-ID: <1119987315.4848.2565.camel@hal.voltaire.com> On Tue, 2005-06-28 at 15:26, Roland Dreier wrote: > Hal> user_mad: In send_handler, only queue_packet on timeout if > Hal> not response > > Why do we want to prevent userspace from seeing these events? What's > special about trap repress and resp methods? Why does user space need to see all the timeouts ? I thought it was only request timeouts which were needed to be seen. The semantics for RMPP timeouts appear to be different. -- Hal From rolandd at cisco.com Tue Jun 28 12:41:26 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 12:41:26 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: (James Lentini's message of "Tue, 28 Jun 2005 15:24:35 -0400 (EDT)") References: Message-ID: <52br5qtgk9.fsf@topspin.com> James> First off, here [are the] requirement we are trying to satisfy: James> On the passive side of a connection, a InfiniBand kDAPL James> provider must determine a source IB address for an James> InfiniBand connection request. This information can be James> obtain by a kDAPL consumer either in the James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or James> dat_cr_query()'s dat_cr_param structure. James> By interoperable, we mean that the solution must not James> introduce a non-standard protocol or force ULPs using kDAPL James> to perform special operations when using an InfiniBand James> network. Since these two points are mutually contradictory -- the IB communication management protocol does not carry enough information for a connection request to be mapped uniquely back to a source address -- we need to figure out which one to drop. I would argue in favor of the solution selected by SDP: when defining the binding of an abstract protocol to the IB transport, put the source and destination IP addresses in the IB-specific connection setup messages. - R. From rolandd at cisco.com Tue Jun 28 12:43:42 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 12:43:42 -0700 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <1119987315.4848.2565.camel@hal.voltaire.com> (Hal Rosenstock's message of "28 Jun 2005 15:35:15 -0400") References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> <1119987315.4848.2565.camel@hal.voltaire.com> Message-ID: <5264vytggh.fsf@topspin.com> Hal> Why does user space need to see all the timeouts ? I thought Hal> it was only request timeouts which were needed to be Hal> seen. The semantics for RMPP timeouts appear to be different. Well, can the MAD core even generate timeouts for trap repress or response messages? If so, I don't see a good reason for these events to be available to the kernel but not to userspace consumers. It's much easier for consumers to ignore events they're not interested in than it is for consumers to deal with not getting events they want. - R. From halr at voltaire.com Tue Jun 28 12:51:34 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 15:51:34 -0400 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <5264vytggh.fsf@topspin.com> References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> <1119987315.4848.2565.camel@hal.voltaire.com> <5264vytggh.fsf@topspin.com> Message-ID: <1119988294.4848.2590.camel@hal.voltaire.com> On Tue, 2005-06-28 at 15:43, Roland Dreier wrote: > Hal> Why does user space need to see all the timeouts ? I thought > Hal> it was only request timeouts which were needed to be > Hal> seen. The semantics for RMPP timeouts appear to be different. > > Well, can the MAD core even generate timeouts for trap repress or > response messages? Yes (At least if the send was RMPP'd). That's what started this thread. Not sure about normal MAD send of those. Sean ? > If so, I don't see a good reason for these events > to be available to the kernel but not to userspace consumers. The semantic of timeout for RMPP send appears to me to be different. > It's much easier for consumers to ignore events they're not interested > in than it is for consumers to deal with not getting events they want. I'm not sure what a consumer would want a timeout to a non request method. Other alternatives are: Change RMPP send timeouts to be same as normal MAD sends or Not return entire send packet on timeout. Is that a requirement also ? -- Hal From rolandd at cisco.com Tue Jun 28 13:12:05 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 13:12:05 -0700 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <1119988294.4848.2590.camel@hal.voltaire.com> (Hal Rosenstock's message of "28 Jun 2005 15:51:34 -0400") References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> <1119987315.4848.2565.camel@hal.voltaire.com> <5264vytggh.fsf@topspin.com> <1119988294.4848.2590.camel@hal.voltaire.com> Message-ID: <52r7emqm0a.fsf@topspin.com> Hal> I'm not sure what a consumer would want a timeout to a non Hal> request method. I guess my view would be that if a consumer would never want a timeout, then the kernel shouldn't generate them. We be consistent between kernel space and user space though -- the umad module shouldn't be filtering out information. Hal> Other alternatives are: Change RMPP send timeouts to be same Hal> as normal MAD sends or Not return entire send packet on Hal> timeout. Is that a requirement also ? I don't think we need to return the whole packet on timed out sends now that the MAD structure has the header first. In fact it probably makes sense to leave the payload off of timeout responses. We just need a way to return the transaction_id of the timed-out send. - R. From halr at voltaire.com Tue Jun 28 13:14:33 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 16:14:33 -0400 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: References: Message-ID: <1119989672.4848.2607.camel@hal.voltaire.com> On Tue, 2005-06-28 at 15:24, James Lentini wrote: > + IPoIB > > IPoIB encapsulates IP packets in InfiniBand messages. There have been > proposals to use the address resolution mechanisms in IPoIB to > implement these features. IPv4 subnets use ARP and IPv6 subnets use > Neighbor Discovery. > > Analysis: > > IPoIB is not free. All nodes would be need to implement it for > this to work. > > The IB address -> IP address mapping on the passive side is > problematic. If a reverse lookup were available, IPoIB would require > both a GID and QP number as input. The passive side would know the GID > but the QP number. > > Further more, reverse lookup is not well supported. On IPv4 subnets, > RARP is quickly becoming (already?) obsolete. The IPoIB HW address includes the QPN (in addition to the GID). This is also problematic. > Neighbor Discovery > doesn't support reverse lookup at all. [RFC 2461] > > In addition to all this, IPoIB restricts an IP subnet to the same scope > as an IB subnet. IPoIB does not limit an IP subnet to an IB subnet. It can span IB subnets. However, IB routers were not completed in the IB architecture. > If a kDAPL consumer desired to communicate between > IB subnet's, IPoIB may not be sufficient. Are you referring to 2 disjoint IB subnets ? What about IB <-> iWARP ? > + GID as an IPv6 Address > > See the attachment to Caitlin Bestler's email: > > http://openib.org/pipermail/openib-general/2005-June/008104.html > > Analysis: > > This has been the least discussed option. One issue is > that GIDs may not be easy to administer. GIDs can be specific > to a particular channel adapter since they can contain EUI-64 > identifiers. Administrators avoid using Ethernet MAC addresses > in configuration files and they should be able to avoid using > adapter specific IB addresses as well. If they don't like ethernet MACs, they really won't like GUIDs/GIDs as they are even longer. > Another issue is how > dynamically assigned SM GIDs would be managed. Do you mean SM (assigned additional) GUIDs ? -- Hal From mst at mellanox.co.il Tue Jun 28 13:18:57 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 23:18:57 +0300 Subject: [openib-general] [PATCH (draft)] sdp: fix aio/sync completion race In-Reply-To: <20050525172954.A2979@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> <20050513135145.GA320@mellanox.co.il> <20050513100149.B24123@topspin.com> <20050525202128.GA7463@mellanox.co.il> <20050525172954.A2979@topspin.com> Message-ID: <20050628201857.GA5987@mellanox.co.il> Libor, here's a stub at solving an old problem. Most of the patch is passing sdp_opt to iocb complete and cancel functions. I didnt yet test it, and I wont have the time today, but maybe you could tell me whether I'm going in the right direction with this. --- Use "users" in the lock to prevent more iocbs from completing before a current aio is done. Any synchronous transfer would then wait till socket is unlocked. (Once we switch to get_user_pages, we'll be able to do it only for receive iocbs). Signed-off-by: Michael S. Tsirkin Index: sdp_write.c =================================================================== --- sdp_write.c (revision 2726) +++ sdp_write.c (working copy) @@ -109,7 +109,7 @@ int sdp_event_write(struct sdp_opt *conn SDP_CONN_STAT_WRITE_INC(conn, iocb->post); SDP_CONN_STAT_WQ_DEC(conn, iocb->size); - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); break; case SDP_DESC_TYPE_NONE: Index: sdp_rcvd.c =================================================================== --- sdp_rcvd.c (revision 2726) +++ sdp_rcvd.c (working copy) @@ -152,7 +152,7 @@ static int sdp_rcvd_send_sm(struct sdp_o conn->src_sent--; - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); } /* * Cancel complete, clear the state. @@ -209,7 +209,7 @@ static int sdp_rcvd_rdma_wr(struct sdp_o conn->snk_sent--; - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); return 0; } @@ -270,7 +270,7 @@ static int sdp_rcvd_rdma_rd(struct sdp_o SDP_CONN_STAT_WRITE_INC(conn, iocb->post); SDP_CONN_STAT_WQ_DEC(conn, iocb->size); - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); } /* * If Source Cancel was in process, and there are no more outstanding @@ -562,7 +562,7 @@ static int sdp_rcvd_snk_cancel_ack(struc while ((iocb = sdp_iocb_q_get_head(&conn->r_snk))) { conn->snk_sent--; - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); } /* * cancellation is complete. Cancel flag is cleared in RECV post. @@ -684,7 +684,7 @@ static int sdp_rcvd_snk_avail(struct sdp sdp_desc_q_put_head(&conn->send_queue, (struct sdpc_desc *)iocb); else - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); } /* * If Source Cancel was in process, it should now Index: sdp_proto.h =================================================================== --- sdp_proto.h (revision 2726) +++ sdp_proto.h (working copy) @@ -162,7 +162,8 @@ void sdp_iocb_q_put_tail(struct sdpc_ioc struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key); -void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp); +void sdp_iocb_q_cancel(struct sdp_opt *conn, struct sdpc_iocb_q *table, + u32 mask, ssize_t comp); void sdp_iocb_q_remove(struct sdpc_iocb *iocb); @@ -170,7 +171,8 @@ int sdp_iocb_register(struct sdpc_iocb * void sdp_iocb_release(struct sdpc_iocb *iocb); -void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status); +void sdp_iocb_complete(struct sdp_opt *conn, struct sdpc_iocb *iocb, + ssize_t status); int sdp_iocb_lock(struct sdpc_iocb *iocb); Index: sdp_read.c =================================================================== --- sdp_read.c (revision 2726) +++ sdp_read.c (working copy) @@ -205,7 +205,7 @@ int sdp_event_read(struct sdp_opt *conn, SDP_CONN_STAT_READ_INC(conn, iocb->post); SDP_CONN_STAT_RQ_DEC(conn, iocb->size); - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); break; case SDP_DESC_TYPE_NONE: @@ -226,7 +226,7 @@ int sdp_event_read(struct sdp_opt *conn, SDP_CONN_STAT_READ_INC(conn, iocb->post); SDP_CONN_STAT_RQ_DEC(conn, iocb->size); - sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0); + sdp_iocb_complete(conn, sdp_iocb_q_get_head(&conn->r_pend), 0); break; default: Index: sdp_send.c =================================================================== --- sdp_send.c (revision 2726) +++ sdp_send.c (working copy) @@ -859,7 +859,7 @@ static int sdp_send_data_iocb(struct sdp SDP_CONN_STAT_WRITE_INC(conn, iocb->post); SDP_CONN_STAT_WQ_DEC(conn, iocb->size); - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); } goto done; @@ -1730,7 +1730,7 @@ static int sdp_inet_write_cancel(struct * needs to be compelted. */ if (iocb->post > 0) { - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); result = -EAGAIN; } else { sdp_iocb_destroy(iocb); Index: sdp_conn.c =================================================================== --- sdp_conn.c (revision 2726) +++ sdp_conn.c (working copy) @@ -697,7 +697,8 @@ static int sdp_desc_q_cancel_lookup_func return ((element->type == SDP_DESC_TYPE_IOCB) ? 0 : -ERANGE); } -static void sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error) +static void sdp_desc_q_cancel_iocb(struct sdp_opt *conn, + struct sdpc_desc_q *table, ssize_t error) { struct sdpc_iocb *iocb; @@ -707,24 +708,24 @@ static void sdp_desc_q_cancel_iocb(struc NULL))) { sdp_iocb_q_remove(iocb); - sdp_iocb_complete(iocb, error); + sdp_iocb_complete(conn, iocb, error); } } void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error) { - sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error); - sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error); + sdp_iocb_q_cancel(conn, &conn->r_pend, SDP_IOCB_F_ALL, error); + sdp_iocb_q_cancel(conn, &conn->r_snk, SDP_IOCB_F_ALL, error); - sdp_desc_q_cancel_iocb(&conn->r_src, error); + sdp_desc_q_cancel_iocb(conn, &conn->r_src, error); } void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error) { - sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error); + sdp_iocb_q_cancel(conn, &conn->w_src, SDP_IOCB_F_ALL, error); - sdp_desc_q_cancel_iocb(&conn->send_queue, error); - sdp_desc_q_cancel_iocb(&conn->w_snk, error); + sdp_desc_q_cancel_iocb(conn, &conn->send_queue, error); + sdp_desc_q_cancel_iocb(conn, &conn->w_snk, error); } void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error) Index: sdp_recv.c =================================================================== --- sdp_recv.c (revision 2726) +++ sdp_recv.c (working copy) @@ -663,7 +663,7 @@ static int sdp_recv_buff_iocb_active(str /* * callback to complete IOCB */ - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); return (buff->tail - buff->data); } @@ -716,7 +716,7 @@ static int sdp_recv_buff_iocb_pending(st /* * callback to complete IOCB */ - sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0); + sdp_iocb_complete(conn, sdp_iocb_q_get_head(&conn->r_pend), 0); } return (buff->tail - buff->data); @@ -875,7 +875,7 @@ static int sdp_inet_read_cancel(struct k /* * callback to complete IOCB, or drop reference */ - sdp_iocb_complete(iocb, 0); + sdp_iocb_complete(conn, iocb, 0); result = -EAGAIN; } else { Index: sdp_conn.h =================================================================== --- sdp_conn.h (revision 2726) +++ sdp_conn.h (working copy) @@ -471,7 +471,7 @@ static inline void sdp_conn_unlock(struc sdp_conn_internal_unlock(conn); } - conn->lock.users = 0; + --conn->lock.users; wake_up(&conn->lock.waitq); spin_unlock_irqrestore(&conn->lock.slock, flags); Index: sdp_iocb.c =================================================================== --- sdp_iocb.c (revision 2726) +++ sdp_iocb.c (working copy) @@ -508,6 +508,8 @@ static void do_iocb_complete(void *arg) * we ignore the value. */ (void)aio_complete(iocb->req, value, 0); + + sdp_conn_unlock(iocb->conn); /* * delete IOCB */ @@ -517,11 +519,14 @@ static void do_iocb_complete(void *arg) /* * sdp_iocb_complete - complete an IOCB */ -void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status) +void sdp_iocb_complete(struct sdp_opt *conn, struct sdpc_iocb *iocb, + ssize_t status) { iocb->status = status; if (in_atomic() || irqs_disabled()) { + conn->lock.users++; + iocb->conn = conn; INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb); schedule_work(&iocb->completion); } else @@ -750,7 +755,8 @@ void sdp_iocb_q_put_head(struct sdpc_ioc /* * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue */ -void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp) +void sdp_iocb_q_cancel(struct sdp_opt *conn, struct sdpc_iocb_q *table, + u32 mask, ssize_t comp) { struct sdpc_iocb *iocb; struct sdpc_iocb *next; @@ -772,7 +778,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_ iocb->post, iocb->len); sdp_iocb_q_remove(iocb); - sdp_iocb_complete(iocb, comp); + sdp_iocb_complete(conn, iocb, comp); } iocb = next; Index: sdp_iocb.h =================================================================== --- sdp_iocb.h (revision 2726) +++ sdp_iocb.h (working copy) @@ -109,7 +109,8 @@ struct sdpc_iocb { int page_count; /* number of physical pages. */ int page_offset; /* offset into first page. */ - struct work_struct completion; /* task for defered completion. */ + struct work_struct completion; /* task for deferred completion. */ + struct sdp_opt *conn; /* conn to unlock for deferred completion. */ /* * kernel iocb structure */ -- MST From mshefty at ichips.intel.com Tue Jun 28 13:23:08 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 28 Jun 2005 13:23:08 -0700 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <1119988294.4848.2590.camel@hal.voltaire.com> References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> <1119987315.4848.2565.camel@hal.voltaire.com> <5264vytggh.fsf@topspin.com> <1119988294.4848.2590.camel@hal.voltaire.com> Message-ID: <42C1B1AC.4060400@ichips.intel.com> Hal Rosenstock wrote: > On Tue, 2005-06-28 at 15:43, Roland Dreier wrote: > >> Hal> Why does user space need to see all the timeouts ? I thought >> Hal> it was only request timeouts which were needed to be >> Hal> seen. The semantics for RMPP timeouts appear to be different. >> >>Well, can the MAD core even generate timeouts for trap repress or >>response messages? > > > Yes (At least if the send was RMPP'd). That's what started this thread. > Not sure about normal MAD send of those. Sean ? A non-RMPP response or trap repress wouldn't receive a timeout, since they are not acknowledged in any way. - Sean From mshefty at ichips.intel.com Tue Jun 28 13:26:50 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 28 Jun 2005 13:26:50 -0700 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <1119988294.4848.2590.camel@hal.voltaire.com> References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> <1119987315.4848.2565.camel@hal.voltaire.com> <5264vytggh.fsf@topspin.com> <1119988294.4848.2590.camel@hal.voltaire.com> Message-ID: <42C1B28A.5000806@ichips.intel.com> Hal Rosenstock wrote: > I'm not sure what a consumer would want a timeout to a non request > method. Since RMPP provides reliability, I think that it makes sense to indicate to the user when an RMPP MAD is not acknowledged. If no error is generated, the user could mistakenly assume that the packet was successfully delivered. > Other alternatives are: > Change RMPP send timeouts to be same as normal MAD sends > or > Not return entire send packet on timeout. Is that a requirement also ? I agree with Roland that there doesn't seem to be a need to return the entire send packet on a timeout. A user could always keep the MAD around until it completes if it needed it for some reason. - Sean From halr at voltaire.com Tue Jun 28 13:21:43 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 16:21:43 -0400 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <52r7emqm0a.fsf@topspin.com> References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> <1119987315.4848.2565.camel@hal.voltaire.com> <5264vytggh.fsf@topspin.com> <1119988294.4848.2590.camel@hal.voltaire.com> <52r7emqm0a.fsf@topspin.com> Message-ID: <1119989741.4848.2610.camel@hal.voltaire.com> On Tue, 2005-06-28 at 16:12, Roland Dreier wrote: > Hal> I'm not sure what a consumer would want a timeout to a non > Hal> request method. > > I guess my view would be that if a consumer would never want a > timeout, then the kernel shouldn't generate them. We be consistent > between kernel space and user space though -- the umad module > shouldn't be filtering out information. > > Hal> Other alternatives are: Change RMPP send timeouts to be same > Hal> as normal MAD sends or Not return entire send packet on > Hal> timeout. Is that a requirement also ? > > I don't think we need to return the whole packet on timed out sends > now that the MAD structure has the header first. In fact it probably > makes sense to leave the payload off of timeout responses. We just > need a way to return the transaction_id of the timed-out send. OK. I'll work on this approach then. I still think there's a semantic difference with timeouts which needs more consistency. -- Hal From rolandd at cisco.com Tue Jun 28 13:29:33 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 13:29:33 -0700 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <1119989741.4848.2610.camel@hal.voltaire.com> (Hal Rosenstock's message of "28 Jun 2005 16:21:43 -0400") References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> <1119987315.4848.2565.camel@hal.voltaire.com> <5264vytggh.fsf@topspin.com> <1119988294.4848.2590.camel@hal.voltaire.com> <52r7emqm0a.fsf@topspin.com> <1119989741.4848.2610.camel@hal.voltaire.com> Message-ID: <52fyv2ql76.fsf@topspin.com> Hal> I still think there's a semantic difference with timeouts Hal> which needs more consistency. Could you elaborate? What are the different situations where timeouts behave differently? - R. From caitlin.bestler at gmail.com Tue Jun 28 13:33:34 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 28 Jun 2005 13:33:34 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <52br5qtgk9.fsf@topspin.com> References: <52br5qtgk9.fsf@topspin.com> Message-ID: <469958e0050628133348e7b64b@mail.gmail.com> On 6/28/05, Roland Dreier wrote: > James> First off, here [are the] requirement we are trying to satisfy: > > James> On the passive side of a connection, a InfiniBand kDAPL > James> provider must determine a source IB address for an > James> InfiniBand connection request. This information can be > James> obtain by a kDAPL consumer either in the > James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or > James> dat_cr_query()'s dat_cr_param structure. > > James> By interoperable, we mean that the solution must not > James> introduce a non-standard protocol or force ULPs using kDAPL > James> to perform special operations when using an InfiniBand > James> network. > > Since these two points are mutually contradictory -- the IB > communication management protocol does not carry enough information > for a connection request to be mapped uniquely back to a source > address -- we need to figure out which one to drop. > > I would argue in favor of the solution selected by SDP: when defining > the binding of an abstract protocol to the IB transport, put the > source and destination IP addresses in the IB-specific connection > setup messages. > > - R. The remote identification is a service being provided *to* the ULP. SDP, or any other application, can provide whatever additional information desired. The "IA Address" is as authenticated as the local network management allows it to be. Private data exchanged by applications is outside the view of the network administrator and therefore can never be authenticated in the same way. For NFSoRDMA, for example, "authenticating" a remote peer based upon an application supplied field in the private data is obviously inappropriate. The GID that InfiniBand does supply *does* fully qualify as an IP Address, and can be the address supplied as the "remote address" even *if* there are additional methods of translating IPv4 addresses to GIDs (or even IPv6, but why you would want to *translate* an IPv6 address to a GID rather than just using the GID *as* an IPv6 address is beyond me). To clarify: DAT *does* fully support the use of GIDs as IA Addresses, however it assumes that link local addresses will not be presented to the Consumer (at least when the host is attached to more than one subnet). It assumes that they will be at least upgraded to site-local. The IA Address can be easily sub-divided into IPv4 addresses that need translation, IPv6 addresses that need translation and IPv6 addresses that are also GIDs and therefore do not need translation. The existence of the alternate solutions was largely driven by the need to deploy early solutions that were *not* integrated with the OS naming system, and therefore could not assume that the OS already knew what to do with a GID. That is no longer a problem. Therefore there is no need to change any of the DAT semantics. They are adequate as they are, and in particular there is no need to eliminate reporting of remote peer addresses -- something that is both easy and useful for IP networks. And a feature that is already in use. From halr at voltaire.com Tue Jun 28 13:31:38 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 16:31:38 -0400 Subject: [openib-general] [PATCH] user_mad: In send_handler, only queue_packet on timeout if not response In-Reply-To: <52fyv2ql76.fsf@topspin.com> References: <1119985980.4848.2560.camel@hal.voltaire.com> <52oe9qth98.fsf@topspin.com> <1119987315.4848.2565.camel@hal.voltaire.com> <5264vytggh.fsf@topspin.com> <1119988294.4848.2590.camel@hal.voltaire.com> <52r7emqm0a.fsf@topspin.com> <1119989741.4848.2610.camel@hal.voltaire.com> <52fyv2ql76.fsf@topspin.com> Message-ID: <1119990698.4848.2613.camel@hal.voltaire.com> On Tue, 2005-06-28 at 16:29, Roland Dreier wrote: > Hal> I still think there's a semantic difference with timeouts > Hal> which needs more consistency. > > Could you elaborate? What are the different situations where timeouts > behave differently? Sean has responded that (since RMPP provides reliability) there is value in indicating when an RMPP MAD is not acknowledged. -- Hal From mst at mellanox.co.il Tue Jun 28 13:46:46 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 28 Jun 2005 23:46:46 +0300 Subject: [openib-general] Re: IP addressing on InfiniBand networks In-Reply-To: References: Message-ID: <20050628204646.GB5987@mellanox.co.il> Hi, James! I dont know much about dapl, so forgive me if the question is naive: Quoting r. James Lentini : > > + CM Private Data > > The active side of an IB connection could place its source IP > address in the CM's private data. The passive side would retrieve > the source IP from this location. > > ... > > The security of this is very week. An end node could easily present > a false IP address. Once you have the IP from CM private data, what prevents you from resolving it back to hardware address (by sending an ARP request with the IP address that you got)? You get back the IPoIB hardware address: GID+QPN, and can verify that the GID matches the GID that you got from CM. The security of this seems to be at least as good as the one you get on regular IP networks. Does this make sense at all? -- MST From nacc at us.ibm.com Tue Jun 28 13:48:55 2005 From: nacc at us.ibm.com (Nishanth Aravamudan) Date: Tue, 28 Jun 2005 13:48:55 -0700 Subject: [openib-general] [PATCH] sdp_inet: fix schedule_timeout() usage Message-ID: <20050628204855.GA24226@us.ibm.com> Hello, My first patch to openib-general. Let me know if I diffed from the wrong directory or anything else. Thanks, Nish Using schedule_timeout() without setting the state first is broken and causes schedule_timeout() to return immediately (effectively you call schedule() without changing your state and are thus going to run again). In each of these loops in sdp_inet.c involving schedule_timeout(), the first iteration is correct, but subsequent ones result in busy-wait. Add the appropriate set_current_state() call to fix the issue. Signed-off-by: Nishanth Aravamudan Index: trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c (revision 2734) +++ trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -388,6 +388,7 @@ static int sdp_inet_release(struct socke if (signal_pending(current)) break; + set_current_state(TASK_INTERRUPTIBLE); } set_current_state(TASK_RUNNING); @@ -627,6 +628,7 @@ static int sdp_inet_connect(struct socke break; } + set_current_state(TASK_INTERRUPTIBLE); } set_current_state(TASK_RUNNING); @@ -782,6 +784,7 @@ static int sdp_inet_accept(struct socket if (signal_pending(current)) break; + set_current_state(TASK_INTERRUPTIBLE); } set_current_state(TASK_RUNNING); From halr at voltaire.com Tue Jun 28 13:44:10 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 16:44:10 -0400 Subject: [openib-general] [PATCH] user_mad: In send_handler, only return header on timeout Message-ID: <1119991449.4848.2617.camel@hal.voltaire.com> user_mad: In send_handler, only return header on timeout Signed-off-by: Hal Rosenstock Index: user_mad.c =================================================================== --- user_mad.c (revision 2731) +++ user_mad.c (working copy) @@ -139,19 +139,28 @@ struct ib_mad_send_wc *send_wc) { struct ib_umad_file *file = agent->context; - struct ib_umad_packet *packet = + struct ib_umad_packet *timeout, *packet = (void *) (unsigned long) send_wc->wr_id; ib_destroy_ah(packet->msg->send_wr.wr.ud.ah); ib_free_send_mad(packet->msg); if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) { - packet->mad.hdr.status = ETIMEDOUT; + timeout = kmalloc(sizeof *timeout + 24, GFP_KERNEL); + if (!timeout) + goto out; - if (!queue_packet(file, agent, packet)) - return; + memset(timeout, 0, sizeof *timeout + 24); + + timeout->length = 24; + timeout->mad.hdr.id = packet->mad.hdr.id; + timeout->mad.hdr.status = ETIMEDOUT; + memcpy(timeout->mad.data, packet->mad.data, 24); + + if (!queue_packet(file, agent, timeout)) + return; } - +out: kfree(packet); } From ftillier at silverstorm.com Tue Jun 28 13:57:20 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Tue, 28 Jun 2005 13:57:20 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1119984398.4848.2549.camel@hal.voltaire.com> Message-ID: <000101c57c23$fac96790$6501a8c0@infiniconsys.com> > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Tuesday, June 28, 2005 11:47 AM > I think I see what is going on here... > > In user_mad.c::send_handler > > > if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) { > packet->mad.hdr.status = ETIMEDOUT; > > if (!queue_packet(file, agent, packet)) > return; > } > > That is what is causing the problem. I think the send side queues the > packet on a timeout and simulates a receive so that a transaction can be > terminated. RMPP sends appear to be a little different in that even non > transactions get timeouts. Why is a receive generated at all? Shouldn't the send completion be enough? It seems odd to just generate a receive indication with the sent data when really, nothing was received. From an app perspective, it seems that handling the IB_WC_RESP_TIMEOUT_ERR case should be sufficient to indicate the transaction is complete. - Fab From rolandd at cisco.com Tue Jun 28 13:58:39 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 13:58:39 -0700 Subject: [openib-general] [PATCH] user_mad: In send_handler, only return header on timeout In-Reply-To: <1119991449.4848.2617.camel@hal.voltaire.com> (Hal Rosenstock's message of "28 Jun 2005 16:44:10 -0400") References: <1119991449.4848.2617.camel@hal.voltaire.com> Message-ID: <52slz2p5a8.fsf@topspin.com> Hal> user_mad: In send_handler, only return header on timeout This seems reasonable. Is there anything more self-documenting we can use instead of hard-coding the magic number of 24? - R. From ardavis at ichips.intel.com Tue Jun 28 13:59:01 2005 From: ardavis at ichips.intel.com (Arlin Davis) Date: Tue, 28 Jun 2005 13:59:01 -0700 Subject: [openib-general] uCM create connection ID Message-ID: <42C1BA15.7060205@ichips.intel.com> Hi Libor, I have a couple of uCM questions regarding create_id and events... Is it possible for a consumer of uCM to provide a context with the create_id that could be returned with the event? I will have some scale up issues if I have to walk a list looking for a uCM provided connection ID instead of a context that could point directly to the appropriate uDAPL CM object. It would also be very helpful if the CM event could be woke from user space. Thanks, -arlin From halr at voltaire.com Tue Jun 28 14:01:11 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 17:01:11 -0400 Subject: [openib-general] [PATCH] user_mad: In send_handler, only return header on timeout In-Reply-To: <52slz2p5a8.fsf@topspin.com> References: <1119991449.4848.2617.camel@hal.voltaire.com> <52slz2p5a8.fsf@topspin.com> Message-ID: <1119992470.4848.2640.camel@hal.voltaire.com> On Tue, 2005-06-28 at 16:58, Roland Dreier wrote: > Hal> user_mad: In send_handler, only return header on timeout > > This seems reasonable. Is there anything more self-documenting we can > use instead of hard-coding the magic number of 24? I think so. I had a feeling that would be a comment. Give me a minute or two... -- Hal From jlentini at netapp.com Tue Jun 28 14:11:00 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 17:11:00 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [udapl] a simple uDAPL test In-Reply-To: <42C07E2D.3040301@ichips.intel.com> References: <42C07E2D.3040301@ichips.intel.com> Message-ID: Committed in revision 2738. On Mon, 27 Jun 2005, Arlin Davis wrote: > James, > > Here is a simple uDAPL test that you can add to your userspace directory. > > -arlin > > > Signed-off-by: Arlin Davis > > Index: dtest/dtest.c > =================================================================== > --- dtest/dtest.c (revision 0) > +++ dtest/dtest.c (revision 0) > @@ -0,0 +1,1700 @@ > +/* > + * Copyright (c) 2005 Intel Corporation. All rights reserved. > + * > + * This software is available to you under a choice of one of two > + * licenses. You may choose to be licensed under the terms of the GNU > + * General Public License (GPL) Version 2, available from the file > + * COPYING in the main directory of this source tree, or the > + * OpenIB.org BSD license below: > + * > + * Redistribution and use in source and binary forms, with or > + * without modification, are permitted provided that the following > + * conditions are met: > + * > + * - Redistributions of source code must retain the above > + * copyright notice, this list of conditions and the following > + * disclaimer. > + * > + * - Redistributions in binary form must reproduce the above > + * copyright notice, this list of conditions and the following > + * disclaimer in the documentation and/or other materials > + * provided with the distribution. > + * > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND > + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS > + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN > + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN > + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE > + * SOFTWARE. > + * > + * $Id: $ > + */ > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#ifndef DAPL_PROVIDER > +#define DAPL_PROVIDER "OpenIB1_2" > +#endif > + > +#define MAX_POLLING_CNT 50000 > + > +/* Header files needed for DAT/uDAPL */ > +#include "dat/udat.h" > + > +/* definitions */ > +#define SERVER_CONN_QUAL 71123 > +#define DTO_TIMEOUT (1000*1000*5) > +#define DTO_FLUSH_TIMEOUT (1000*1000*2) > +#define CONN_TIMEOUT (1000*1000*10) > +#define SERVER_TIMEOUT (1000*1000*20) > +#define RDMA_BUFFER_SIZE (64) > + > +/* Global DAT vars */ > +static DAT_IA_HANDLE h_ia = DAT_HANDLE_NULL; > +static DAT_PZ_HANDLE h_pz = DAT_HANDLE_NULL; > +static DAT_EP_HANDLE h_ep = DAT_HANDLE_NULL; > +static DAT_PSP_HANDLE h_psp = DAT_HANDLE_NULL; > +static DAT_CR_HANDLE h_cr = DAT_HANDLE_NULL; > + > +static DAT_EVD_HANDLE h_async_evd = DAT_HANDLE_NULL; > +static DAT_EVD_HANDLE h_dto_evd = DAT_HANDLE_NULL; > +static DAT_EVD_HANDLE h_cr_evd = DAT_HANDLE_NULL; > +static DAT_EVD_HANDLE h_conn_evd = DAT_HANDLE_NULL; > +static DAT_CNO_HANDLE h_dto_cno = DAT_HANDLE_NULL; > + > +/* RDMA buffers */ > +static DAT_LMR_HANDLE h_lmr_send = DAT_HANDLE_NULL; > +static DAT_LMR_HANDLE h_lmr_recv = DAT_HANDLE_NULL; > +static DAT_LMR_CONTEXT lmr_context_send; > +static DAT_LMR_CONTEXT lmr_context_recv; > +static DAT_RMR_CONTEXT rmr_context_send; > +static DAT_RMR_CONTEXT rmr_context_recv; > +static DAT_VLEN registered_size_send; > +static DAT_VLEN registered_size_recv; > +static DAT_VADDR registered_addr_send; > +static DAT_VADDR registered_addr_recv; > + > +/* Initial msg receive buf, RMR exchange, and Rdma-write notification */ > +#define MSG_BUF_COUNT 3 > +static DAT_RMR_TRIPLET rmr_recv_msg[ MSG_BUF_COUNT ]; > +static DAT_LMR_HANDLE h_lmr_recv_msg = DAT_HANDLE_NULL; > +static DAT_LMR_CONTEXT lmr_context_recv_msg; > +static DAT_RMR_CONTEXT rmr_context_recv_msg; > +static DAT_VLEN registered_size_recv_msg; > +static DAT_VADDR registered_addr_recv_msg; > + > +/* message send buffer */ > +static DAT_RMR_TRIPLET rmr_send_msg; > +static DAT_LMR_HANDLE h_lmr_send_msg = DAT_HANDLE_NULL; > +static DAT_LMR_CONTEXT lmr_context_send_msg; > +static DAT_RMR_CONTEXT rmr_context_send_msg; > +static DAT_VLEN registered_size_send_msg; > +static DAT_VADDR registered_addr_send_msg; > +static DAT_EP_ATTR ep_attr; > +char hostname[256] = {0}; > + > +/* rdma pointers */ > +char *rbuf = NULL; > +char *sbuf = NULL; > +int status; > + > +/* timers */ > +double start,stop,total_us,total_sec; > +struct { > + double total; > + double open; > + double reg; > + double unreg; > + double pzc; > + double pzf; > + double evdc; > + double evdf; > + double cnoc; > + double cnof; > + double epc; > + double epf; > + double rdma_wr; > + double rdma_rd; > + double rtt; > + double close; > +} time; > + > +/* defaults */ > +static int parent=1; > +static int connected=0; > +static int burst=10; > +static int server=1; > +static int verbose=0; > +static int polling=1; > +static int poll_count=0; > +static int rdma_wr_poll_count=0; > +static int rdma_rd_poll_count=0; > +static int pin_memory=0; > +static int delay=0; > +static int buf_len=RDMA_BUFFER_SIZE; > +static int use_cno=0; > +static int post_recv_count=MSG_BUF_COUNT; > +static int recv_msg_index=0; > +static int burst_msg_posted=0; > +static int burst_msg_index=0; > + > +#define MAX_RDMA_RD 4 > +#define MAX_PROCS 1000 > + > +static pid_t child[MAX_PROCS+1]; > + > +/* forward prototypes */ > +const char * DT_RetToString (DAT_RETURN ret_value); > +const char * DT_EventToSTr (DAT_EVENT_NUMBER event_code); > +void print_usage(); > +double get_time(); > +void init_data(); > + > +DAT_RETURN send_msg( void *data, > + DAT_COUNT size, > + DAT_LMR_CONTEXT context, > + DAT_DTO_COOKIE cookie, > + DAT_COMPLETION_FLAGS flags ); > + > +DAT_RETURN connect_ep( char *hostname, int conn_id ); > +void disconnect_ep( void ); > +DAT_RETURN register_rdma_memory( void ); > +DAT_RETURN unregister_rdma_memory( void ); > +DAT_RETURN create_events( void ); > +DAT_RETURN destroy_events(void); > +DAT_RETURN do_rdma_write_with_msg( void ); > +DAT_RETURN do_rdma_read_with_msg( void ); > +DAT_RETURN do_ping_pong_msg( void ); > + > +#define LOGPRINTF(_format, _aa...) \ > + if (verbose) \ > + printf(_format, ##_aa) > + > +main(int argc, char **argv) > +{ > + int c; > + DAT_RETURN ret; > + > + /* parse arguments */ > + while ((c = getopt(argc, argv, "scvpb:d:B:h:")) != -1) > + { > + switch(c) > + { > + case 's': > + server = 1; > + printf("%d Running as server\n",getpid()); > + fflush(stdout); > + break; > + case 'c': > + use_cno = 1; > + printf("%d Creating CNO for DTO > EVD's\n",getpid()); > + fflush(stdout); > + break; > + case 'v': > + verbose = 1; > + printf("%d Verbose\n",getpid()); > + fflush(stdout); > + break; > + case 'p': > + polling = 1; > + printf("%d Polling\n",getpid()); > + fflush(stdout); > + break; > + case 'B': > + burst = atoi(optarg); > + break; > + case 'd': > + delay = atoi(optarg); > + break; > + case 'b': > + buf_len = atoi(optarg); > + break; > + case 'h': > + server = 0; > + strcpy (hostname, optarg); > + break; > + default: > + print_usage(); > + exit(-12); > + } > + } > + > + if (!server) { > + printf("%d Running as client\n",getpid()); fflush(stdout); > + } else { > + printf("%d Running as server\n",getpid()); fflush(stdout); > + } > + > + /* allocate send and receive buffers */ > + if (((rbuf = malloc(buf_len*burst)) == NULL) || > + ((sbuf = malloc(buf_len*burst)) == NULL)) { > + perror("malloc"); > + exit(1); > + } > + memset( &time, sizeof(time), 0); > + LOGPRINTF("%d Allocated RDMA buffers (r:%p,s:%p) len %d \n", > + getpid(), rbuf, sbuf, buf_len); > + > + /* dat_ia_open, dat_pz_create */ > + h_async_evd = DAT_HANDLE_NULL; > + start = get_time(); > + ret = dat_ia_open( DAPL_PROVIDER, 8, &h_async_evd, &h_ia ); > + stop = get_time(); > + time.open += ((stop - start)*1.0e6); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: Error Adaptor open: %s\n", > + getpid(),DT_RetToString(ret)); > + exit(1); > + } else > + LOGPRINTF("%d Opened Interface Adaptor\n",getpid()); > + > + /* Create Protection Zone */ > + start = get_time(); > + LOGPRINTF("%d Create Protection Zone\n",getpid()); > + ret = dat_pz_create(h_ia, &h_pz); > + stop = get_time(); > + time.pzc += ((stop - start)*1.0e6); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, > + "%d Error creating Protection Zone: %s\n", > + getpid(),DT_RetToString(ret)); > + exit(1); > + } else > + LOGPRINTF("%d Created Protection Zone\n",getpid()); > + > + /* Register memory */ > + LOGPRINTF("%d Register RDMA memory\n", getpid()); > + ret = register_rdma_memory(); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error creating events: %s\n", > + getpid(),DT_RetToString(ret)); > + goto cleanup; > + } else > + LOGPRINTF("%d Register RDMA memory done\n", getpid()); > + > + LOGPRINTF("%d Create events\n", getpid()); > + ret = create_events(); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error creating events: %s\n", > + getpid(),DT_RetToString(ret)); > + goto cleanup; > + } else { > + LOGPRINTF("%d Create events done\n", getpid()); > + } > + > + /* create EP */ > + memset( &ep_attr, 0, sizeof(ep_attr) ); > + ep_attr.service_type = DAT_SERVICE_TYPE_RC; > + ep_attr.max_rdma_size = 0x10000; > + ep_attr.qos = 0; > + ep_attr.recv_completion_flags = 0; > + ep_attr.max_recv_dtos = MSG_BUF_COUNT + (burst*3); > + ep_attr.max_request_dtos = MSG_BUF_COUNT + (burst*3) + > MAX_RDMA_RD; > + ep_attr.max_recv_iov = 1; > + ep_attr.max_request_iov = 1; > + ep_attr.max_rdma_read_in = MAX_RDMA_RD; > + ep_attr.max_rdma_read_out = MAX_RDMA_RD; > + ep_attr.request_completion_flags = DAT_COMPLETION_DEFAULT_FLAG; > + ep_attr.ep_transport_specific_count = 0; > + ep_attr.ep_transport_specific = NULL; > + ep_attr.ep_provider_specific_count = 0; > + ep_attr.ep_provider_specific = NULL; > + > + start = get_time(); > + ret = dat_ep_create( h_ia, h_pz, h_dto_evd, h_dto_evd, h_conn_evd, > &ep_attr, &h_ep ); > + stop = get_time(); > + time.epc += ((stop - start)*1.0e6); > + time.total += time.epc; > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_ep_create: %s\n", > + getpid(),DT_RetToString(ret)); > + goto cleanup; > + } else > + LOGPRINTF("%d EP created %p \n", getpid(), h_ep); > + > + /* > + * register message buffers, establish connection, and > + * exchange DMA RMR information info via messages > + */ > + ret = connect_ep( hostname, SERVER_CONN_QUAL ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error connect_ep: %s\n", > + getpid(),DT_RetToString(ret)); > + goto cleanup; > + } else > + LOGPRINTF("%d connect_ep complete\n", getpid()); > + > + /*********** RDMA write data *************/ > + ret = do_rdma_write_with_msg(); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error do_rdma_write_with_msg: %s\n", > + getpid(),DT_RetToString(ret)); > + goto cleanup; > + } else > + LOGPRINTF("%d do_rdma_write_with_msg complete\n", getpid()); > + > + /*********** RDMA read data *************/ > + ret = do_rdma_read_with_msg(); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error do_rdma_read_with_msg: %s\n", > + getpid(),DT_RetToString(ret)); > + goto cleanup; > + } else > + LOGPRINTF("%d do_rdma_read_with_msg complete\n", getpid()); > + > + /*********** PING PING messages ************/ > + ret = do_ping_pong_msg(); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error do_ping_pong_msg: %s\n", > + getpid(),DT_RetToString(ret)); > + goto cleanup; > + } else > + LOGPRINTF("%d do_ping_pong_msg complete\n", getpid()); > + > +cleanup: > + /* disconnect and free EP resources */ > + if ( h_ep != DAT_HANDLE_NULL ) { > + /* unregister message buffers and tear down connection */ > + LOGPRINTF("%d Disconnect and Free EP %p \n",getpid(),h_ep); > + disconnect_ep(); > + } > + > + /* free EP */ > + LOGPRINTF("%d Free EP %p \n",getpid(),h_ep); > + start = get_time(); > + ret = dat_ep_free( h_ep ); > + stop = get_time(); > + time.epf += ((stop - start)*1.0e6); > + time.total += time.epf; > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error freeing EP: %s\n", > + getpid(), DT_RetToString(ret)); > + } else { > + LOGPRINTF("%d Freed EP\n",getpid()); > + h_ep = DAT_HANDLE_NULL; > + } > + > + /* free EVDs */ > + LOGPRINTF("%d destroy events\n", getpid()); > + ret = destroy_events(); > + if(ret != DAT_SUCCESS) > + fprintf(stderr, "%d Error destroy_events: %s\n", > + getpid(),DT_RetToString(ret)); > + else > + LOGPRINTF("%d destroy events done\n", getpid()); > + > + > + ret = unregister_rdma_memory(); > + LOGPRINTF("%d unregister_rdma_memory \n", getpid()); > + if(ret != DAT_SUCCESS) > + fprintf(stderr, "%d Error unregister_rdma_memory: %s\n", > + getpid(),DT_RetToString(ret)); > + else > + LOGPRINTF("%d unregister_rdma_memory done\n", getpid()); > + > + > + if (delay) sleep(delay); > + > + /* Free protection domain */ > + LOGPRINTF("%d Freeing pz\n",getpid()); > + start = get_time(); > + ret = dat_pz_free( h_pz ); > + stop = get_time(); > + time.pzf += ((stop - start)*1.0e6); > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error freeing PZ: %s\n", > + getpid(), DT_RetToString(ret)); > + } else { > + LOGPRINTF("%d Freed pz\n",getpid()); > + h_pz = NULL; > + } > + > + /* close the device */ > + LOGPRINTF("%d Closing Interface Adaptor\n",getpid()); > + start = get_time(); > + ret = dat_ia_close( h_ia, DAT_CLOSE_ABRUPT_FLAG ); > + stop = get_time(); > + time.close += ((stop - start)*1.0e6); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: Error Adaptor close: %s\n", > + getpid(),DT_RetToString(ret)); > + exit(1); > + } else > + LOGPRINTF("%d Closed Interface Adaptor\n",getpid()); > + > + printf( "\n%d: DAPL Test Complete.\n\n",getpid()); > + printf( "%d: RDMA write: Total=%10.2lf usec, %d bursts, > itime=%10.2lf usec, pc=%d\n", > + getpid(), time.rdma_wr, burst, time.rdma_wr/burst, > rdma_wr_poll_count ); > + printf( "%d: RDMA read: Total=%10.2lf usec, %d bursts, > itime=%10.2lf usec, pc=%d\n", > + getpid(), time.rdma_rd, MAX_RDMA_RD, > time.rdma_rd/MAX_RDMA_RD, rdma_rd_poll_count ); > + printf( "%d: Message RTT: Total=%10.2lf usec, %d bursts, > itime=%10.2lf usec, pc=%d\n\n", > + getpid(), time.rtt, burst, time.rtt/burst, poll_count ); > + > + printf( "%d: open: %10.2lf usec\n", getpid(), time.open ); > + printf( "%d: close: %10.2lf usec\n", getpid(), time.close ); > + printf( "%d: PZ create: %10.2lf usec\n", getpid(), time.pzc ); > + printf( "%d: PZ free: %10.2lf usec\n", getpid(), time.pzf ); > + printf( "%d: LMR create:%10.2lf usec\n", getpid(), time.reg ); > + printf( "%d: LMR free: %10.2lf usec\n", getpid(), time.unreg ); > + printf( "%d: EVD create:%10.2lf usec\n", getpid(), time.evdc ); > + printf( "%d: EVD free: %10.2lf usec\n", getpid(), time.evdf ); > + if (use_cno) { > + printf( "%d: CNO create: %10.2lf usec\n", getpid(), time.cnoc ); > + printf( "%d: CNO free: %10.2lf usec\n", getpid(), time.cnof ); > + } > + printf( "%d: EP create: %10.2lf usec\n",getpid(), time.epc ); > + printf( "%d: EP free: %10.2lf usec\n",getpid(), time.epf ); > + printf( "%d: TOTAL: %10.2lf usec\n",getpid(), time.total ); > + > + /* free rdma buffers */ > + free(rbuf); > + free(sbuf); > +} > + > + > +double get_time() > +{ > + struct timeval tp; > + > + gettimeofday(&tp, NULL); > + return ((double) tp.tv_sec + (double) tp.tv_usec * 1e-6); > +} > + > +void init_data() > +{ > + memset(rbuf, 'a', buf_len); > + memset(sbuf, 'b', buf_len); > +} > + > + > +DAT_RETURN > +send_msg( void *data, > + DAT_COUNT size, > + DAT_LMR_CONTEXT context, > + DAT_DTO_COOKIE cookie, > + DAT_COMPLETION_FLAGS flags ) > +{ > + DAT_LMR_TRIPLET iov; > + DAT_EVENT event; > + DAT_COUNT nmore; > + DAT_RETURN ret; > + > + iov.lmr_context = context; > + iov.pad = 0; > + iov.virtual_address = (DAT_VADDR)(unsigned long)data; > + iov.segment_length = size; > + LOGPRINTF("%d calling post_send\n", getpid()); > + cookie.as_64 = 0xaaaa; > + ret = dat_ep_post_send( h_ep, > + 1, > + &iov, > + cookie, > + flags ); > + > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: ERROR: dat_ep_post_send() %s\n", > + getpid(),DT_RetToString(ret)); > + return ret; > + } > + > + if (!(flags & DAT_COMPLETION_SUPPRESS_FLAG)) { > + if ( polling ) { > + printf("%d Polling post send completion...\n",getpid()); > + while ( dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY > ); > + } > + else { > + LOGPRINTF("%d waiting for post_send completion event\n", > getpid()); > + if (use_cno) { > + DAT_EVD_HANDLE evd = DAT_HANDLE_NULL; > + ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd ); > + LOGPRINTF("%d cno wait return evd_handle=%p\n", > getpid(),evd); > + if ( evd != h_dto_evd ) { > + fprintf(stderr, > + "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", > + getpid()); > + return( DAT_ABORT ); > + } > + } > + /* use wait to dequeue */ > + ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore ); > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", > + getpid(),DT_RetToString(ret)); > + return ret; > + } > + } > + > + /* validate event number, len, cookie, and status */ > + if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) { > + fprintf(stderr, "%d: ERROR: DTO event number %s\n", > + getpid(),DT_EventToSTr(event.event_number)); > + return( DAT_ABORT ); > + } > + > + if ((event.event_data.dto_completion_event_data.transfered_length != > size ) || > + (event.event_data.dto_completion_event_data.user_cookie.as_64 != > 0xaaaa )) { > + fprintf(stderr, "%d: ERROR: DTO len %d or cookie %x\n", > + getpid(), > + event.event_data.dto_completion_event_data.transfered_length, > + event.event_data.dto_completion_event_data.user_cookie.as_64 > ); > + return( DAT_ABORT ); > + > + } > + if (event.event_data.dto_completion_event_data.status != DAT_SUCCESS) > { > + fprintf(stderr, "%d: ERROR: DTO event status %s\n", > + getpid(),DT_RetToString(ret)); > + return( DAT_ABORT ); > + } > + } > + > + return DAT_SUCCESS; > +} > + > + > +DAT_RETURN > +connect_ep( char *hostname, int conn_id ) > +{ > + DAT_SOCK_ADDR remote_addr; > + DAT_EP_ATTR ep_attr; > + DAT_RETURN ret; > + DAT_REGION_DESCRIPTION region; > + DAT_EVENT event; > + DAT_COUNT nmore; > + DAT_LMR_TRIPLET l_iov; > + DAT_RMR_TRIPLET r_iov; > + DAT_DTO_COOKIE cookie; > + int i; > + > + /* Register send message buffer */ > + LOGPRINTF("%d Registering send Message Buffer %p, len %d\n", > + getpid(), &rmr_send_msg, sizeof(DAT_RMR_TRIPLET) ); > + region.for_va = &rmr_send_msg; > + ret = dat_lmr_create( h_ia, > + DAT_MEM_TYPE_VIRTUAL, > + region, > + sizeof(DAT_RMR_TRIPLET), > + h_pz, > + DAT_MEM_PRIV_LOCAL_WRITE_FLAG, > + &h_lmr_send_msg, > + &lmr_context_send_msg, > + &rmr_context_send_msg, > + ®istered_size_send_msg, > + ®istered_addr_send_msg ); > + > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error registering send msg buffer: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d Registered send Message Buffer %p \n", > + getpid(),region.for_va ); > + > + /* Register Receive buffers */ > + LOGPRINTF("%d Registering Receive Message Buffer %p\n", > + getpid(), rmr_recv_msg ); > + region.for_va = rmr_recv_msg; > + ret = dat_lmr_create( h_ia, > + DAT_MEM_TYPE_VIRTUAL, > + region, > + sizeof(DAT_RMR_TRIPLET)*MSG_BUF_COUNT, > + h_pz, > + DAT_MEM_PRIV_LOCAL_WRITE_FLAG, > + &h_lmr_recv_msg, > + &lmr_context_recv_msg, > + &rmr_context_recv_msg, > + ®istered_size_recv_msg, > + ®istered_addr_recv_msg ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error registering recv msg buffer: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d Registered Receive Message Buffer %p\n", > + getpid(),region.for_va); > + > + for ( i = 0; i < MSG_BUF_COUNT; i++ ) { > + cookie.as_64 = i; > + l_iov.lmr_context = lmr_context_recv_msg; > + l_iov.pad = 0; > + l_iov.virtual_address = (DAT_VADDR)(unsigned long)&rmr_recv_msg[ i ]; > + l_iov.segment_length = sizeof(DAT_RMR_TRIPLET); > + > + LOGPRINTF("%d Posting Receive Message Buffer %p\n", > + getpid(), &rmr_recv_msg[ i ]); > + ret = dat_ep_post_recv( h_ep, > + 1, > + &l_iov, > + cookie, > + DAT_COMPLETION_DEFAULT_FLAG ); > + > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error registering recv msg buffer: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d Registered Receive Message Buffer %p\n", > + getpid(),region.for_va); > + > + } > + > + /* setup receive rdma buffer to initial string to be overwritten */ > + strcpy( (char*)rbuf, "blah, blah, blah\n" ); > + > + if ( server ) { /* SERVER */ > + > + /* create the service point for server listen */ > + LOGPRINTF("%d Creating service point for listen\n",getpid()); > + ret = dat_psp_create( h_ia, > + conn_id, > + h_cr_evd, > + DAT_PSP_CONSUMER_FLAG, > + &h_psp ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_psp_create: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d dat_psp_created for server listen\n", getpid()); > + > + printf("%d Server waiting for connect request..\n", getpid()); > + ret = dat_evd_wait( h_cr_evd, SERVER_TIMEOUT, 1, &event, &nmore ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_evd_wait: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d dat_evd_wait for cr_evd completed\n", getpid()); > + > + if ( event.event_number != DAT_CONNECTION_REQUEST_EVENT ) { > + fprintf(stderr, "%d Error unexpected cr event : %s\n", > + > getpid(),DT_EventToSTr(event.event_number)); > + return( DAT_ABORT ); > + } > + if ( (event.event_data.cr_arrival_event_data.conn_qual != > SERVER_CONN_QUAL) || > + (event.event_data.cr_arrival_event_data.sp_handle.psp_handle != > h_psp) ) { > + fprintf(stderr, "%d Error wrong cr event data : %s\n", > + getpid(),DT_EventToSTr(event.event_number)); > + return( DAT_ABORT ); > + } > + > + /* accept connect request from client */ > + h_cr = event.event_data.cr_arrival_event_data.cr_handle; > + LOGPRINTF("%d Accepting connect request from client\n",getpid()); > + ret = dat_cr_accept( h_cr, h_ep, 0, (DAT_PVOID)0 ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_cr_accept: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d dat_cr_accept completed\n", getpid()); > + } > + else { /* CLIENT */ > + struct addrinfo *target; > + int rval; > + > + if (getaddrinfo (hostname, NULL, NULL, &target) != 0) { > + printf("\n remote name resolution failed!\n"); > + exit ( 1 ); > + } > + > + rval = ((struct sockaddr_in *)target->ai_addr)->sin_addr.s_addr; > + printf ("%d Server Name: %s \n", getpid(), hostname); > + printf ("%d Server Net Address: %d.%d.%d.%d\n", getpid(), > + (rval >> 0) & 0xff, > + (rval >> 8) & 0xff, > + (rval >> 16) & 0xff, > + (rval >> 24) & 0xff); > + > + remote_addr = *((DAT_IA_ADDRESS_PTR)target->ai_addr); > + > + LOGPRINTF("%d Connecting to server\n",getpid()); > + ret = dat_ep_connect( h_ep, > + &remote_addr, > + conn_id, > + CONN_TIMEOUT, > + 0, > + (DAT_PVOID)0, > + 0, > + DAT_CONNECT_DEFAULT_FLAG ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_ep_connect: %s\n", > + getpid(), DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d dat_ep_connect completed\n", getpid()); > + } > + > + printf("%d Waiting for connect response\n",getpid()); > + > + ret = dat_evd_wait( h_conn_evd, DAT_TIMEOUT_INFINITE, 1, &event, &nmore > ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_evd_wait: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d dat_evd_wait for h_conn_evd completed\n", > getpid()); > + > + if ( event.event_number != DAT_CONNECTION_EVENT_ESTABLISHED ) { > + fprintf(stderr, "%d Error unexpected conn event : %s\n", > + getpid(),DT_EventToSTr(event.event_number)); > + return( DAT_ABORT ); > + } > + printf("\n%d CONNECTED!\n\n",getpid()); > + connected = 1; > + > + /* > + * Setup our remote memory and tell the other side about it > + */ > + rmr_send_msg.rmr_context = rmr_context_recv; > + rmr_send_msg.pad = 0; > + rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf; > + rmr_send_msg.segment_length = RDMA_BUFFER_SIZE; > + > + printf("%d Send RMR to remote: snd_msg: > r_key_ctx=%x,pad=%x,va=%llx,len=0x%x\n", > + getpid(), rmr_send_msg.rmr_context, rmr_send_msg.pad, > + rmr_send_msg.target_address, rmr_send_msg.segment_length ); > + > + ret = send_msg( &rmr_send_msg, > + sizeof( DAT_RMR_TRIPLET ), > + lmr_context_send_msg, > + cookie, > + DAT_COMPLETION_SUPPRESS_FLAG ); > + > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error send_msg: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else > + LOGPRINTF("%d send_msg completed\n", getpid()); > + > + /* > + * Wait for remote RMR information for RDMA > + */ > + if ( polling ) { > + printf("%d Polling for remote to send RMR data\n",getpid()); > + while ( dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY ); > + } > + else { > + printf("%d Waiting for remote to send RMR data\n",getpid()); > + if (use_cno) > + { > + DAT_EVD_HANDLE evd = DAT_HANDLE_NULL; > + ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd ); > + LOGPRINTF("%d cno wait return evd_handle=%p\n", getpid(),evd); > + if ( evd != h_dto_evd ) { > + fprintf(stderr, > + "%d Error waiting on h_dto_cno: evd != h_dto_evd\n", > + getpid()); > + return( DAT_ABORT ); > + } > + } > + /* use wait to dequeue */ > + ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error waiting on h_dto_evd: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else { > + LOGPRINTF("%d dat_evd_wait h_dto_evd completed\n", getpid()); > + } > + } > + > + printf("%d remote RMR data arrived!\n",getpid()); > + > + if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) { > + fprintf(stderr, "%d Error unexpected DTO event : %s\n", > + getpid(),DT_EventToSTr(event.event_number)); > + return( DAT_ABORT ); > + } > + if ((event.event_data.dto_completion_event_data.transfered_length != > + sizeof( DAT_RMR_TRIPLET )) || > + (event.event_data.dto_completion_event_data.user_cookie.as_64 != > + recv_msg_index) ) { > + fprintf(stderr,"ERR recv event: len=%d cookie=%d expected %d/%d\n", > + > (int)event.event_data.dto_completion_event_data.transfered_length, > + > (int)event.event_data.dto_completion_event_data.user_cookie.as_64, > + sizeof(DAT_RMR_TRIPLET), recv_msg_index ); > + return( DAT_ABORT ); > + } > + > + r_iov = rmr_recv_msg[ recv_msg_index ]; > + > + printf("%d Received RMR from remote: r_iov: > r_key_ctx=%x,pad=%x,va=%llx,len=0x%x\n", > + getpid(), r_iov.rmr_context, r_iov.pad, > + r_iov.target_address, r_iov.segment_length ); > + > + recv_msg_index++; > + > + return ( DAT_SUCCESS ); > +} > + > + > +void > +disconnect_ep() > +{ > + DAT_RETURN ret; > + DAT_EVENT event; > + DAT_COUNT nmore; > + int i,flush_cnt; > + > + if (connected) { > + > + LOGPRINTF("%d dat_ep_disconnect\n", getpid()); > + ret = dat_ep_disconnect( h_ep, DAT_CLOSE_DEFAULT ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_ep_disconnect: %s\n", > + getpid(),DT_RetToString(ret)); > + } > + else { > + LOGPRINTF("%d dat_ep_disconnect completed\n", getpid()); > + } > + } > + > + /* destroy service point */ > + if (( server ) && ( h_psp != DAT_HANDLE_NULL )) { > + ret = dat_psp_free( h_psp ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_psp_free: %s\n", > + getpid(),DT_RetToString(ret)); > + } > + else { > + LOGPRINTF("%d dat_psp_free completed\n", getpid()); > + } > + } > + > + /* Unregister Send message Buffer */ > + if ( h_lmr_send_msg != DAT_HANDLE_NULL ) { > + LOGPRINTF("%d Unregister send message h_lmr %p > \n",getpid(),h_lmr_send_msg); > + ret = dat_lmr_free(h_lmr_send_msg); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error deregistering send msg mr: %s\n", > + getpid(), DT_RetToString(ret)); > + } else { > + LOGPRINTF("%d Unregistered send message Buffer\n",getpid()); > + h_lmr_send_msg = NULL; > + } > + } > + > + /* Unregister recv message Buffer */ > + if ( h_lmr_recv_msg != DAT_HANDLE_NULL ) { > + LOGPRINTF("%d Unregister recv message h_lmr %p > \n",getpid(),h_lmr_recv_msg); > + ret = dat_lmr_free(h_lmr_recv_msg); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error deregistering recv msg mr: %s\n", > + getpid(), DT_RetToString(ret)); > + } else { > + LOGPRINTF("%d Unregistered recv message Buffer\n",getpid()); > + h_lmr_recv_msg = NULL; > + } > + } > + return; > +} > + > + > +DAT_RETURN > +do_rdma_write_with_msg( ) > +{ > + DAT_REGION_DESCRIPTION region; > + DAT_EVENT event; > + DAT_COUNT nmore; > + DAT_LMR_TRIPLET l_iov; > + DAT_RMR_TRIPLET r_iov; > + DAT_DTO_COOKIE cookie; > + DAT_RMR_CONTEXT their_context; > + DAT_RETURN ret; > + int i; > + > + printf("\n %d RDMA WRITE DATA with SEND MSG\n\n",getpid()); > + > + cookie.as_64 = 0x5555; > + > + if ( recv_msg_index >= MSG_BUF_COUNT ) > + return( DAT_ABORT ); > + > + /* get RMR information from previously received message */ > + r_iov = rmr_recv_msg[ recv_msg_index-1 ]; > + > + if ( server ) > + strcpy( (char*)sbuf, "server written data..." ); > + else > + strcpy( (char*)sbuf, "client written data..." ); > + > + l_iov.lmr_context = lmr_context_send; > + l_iov.pad = 0; > + l_iov.virtual_address = (DAT_VADDR)(unsigned long)sbuf; > + l_iov.segment_length = buf_len; > + > + start = get_time(); > + for (i=0;i + cookie.as_64 = 0x9999; > + ret = dat_ep_post_rdma_write( h_ep, // ep_handle > + 1, // > num_segments > + &l_iov, // LMR > + cookie, // > user_cookie > + &r_iov, // RMR > + DAT_COMPLETION_SUPPRESS_FLAG ); > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: ERROR: dat_ep_post_rdma_write() %s\n", > + getpid(),DT_RetToString(ret)); > + return( DAT_ABORT ); > + } > + LOGPRINTF("%d rdma_write # %d completed\n", getpid(),i+1); > + } > + > + /* > + * Send RMR information a 2nd time to indicate completion > + */ > + rmr_send_msg.rmr_context = rmr_context_recv; > + rmr_send_msg.pad = 0; > + rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf; > + rmr_send_msg.segment_length = RDMA_BUFFER_SIZE; > + > + printf("%d Sending completion message\n",getpid()); > + > + ret = send_msg( &rmr_send_msg, > + sizeof( DAT_RMR_TRIPLET ), > + lmr_context_send_msg, > + cookie, > + DAT_COMPLETION_SUPPRESS_FLAG ); > + > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error send_msg: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } else { > + LOGPRINTF("%d send_msg completed\n", getpid()); > + } > + > + /* > + * Collect first event, write completion or the inbound recv with > immed > + */ > + if ( polling ) { > + while ( dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY > ) > + rdma_wr_poll_count++; > + } > + else { > + LOGPRINTF("%d waiting for message receive event\n", getpid()); > + if (use_cno) { > + DAT_EVD_HANDLE evd = DAT_HANDLE_NULL; > + ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd ); > + LOGPRINTF("%d cno wait return evd_handle=%p\n", > getpid(),evd); > + if ( evd != h_dto_evd ) { > + fprintf(stderr, "%d Error waiting on h_dto_cno: > evd != h_dto_evd\n", > + getpid()); > + return( ret ); > + } > + } > + /* use wait to dequeue */ > + ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore ); > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", > + getpid(),DT_RetToString(ret)); > + return( ret ); > + } > + } > + stop = get_time(); > + time.rdma_wr = ((stop - start)*1.0e6); > + > + /* validate event number and status */ > + printf("%d inbound rdma_write; send message arrived!\n",getpid()); > + if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) { > + fprintf(stderr, "%d Error unexpected DTO event : %s\n", > + getpid(),DT_EventToSTr(event.event_number)); > + return( DAT_ABORT ); > + } > + > + if ( (event.event_data.dto_completion_event_data.transfered_length != > sizeof( DAT_RMR_TRIPLET )) || > + (event.event_data.dto_completion_event_data.user_cookie.as_64 != > recv_msg_index) ) { > + > + fprintf(stderr,"unexpected event data for receive: len=%d > cookie=%d exp %d/%d\n", > + > (int)event.event_data.dto_completion_event_data.transfered_length, > + > (int)event.event_data.dto_completion_event_data.user_cookie.as_64, > + sizeof(DAT_RMR_TRIPLET), recv_msg_index ); > + > + return( DAT_ABORT ); > + } > + > + r_iov = rmr_recv_msg[ recv_msg_index ]; > + > + printf("%d Received RMR from remote: r_iov: > ctx=%x,pad=%x,va=%p,len=0x%x\n", > + getpid(), r_iov.rmr_context, > + r_iov.pad, > + (void*)(unsigned long)r_iov.target_address, > + r_iov.segment_length ); > + > + LOGPRINTF("%d inbound rdma_write; send msg event SUCCESS!!!\n", > getpid()); > + > + printf("%d %s RDMA write buffer contains: %s\n", > + getpid(), > + server ? "SERVER:" : "CLIENT:", > + rbuf ); > + > + recv_msg_index++; > + > + return ( DAT_SUCCESS ); > +} > + > +DAT_RETURN > +do_rdma_read_with_msg( ) > +{ > + DAT_REGION_DESCRIPTION region; > + DAT_EVENT event; > + DAT_COUNT nmore; > + DAT_LMR_TRIPLET l_iov; > + DAT_RMR_TRIPLET r_iov; > + DAT_DTO_COOKIE cookie; > + DAT_RMR_CONTEXT their_context; > + DAT_RETURN ret; > + int i; > + > + printf("\n %d RDMA READ DATA with SEND MSG\n\n",getpid()); > + > + cookie.as_64 = 0x5555; > + > + if ( recv_msg_index >= MSG_BUF_COUNT ) > + return( DAT_ABORT ); > + > + /* get RMR information from previously received message */ > + r_iov = rmr_recv_msg[ recv_msg_index-1 ]; > + > + /* setup rdma read buffer to initial string to be overwritten */ > + strcpy( (char*)sbuf, "blah, blah, blah\n" ); > + > + if ( server ) > + strcpy( (char*)rbuf, "server read data..." ); > + else > + strcpy( (char*)rbuf, "client read data..." ); > + > + l_iov.lmr_context = lmr_context_send; > + l_iov.pad = 0; > + l_iov.virtual_address = (DAT_VADDR)(unsigned long)sbuf; > + l_iov.segment_length = buf_len; > + > + start = get_time(); > + for (i=0;i + cookie.as_64 = 0x9999; > + ret = dat_ep_post_rdma_read( h_ep, // ep_handle > + 1, // > num_segments > + &l_iov, // LMR > + cookie, // > user_cookie > + &r_iov, // RMR > + DAT_COMPLETION_SUPPRESS_FLAG ); > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: ERROR: dat_ep_post_rdma_read() %s\n", > + getpid(),DT_RetToString(ret)); > + return( DAT_ABORT ); > + } > + LOGPRINTF("%d rdma_read # %d completed\n", getpid(),i+1); > + } > + > + /* > + * Send RMR information a 2nd time to indicate completion > + */ > + rmr_send_msg.rmr_context = rmr_context_recv; > + rmr_send_msg.pad = 0; > + rmr_send_msg.target_address = (DAT_VADDR)(unsigned long)rbuf; > + rmr_send_msg.segment_length = RDMA_BUFFER_SIZE; > + > + printf("%d Sending completion message\n",getpid()); > + > + ret = send_msg( &rmr_send_msg, > + sizeof( DAT_RMR_TRIPLET ), > + lmr_context_send_msg, > + cookie, > + DAT_COMPLETION_SUPPRESS_FLAG ); > + > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error send_msg: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } else { > + LOGPRINTF("%d send_msg completed\n", getpid()); > + } > + > + /* > + * Collect first event, write completion or the inbound recv with > immed > + */ > + printf("%d Waiting for inbound message....\n",getpid()); > + if ( polling ) { > + while ( dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY > ) > + rdma_rd_poll_count++; > + } > + else { > + LOGPRINTF("%d waiting for message receive event\n", getpid()); > + if (use_cno) { > + DAT_EVD_HANDLE evd = DAT_HANDLE_NULL; > + ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd ); > + LOGPRINTF("%d cno wait return evd_handle=%p\n", > getpid(),evd); > + if ( evd != h_dto_evd ) { > + fprintf(stderr, "%d Error waiting on h_dto_cno: > evd != h_dto_evd\n", > + getpid()); > + return( ret ); > + } > + } > + /* use wait to dequeue */ > + ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore ); > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", > + getpid(),DT_RetToString(ret)); > + return( ret ); > + } > + } > + stop = get_time(); > + time.rdma_rd = ((stop - start)*1.0e6); > + > + /* validate event number and status */ > + printf("%d inbound rdma_read; send message arrived!\n",getpid()); > + if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) { > + fprintf(stderr, "%d Error unexpected DTO event : %s\n", > + getpid(),DT_EventToSTr(event.event_number)); > + return( DAT_ABORT ); > + } > + > + if ( (event.event_data.dto_completion_event_data.transfered_length != > sizeof( DAT_RMR_TRIPLET )) || > + (event.event_data.dto_completion_event_data.user_cookie.as_64 != > recv_msg_index) ) { > + > + fprintf(stderr,"unexpected event data for receive: len=%d > cookie=%d exp %d/%d\n", > + > (int)event.event_data.dto_completion_event_data.transfered_length, > + > (int)event.event_data.dto_completion_event_data.user_cookie.as_64, > + sizeof(DAT_RMR_TRIPLET), recv_msg_index ); > + > + return( DAT_ABORT ); > + } > + > + r_iov = rmr_recv_msg[ recv_msg_index ]; > + > + printf("%d Received RMR from remote: r_iov: > ctx=%x,pad=%x,va=%p,len=0x%x\n", > + getpid(), r_iov.rmr_context, r_iov.pad, > + (void*)(unsigned long)r_iov.target_address, > r_iov.segment_length ); > + > + LOGPRINTF("%d inbound rdma_write; send msg event SUCCESS!!!\n", > getpid()); > + > + printf("%d %s RCV RDMA read buffer contains: %s\n", > + getpid(), > + server ? "SERVER:" : "CLIENT:", > + sbuf ); > + > + recv_msg_index++; > + > + return ( DAT_SUCCESS ); > +} > + > + > +DAT_RETURN > +do_ping_pong_msg( ) > +{ > + DAT_EVENT event; > + DAT_COUNT nmore; > + DAT_DTO_COOKIE cookie; > + DAT_LMR_TRIPLET l_iov; > + DAT_RETURN ret; > + int i; > + unsigned char *snd_buf; > + unsigned char *rcv_buf; > + > + printf("\n %d PING DATA with SEND MSG\n\n",getpid()); > + > + snd_buf = sbuf; > + rcv_buf = rbuf; > + > + /* pre-post all buffers */ > + for ( i=0; i < burst; i++ ) { > + burst_msg_posted++; > + cookie.as_64 = i; > + l_iov.lmr_context = lmr_context_recv; > + l_iov.pad = 0; > + l_iov.virtual_address = (DAT_VADDR)(unsigned long)rcv_buf; > + l_iov.segment_length = buf_len; > + > + LOGPRINTF("%d Pre-posting Receive Message Buffers %p\n", > + getpid(), rcv_buf ); > + > + ret = dat_ep_post_recv( h_ep, > + 1, > + &l_iov, > + cookie, > + DAT_COMPLETION_DEFAULT_FLAG ); > + > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error posting recv msg buffer: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else { > + LOGPRINTF("%d Posted Receive Message Buffer %p\n", > + getpid(),rcv_buf); > + } > + > + /* next buffer */ > + rcv_buf += buf_len; > + } > + sleep(1); > + > + /* Initialize recv_buf and index to beginning */ > + rcv_buf = rbuf; > + burst_msg_index=0; > + > + /* client ping 0x55, server pong 0xAA in first byte */ > + start = get_time(); > + for ( i=0;i + /* walk the send and recv buffers */ > + if ( !server ) { > + *snd_buf = 0x55; > + > + LOGPRINTF("%d %s SND buffer %p contains: 0x%x len=%d\n", > + getpid(), server ? "SERVER:" : "CLIENT:", > + snd_buf, *snd_buf, buf_len ); > + > + ret = send_msg( snd_buf, > + buf_len, > + lmr_context_send, > + cookie, > + DAT_COMPLETION_SUPPRESS_FLAG ); > + > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error send_msg: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else { > + LOGPRINTF("%d send_msg completed\n", getpid()); > + } > + } > + > + /* Wait for recv message */ > + if ( polling ) { > + poll_count=0; > + LOGPRINTF("%d Polling for message receive event\n", getpid()); > + while ( dat_evd_dequeue( h_dto_evd, &event ) == DAT_QUEUE_EMPTY > ) > + poll_count++; > + } > + else { > + LOGPRINTF("%d waiting for message receive event\n", getpid()); > + if (use_cno) { > + DAT_EVD_HANDLE evd = DAT_HANDLE_NULL; > + ret = dat_cno_wait( h_dto_cno, DTO_TIMEOUT, &evd ); > + LOGPRINTF("%d cno wait return evd_handle=%p\n", > getpid(),evd); > + if ( evd != h_dto_evd ) > + { > + fprintf(stderr, "%d Error waiting on h_dto_cno: evd != > h_dto_evd\n", > + getpid()); > + return( ret ); > + } > + } > + /* use wait to dequeue */ > + ret = dat_evd_wait( h_dto_evd, DTO_TIMEOUT, 1, &event, &nmore ); > + if (ret != DAT_SUCCESS) { > + fprintf(stderr, "%d: ERROR: DTO dat_evd_wait() %s\n", > + getpid(),DT_RetToString(ret)); > + return( ret ); > + } > + } > + /* start timer after first message arrives on server */ > + if ( i == 0) { > + start = get_time(); > + } > + /* validate event number and status */ > + LOGPRINTF("%d inbound message; message arrived!\n",getpid()); > + if ( event.event_number != DAT_DTO_COMPLETION_EVENT ) { > + fprintf(stderr, "%d Error unexpected DTO event : %s\n", > + getpid(),DT_EventToSTr(event.event_number)); > + return( DAT_ABORT ); > + } > + if ((event.event_data.dto_completion_event_data.transfered_length > + != buf_len) || > + (event.event_data.dto_completion_event_data.user_cookie.as_64 > + != burst_msg_index) ) { > + fprintf(stderr,"ERR: recv event: len=%d cookie=%d exp %d/%d\n", > + > (int)event.event_data.dto_completion_event_data.transfered_length, > + > (int)event.event_data.dto_completion_event_data.user_cookie.as_64, > + buf_len, burst_msg_index ); > + > + return( DAT_ABORT ); > + } > + > + LOGPRINTF("%d %s RCV buffer %p contains: 0x%x len=%d\n", > + getpid(), server ? "SERVER:" : "CLIENT:", > + rcv_buf, *rcv_buf, buf_len ); > + > + burst_msg_index++; > + > + /* If server, change data and send it back to client */ > + if ( server ) { > + *snd_buf = 0xaa; > + > + LOGPRINTF("%d %s SND buffer %p contains: 0x%x len=%d\n", > + getpid(), server ? "SERVER:" : "CLIENT:", > + snd_buf, *snd_buf, buf_len ); > + > + ret = send_msg( snd_buf, > + buf_len, > + lmr_context_send, > + cookie, > + DAT_COMPLETION_SUPPRESS_FLAG ); > + > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error send_msg: %s\n", > + getpid(),DT_RetToString(ret)); > + return(ret); > + } > + else { > + LOGPRINTF("%d send_msg completed\n", getpid()); > + } > + } > + > + /* next buffers */ > + rcv_buf += buf_len; > + snd_buf += buf_len; > + } > + stop = get_time(); > + time.rtt = ((stop - start)*1.0e6); > + > + return ( DAT_SUCCESS ); > +} > + > +/* Register RDMA Receive buffer */ > +DAT_RETURN > +register_rdma_memory(void) > +{ > + DAT_RETURN ret; > + DAT_REGION_DESCRIPTION region; > + > + region.for_va = rbuf; > + start = get_time(); > + ret = dat_lmr_create( h_ia, > + DAT_MEM_TYPE_VIRTUAL, > + region, > + buf_len*burst, > + h_pz, > + DAT_MEM_PRIV_ALL_FLAG, > + &h_lmr_recv, > + &lmr_context_recv, > + &rmr_context_recv, > + ®istered_size_recv, > + ®istered_addr_recv ); > + stop = get_time(); > + time.reg += ((stop - start)*1.0e6); > + time.total += time.reg; > + > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error registering recv buffer: %s\n", > + getpid(),DT_RetToString(ret)); > + return (ret); > + } else { > + LOGPRINTF("%d Registered Receive RDMA Buffer %p\n", > + getpid(),region.for_va); > + } > + > + /* Register RDMA Send buffer */ > + region.for_va = sbuf; > + ret = dat_lmr_create( h_ia, > + DAT_MEM_TYPE_VIRTUAL, > + region, > + buf_len*burst, > + h_pz, > + DAT_MEM_PRIV_ALL_FLAG, > + &h_lmr_send, > + &lmr_context_send, > + &rmr_context_send, > + ®istered_size_send, > + ®istered_addr_send ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error registering send RDMA buffer: %s\n", > + getpid(),DT_RetToString(ret)); > + return (ret); > + } else { > + LOGPRINTF("%d Registered Send RDMA Buffer %p\n", > + getpid(),region.for_va); > + } > + > + return DAT_SUCCESS; > +} > + > +/* > + * Unregister RDMA memory > + */ > +DAT_RETURN > +unregister_rdma_memory(void) > +{ > + DAT_RETURN ret; > + > + /* Unregister Recv Buffer */ > + if ( h_lmr_recv != DAT_HANDLE_NULL ) { > + LOGPRINTF("%d Unregister h_lmr %p \n",getpid(),h_lmr_recv); > + start = get_time(); > + ret = dat_lmr_free(h_lmr_recv); > + stop = get_time(); > + time.unreg += ((stop - start)*1.0e6); > + time.total += time.unreg; > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error deregistering recv mr: %s\n", > + getpid(), DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d Unregistered Recv Buffer\n",getpid()); > + h_lmr_recv = NULL; > + } > + } > + > + /* Unregister Send Buffer */ > + if ( h_lmr_send != DAT_HANDLE_NULL ) { > + LOGPRINTF("%d Unregister h_lmr %p \n",getpid(),h_lmr_send); > + ret = dat_lmr_free(h_lmr_send); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error deregistering send mr: %s\n", > + getpid(), DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d Unregistered send Buffer\n",getpid()); > + h_lmr_send = NULL; > + } > + } > + return DAT_SUCCESS; > +} > + > + /* > + * Create CNO, CR, CONN, and DTO events > + */ > +DAT_RETURN > +create_events(void) > +{ > + DAT_RETURN ret; > + > + /* create CNO */ > + if (use_cno) { > + start = get_time(); > + ret = dat_cno_create( h_ia, DAT_OS_WAIT_PROXY_AGENT_NULL, &h_dto_cno > ); > + stop = get_time(); > + time.cnoc += ((stop - start)*1.0e6); > + time.total += time.cnoc; > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_cno_create: %s\n", > + getpid(),DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d cr_evd created, %p\n", getpid(), h_dto_cno); > + } > + } > + > + /* create cr EVD */ > + start = get_time(); > + ret = dat_evd_create( h_ia, 10, DAT_HANDLE_NULL, DAT_EVD_CR_FLAG, > &h_cr_evd ); > + stop = get_time(); > + time.evdc += ((stop - start)*1.0e6); > + time.total += time.evdc; > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_evd_create: %s\n", > + getpid(),DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d cr_evd created %p\n", getpid(),h_cr_evd); > + } > + /* create conn EVD */ > + ret = dat_evd_create( h_ia, 10, DAT_HANDLE_NULL, > DAT_EVD_CONNECTION_FLAG, &h_conn_evd ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_evd_create: %s\n", > + getpid(),DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d con_evd created %p\n", getpid(),h_conn_evd); > + } > + > + /* create dto EVD, with CNO if use_cno was set */ > + ret = dat_evd_create( h_ia, > + (MSG_BUF_COUNT*2)+burst*2, > + h_dto_cno, > + DAT_EVD_DTO_FLAG, > + &h_dto_evd ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error dat_evd_create: %s\n", > + getpid(),DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d dto_evd created %p\n", getpid(), h_dto_evd ); > + } > + > + return DAT_SUCCESS; > +} > + > +/* > + * Destroy CR, CONN, CNO, and DTO events > + */ > +DAT_RETURN > +destroy_events(void) > +{ > + DAT_RETURN ret; > + > + /* free cr EVD */ > + if ( h_cr_evd != DAT_HANDLE_NULL ) { > + LOGPRINTF("%d Free cr EVD %p \n",getpid(),h_cr_evd); > + ret = dat_evd_free( h_cr_evd ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error freeing cr EVD: %s\n", > + getpid(), DT_RetToString(ret)); > + return (ret); > + } else { > + LOGPRINTF("%d Freed cr EVD\n",getpid()); > + h_cr_evd = DAT_HANDLE_NULL; > + } > + } > + > + /* free conn EVD */ > + if ( h_conn_evd != DAT_HANDLE_NULL ) { > + LOGPRINTF("%d Free conn EVD %p \n",getpid(),h_conn_evd); > + ret = dat_evd_free( h_conn_evd ); > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error freeing conn EVD: %s\n", > + getpid(), DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d Freed conn EVD\n",getpid()); > + h_conn_evd = DAT_HANDLE_NULL; > + } > + } > + > + /* free dto EVD */ > + if ( h_dto_evd != DAT_HANDLE_NULL ) { > + LOGPRINTF("%d Free dto EVD %p \n",getpid(),h_dto_evd); > + start = get_time(); > + ret = dat_evd_free( h_dto_evd ); > + stop = get_time(); > + time.evdf += ((stop - start)*1.0e6); > + time.total += time.evdf; > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error freeing dto EVD: %s\n", > + getpid(), DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d Freed dto EVD\n",getpid()); > + h_dto_evd = DAT_HANDLE_NULL; > + } > + } > + > + /* free CNO */ > + if ( h_dto_cno != DAT_HANDLE_NULL ) { > + LOGPRINTF("%d Free dto CNO %p \n",getpid(),h_dto_cno); > + start = get_time(); > + ret = dat_cno_free( h_dto_cno ); > + stop = get_time(); > + time.cnof += ((stop - start)*1.0e6); > + time.total += time.cnof; > + if(ret != DAT_SUCCESS) { > + fprintf(stderr, "%d Error freeing dto CNO: %s\n", > + getpid(), DT_RetToString(ret)); > + return (ret); > + } > + else { > + LOGPRINTF("%d Freed dto CNO\n",getpid()); > + h_dto_cno = DAT_HANDLE_NULL; > + } > + } > + return DAT_SUCCESS; > +} > + > +/* > + * Map DAT_RETURN values to readable strings, > + * but don't assume the values are zero-based or contiguous. > + */ > +char errmsg[512] = {0}; > +const char * > +DT_RetToString (DAT_RETURN ret_value) > +{ > + const char *major_msg, *minor_msg; > + int sz; > + > + dat_strerror (ret_value, &major_msg, &minor_msg); > + > + strcpy(errmsg, major_msg); > + strcat(errmsg, " "); > + strcat(errmsg, minor_msg); > + > + return errmsg; > +} > + > +/* > + * Map DAT_EVENT_CODE values to readable strings > + */ > +const char * > +DT_EventToSTr (DAT_EVENT_NUMBER event_code) > +{ > + unsigned int i; > + static struct { > + const char *name; > + DAT_RETURN value; > + } > + dat_events[] = > + { > + # define DATxx(x) { # x, x } > + DATxx (DAT_DTO_COMPLETION_EVENT), > + DATxx (DAT_RMR_BIND_COMPLETION_EVENT), > + DATxx (DAT_CONNECTION_REQUEST_EVENT), > + DATxx (DAT_CONNECTION_EVENT_ESTABLISHED), > + DATxx (DAT_CONNECTION_EVENT_PEER_REJECTED), > + DATxx (DAT_CONNECTION_EVENT_NON_PEER_REJECTED), > + DATxx (DAT_CONNECTION_EVENT_ACCEPT_COMPLETION_ERROR), > + DATxx (DAT_CONNECTION_EVENT_DISCONNECTED), > + DATxx (DAT_CONNECTION_EVENT_BROKEN), > + DATxx (DAT_CONNECTION_EVENT_TIMED_OUT), > + DATxx (DAT_CONNECTION_EVENT_UNREACHABLE), > + DATxx (DAT_ASYNC_ERROR_EVD_OVERFLOW), > + DATxx (DAT_ASYNC_ERROR_IA_CATASTROPHIC), > + DATxx (DAT_ASYNC_ERROR_EP_BROKEN), > + DATxx (DAT_ASYNC_ERROR_TIMED_OUT), > + DATxx (DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR), > + DATxx (DAT_SOFTWARE_EVENT) > + # undef DATxx > + }; > + # define NUM_EVENTS (sizeof(dat_events)/sizeof(dat_events[0])) > + > + for (i = 0; i < NUM_EVENTS; i++) { > + if (dat_events[i].value == event_code) > + { > + return ( dat_events[i].name ); > + } > + } > + > + return ( "Invalid_DAT_EVENT_NUMBER" ); > +} > + > + > +void print_usage() > +{ > + printf("\n DAPL USAGE \n\n"); > + printf("s: server\n"); > + printf("c: use cno\n"); > + printf("v: verbose\n"); > + printf("p: polling\n"); > + printf("d: delay before close\n"); > + printf("b: buf length to allocate\n"); > + printf("B: burst count, rdma and msgs \n"); > + printf("h: hostname\n"); > + printf("\n"); > +} > + > Index: dtest/dat.conf > =================================================================== > --- dtest/dat.conf (revision 0) > +++ dtest/dat.conf (revision 0) > @@ -0,0 +1,11 @@ > +# > +# DAT 1.1 and 1.2 configuration file > +# > +# Each entry should have the following fields: > +# > +# \ > +# > +# > +# Example for openib using the first Mellanox adapter, port 1 and port 2 > + > +IB1 u1.2 nonthreadsafe default > /home/ardavis/Proj/openib/gen2/users/jlentini/userspace/dapl/udapl/Target/libdapl.so > r.1.2 "mthca0 1" "" > Index: dtest/README > =================================================================== > --- dtest/README (revision 0) > +++ dtest/README (revision 0) > @@ -0,0 +1,19 @@ > +simple dapl test just for initial openIB uDAPL testing... > + > + dtest/dtest.c > + dtest/makefile > + dtest/dat.conf > + > +to build (default uDAPL name == IB1, ib device == mthca0, port == 1) > + edit makefile and change path (DAT_LIB) to appropriate libdat.so > + edit dat.conf and change path to appropriate libdapl.so > + cp dat.conf to /etc/dat.conf > + > +to run: > + server: dtest > + client: dtest -h hostname > + > +for verbose uDAPL and uDAT debug: > + > + export DAPL_DBG_TYPE=0xffff > + export DAT_DBG_TYPE=0xffff > Index: dtest/makefile > =================================================================== > --- dtest/makefile (revision 0) > +++ dtest/makefile (revision 0) > @@ -0,0 +1,16 @@ > +CC = gcc > +CFLAGS = -O2 -g > + > +DAT_INC = ../dat/include > +DAT_LIB = /usr/lib64 > + > +all: dtest > + > +clean: > + rm -f *.o;touch *.c;rm -f dtest > + > +dtest: ./dtest.c > + $(CC) $(CFLAGS) ./dtest.c -o dtest \ > + -DDAPL_PROVIDER='"IB1"' \ > + -I $(DAT_INC) -L $(DAT_LIB) -ldat > + > From halr at voltaire.com Tue Jun 28 14:09:36 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 17:09:36 -0400 Subject: [openib-general] Re: [openib-commits] r2736 - gen2/trunk/src/linux-kernel/infiniband/hw/mthca In-Reply-To: <20050628211705.410AD22834D@openib.ca.sandia.gov> References: <20050628211705.410AD22834D@openib.ca.sandia.gov> Message-ID: <1119992976.4848.2652.camel@hal.voltaire.com> On Tue, 2005-06-28 at 17:17, roland at openib.org wrote: > Medium projects (well understood but require a fair amount of code): > MW support: ib_mthca does not support memory windows. > - FMR support: ib_mthca does not support Mellanox-style "fast > - memory regions" as used by SDP and SRP. > SRQ support: ib_mthca does not support shared receive queues. Should SMR support be added to this list ? -- Hal From jlentini at netapp.com Tue Jun 28 14:15:53 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 17:15:53 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [udapl] fix build for x86_64 In-Reply-To: <42C1B569.1020603@ichips.intel.com> References: <42C07A5D.5000201@ichips.intel.com> <42C1B569.1020603@ichips.intel.com> Message-ID: Committed in revision 2739. On Tue, 28 Jun 2005, Arlin Davis wrote: ardavis> James Lentini wrote: ardavis> ardavis> > ardavis> > Arlin, ardavis> > ardavis> > Would this be a more standard way to do the cast: ardavis> ardavis> yes, I agree. ardavis> ardavis> > ardavis> > On Mon, 27 Jun 2005, Arlin Davis wrote: ardavis> > ardavis> > > Signed-off-by: Arlin Davis ardavis> > > ardavis> > > Index: dapl/openib/dapl_ib_dto.h ardavis> > > =================================================================== ardavis> > > --- dapl/openib/dapl_ib_dto.h (revision 2720) ardavis> > > +++ dapl/openib/dapl_ib_dto.h (working copy) ardavis> > > @@ -88,7 +88,7 @@ ardavis> > > total_len = 0; ardavis> > > wr.next = 0; ardavis> > > wr.num_sge = 0; ardavis> > > - wr.wr_id = (uint64_t)cookie; ardavis> > > + wr.wr_id = (uint64_t)(unsigned long)cookie; ardavis> > ardavis> > ardavis> > wr.wr_id = (uint64_t)(uintptr_t)cookie; ardavis> > ardavis> > > wr.sg_list = ds_array_p; ardavis> > > ardavis> > > for (i = 0; i < segments; i++ ) { ardavis> > > @@ -162,7 +162,7 @@ ardavis> > > wr.opcode = op_type; ardavis> > > wr.num_sge = 0; ardavis> > > wr.send_flags = 0; ardavis> > > - wr.wr_id = (uint64_t)cookie; ardavis> > > + wr.wr_id = (uint64_t)(unsigned long)cookie; ardavis> > ardavis> > ardavis> > wr.wr_id = (uint64_t)(uintptr_t)cookie; ardavis> > ardavis> > > wr.sg_list = ds_array_p; ardavis> > > total_len = 0; ardavis> > > ardavis> > ardavis> From halr at voltaire.com Tue Jun 28 14:13:52 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 17:13:52 -0400 Subject: [openib-general] [PATCH] user_mad: In send_handler, only return header on timeout Message-ID: <1119993231.4848.2661.camel@hal.voltaire.com> user_mad: In send_handler, only return header on timeout Signed-off-by: Hal Rosenstock Index: user_mad.c =================================================================== --- user_mad.c (revision 2731) +++ user_mad.c (working copy) @@ -139,19 +139,30 @@ struct ib_mad_send_wc *send_wc) { struct ib_umad_file *file = agent->context; - struct ib_umad_packet *packet = + struct ib_umad_packet *timeout, *packet = (void *) (unsigned long) send_wc->wr_id; ib_destroy_ah(packet->msg->send_wr.wr.ud.ah); ib_free_send_mad(packet->msg); if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) { - packet->mad.hdr.status = ETIMEDOUT; + timeout = kmalloc(sizeof *timeout + sizeof (struct ib_mad_hdr), + GFP_KERNEL); + if (!timeout) + goto out; - if (!queue_packet(file, agent, packet)) - return; + memset(timeout, 0, sizeof *timeout + sizeof (struct ib_mad_hdr)); + + timeout->length = sizeof (struct ib_mad_hdr); + timeout->mad.hdr.id = packet->mad.hdr.id; + timeout->mad.hdr.status = ETIMEDOUT; + memcpy(timeout->mad.data, packet->mad.data, + sizeof (struct ib_mad_hdr)); + + if (!queue_packet(file, agent, timeout)) + return; } - +out: kfree(packet); } From krause at cup.hp.com Tue Jun 28 14:11:41 2005 From: krause at cup.hp.com (Michael Krause) Date: Tue, 28 Jun 2005 14:11:41 -0700 Subject: [openib-general] mapping between IP address and device name In-Reply-To: <52vf43hdc1.fsf@topspin.com> References: <52vf48wf5s.fsf@topspin.com> <1119359005.4476.2373.camel@localhost.localdomain> <527jgloz2d.fsf@topspin.com> <1119542588.4477.2.camel@localhost.localdomain> <52ekatm13h.fsf@topspin.com> <6.2.1.2.2.20050624094520.050243d0@exnane01.nane.netapp.com> <524qbnk9rm.fsf@topspin.com> <6.2.1.2.2.20050624122823.042722f0@exnane01.nane.netapp.com> <52vf43iu4i.fsf@topspin.com> <6.2.1.2.2.20050624132030.06040730@exnane01.nane.netapp.com> <52vf43hdc1.fsf@topspin.com> Message-ID: <6.2.0.14.2.20050628140837.028b2888@esmail.cup.hp.com> At 10:30 AM 6/24/2005, Roland Dreier wrote: > Thomas> As I said - I am not attached to ATS. I would welcome an > Thomas> alternative. > >Sure, understood. I'm suggesting a slight tweak to the IB wire >protocol. I don't think there's a difference in the security >provided, and carrying the peer address in the CM private data avoids >a lot of the conceptual and implementation difficulties of ATS. > > Thomas> But in the absence of one, I like what we have. Also, I do > Thomas> not want to saddle the NFS/RDMA transport with carrying an > Thomas> IP address purely for the benefit of a missing transport > Thomas> facility. After all NFS/RDMA works on iWARP too. > >I'm not sure I understand this objection. We wouldn't be saddling the >transport with anything -- simply specifying in the binding of >NFS/RDMA to IB that certain information is carried in the private data >fields of the CM messages used to establish a connection. Clearly >iWARP would use its own mechanism for providing the peer address. > >This would be exactly analogous to the situation for SDP -- obviously >SDP running on iWARP does not use the IB CM to exchange IP address >information in the same way the SDP over IB does. Actually, SDP on iWARP uses the SDP port mapper protocol to comprehend the IP address / port tuples used on both sides of the communication before the connection is established (this protocol could be used by any mapping service since it is implemented on top of UDP so could be re-used by other subsystems like NFS. The TCP transport then connects normally and one can ask it for the IP address / port tuple that is really being used. Port mapper may be viewed as akin to the SID protocol defined for IB. The SDP hello is then exchange in byte stream as opposed to IB CM. The port mapper supports both centrally managed and distributed usage models, supports the ability to return diff IP address than requested, support multiple IP addresses per port, etc. One can construct a very flexible infrastructure that supports nearly any type of mapping one desires to same or different hardware or endnodes. It is fairly light weight and can support caching of data for a period of time or even a one-shot connection attempt. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Tue Jun 28 14:17:13 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 17:17:13 -0400 Subject: [openib-general] Re: [openib-commits] r2736 - gen2/trunk/src/linux-kernel/infiniband/hw/mthca In-Reply-To: <20050628211705.410AD22834D@openib.ca.sandia.gov> References: <20050628211705.410AD22834D@openib.ca.sandia.gov> Message-ID: <1119993372.4848.2670.camel@hal.voltaire.com> On Tue, 2005-06-28 at 17:17, roland at openib.org wrote: > Medium projects (well understood but require a fair amount of code): > MW support: ib_mthca does not support memory windows. > - FMR support: ib_mthca does not support Mellanox-style "fast > - memory regions" as used by SDP and SRP. > SRQ support: ib_mthca does not support shared receive queues. Should SMR support be added to this list ? -- Hal From caitlin.bestler at gmail.com Tue Jun 28 14:25:35 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 28 Jun 2005 14:25:35 -0700 Subject: [openib-general] Re: IP addressing on InfiniBand networks In-Reply-To: <20050628204646.GB5987@mellanox.co.il> References: <20050628204646.GB5987@mellanox.co.il> Message-ID: <469958e005062814256176c527@mail.gmail.com> On 6/28/05, Michael S. Tsirkin wrote: > Hi, James! > > I dont know much about dapl, so forgive me if the question is naive: > > Quoting r. James Lentini : > > > > + CM Private Data > > > > The active side of an IB connection could place its source IP > > address in the CM's private data. The passive side would retrieve > > the source IP from this location. > > > > ... > > > > The security of this is very week. An end node could easily present > > a false IP address. > > Once you have the IP from CM private data, what prevents you from resolving it > back to hardware address (by sending an ARP request with the IP address that > you got)? > > You get back the IPoIB hardware address: GID+QPN, and can verify that > the GID matches the GID that you got from CM. > > The security of this seems to be at least as good as the one you get on > regular IP networks. > > Does this make sense at all? > The CM private data is private. It is not supposed to be interpreted by the Provider, only by the application. The essential issue here is how you validate an L3 network address. DAT defines it as having been done, however, before it is delivered to the applicaition. I haven't seen an argument yet as to why this goal is not achievable without changing the API -- which works perfectly well over IP networks. From halr at voltaire.com Tue Jun 28 14:22:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 17:22:17 -0400 Subject: [openib-general] [PATCH] sa_query: Add service record support Message-ID: <1119993488.4848.2676.camel@hal.voltaire.com> sa_query: Add service record support Signed-off-by: Hal Rosenstock Index: include/ib_sa.h =================================================================== -- include/ib_sa.h (revision 2732) +++ include/ib_sa.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -193,6 +194,61 @@ int proxy_join; }; +/* Service Record Component Mask Sec 15.2.5.14 Ver 1.1 */ +#define IB_SR_COMPMASK_SID IB_SA_COMP_MASK(0) +#define IB_SR_COMPMASK_SGID IB_SA_COMP_MASK(1) +#define IB_SR_COMPMASK_SPKEY IB_SA_COMP_MASK(2) +#define IB_SR_COMPMASK_RES1 IB_SA_COMP_MASK(3) +#define IB_SR_COMPMASK_SLEASE IB_SA_COMP_MASK(4) +#define IB_SR_COMPMASK_SKEY IB_SA_COMP_MASK(5) +#define IB_SR_COMPMASK_SNAME IB_SA_COMP_MASK(6) +#define IB_SR_COMPMASK_SDATA8_0 IB_SA_COMP_MASK(7) +#define IB_SR_COMPMASK_SDATA8_1 IB_SA_COMP_MASK(8) +#define IB_SR_COMPMASK_SDATA8_2 IB_SA_COMP_MASK(9) +#define IB_SR_COMPMASK_SDATA8_3 IB_SA_COMP_MASK(10) +#define IB_SR_COMPMASK_SDATA8_4 IB_SA_COMP_MASK(11) +#define IB_SR_COMPMASK_SDATA8_5 IB_SA_COMP_MASK(12) +#define IB_SR_COMPMASK_SDATA8_6 IB_SA_COMP_MASK(13) +#define IB_SR_COMPMASK_SDATA8_7 IB_SA_COMP_MASK(14) +#define IB_SR_COMPMASK_SDATA8_8 IB_SA_COMP_MASK(15) +#define IB_SR_COMPMASK_SDATA8_9 IB_SA_COMP_MASK(16) +#define IB_SR_COMPMASK_SDATA8_10 IB_SA_COMP_MASK(17) +#define IB_SR_COMPMASK_SDATA8_11 IB_SA_COMP_MASK(18) +#define IB_SR_COMPMASK_SDATA8_12 IB_SA_COMP_MASK(19) +#define IB_SR_COMPMASK_SDATA8_13 IB_SA_COMP_MASK(20) +#define IB_SR_COMPMASK_SDATA8_14 IB_SA_COMP_MASK(21) +#define IB_SR_COMPMASK_SDATA8_15 IB_SA_COMP_MASK(22) +#define IB_SR_COMPMASK_SDATA16_0 IB_SA_COMP_MASK(23) +#define IB_SR_COMPMASK_SDATA16_1 IB_SA_COMP_MASK(24) +#define IB_SR_COMPMASK_SDATA16_2 IB_SA_COMP_MASK(25) +#define IB_SR_COMPMASK_SDATA16_3 IB_SA_COMP_MASK(26) +#define IB_SR_COMPMASK_SDATA16_4 IB_SA_COMP_MASK(27) +#define IB_SR_COMPMASK_SDATA16_5 IB_SA_COMP_MASK(28) +#define IB_SR_COMPMASK_SDATA16_6 IB_SA_COMP_MASK(29) +#define IB_SR_COMPMASK_SDATA16_7 IB_SA_COMP_MASK(30) +#define IB_SR_COMPMASK_SDATA32_0 IB_SA_COMP_MASK(31) +#define IB_SR_COMPMASK_SDATA32_1 IB_SA_COMP_MASK(32) +#define IB_SR_COMPMASK_SDATA32_2 IB_SA_COMP_MASK(33) +#define IB_SR_COMPMASK_SDATA32_3 IB_SA_COMP_MASK(34) +#define IB_SR_COMPMASK_SDATA64_0 IB_SA_COMP_MASK(35) +#define IB_SR_COMPMASK_SDATA64_1 IB_SA_COMP_MASK(36) + +#define IB_DEFAULT_SERVICE_LEASE 0xFFFFFFFF + +struct ib_sa_service_rec { + u64 id; + union ib_gid gid; + u16 pkey; + /* u16 resv; */ + u32 lease; + u8 key[16]; + u8 name[64]; + u8 data8[16]; + u16 data16[8]; + u32 data32[4]; + u64 data64[2]; +}; + struct ib_sa_query; void ib_sa_cancel_query(int id, struct ib_sa_query *query); @@ -218,6 +274,17 @@ void *context, struct ib_sa_query **query); +int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, + u8 method, + struct ib_sa_service_rec *rec, + ib_sa_comp_mask comp_mask, + int timeout_ms, int gfp_mask, + void (*callback)(int status, + struct ib_sa_service_rec *resp, + void *context), + void *context, + struct ib_sa_query **sa_query); + /** * ib_sa_mcmember_rec_set - Start an MCMember set query * @device:device to send query on Index: core/sa_query.c =================================================================== -- core/sa_query.c (revision 2732) +++ core/sa_query.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -79,6 +80,12 @@ int id; }; +struct ib_sa_srvrec_query { + void (*callback)(int, struct ib_sa_service_rec *, void *); + void *context; + struct ib_sa_query sa_query; +}; + struct ib_sa_path_query { void (*callback)(int, struct ib_sa_path_rec *, void *); void *context; @@ -320,6 +327,54 @@ .size_bits = 23 }, }; +#define SERVICE_REC_FIELD(field) \ + .struct_offset_bytes = offsetof(struct ib_sa_service_rec, field), \ + .struct_size_bytes = sizeof ((struct ib_sa_service_rec *) 0)->field, \ + .field_name = "sa_service_rec:" #field + +static const struct ib_field service_rec_table[] = { + { SERVICE_REC_FIELD(id), + .offset_words = 0, + .offset_bits = 0, + .size_bits = 64 }, + { SERVICE_REC_FIELD(gid), + .offset_words = 2, + .offset_bits = 0, + .size_bits = 128 }, + { SERVICE_REC_FIELD(pkey), + .offset_words = 6, + .offset_bits = 0, + .size_bits = 16 }, + { SERVICE_REC_FIELD(lease), + .offset_words = 7, + .offset_bits = 0, + .size_bits = 32 }, + { SERVICE_REC_FIELD(key), + .offset_words = 8, + .offset_bits = 0, + .size_bits = 128 }, + { SERVICE_REC_FIELD(name), + .offset_words = 12, + .offset_bits = 0, + .size_bits = 64*8 }, + { SERVICE_REC_FIELD(data8), + .offset_words = 28, + .offset_bits = 0, + .size_bits = 16*8 }, + { SERVICE_REC_FIELD(data16), + .offset_words = 32, + .offset_bits = 0, + .size_bits = 8*16 }, + { SERVICE_REC_FIELD(data32), + .offset_words = 36, + .offset_bits = 0, + .size_bits = 4*32 }, + { SERVICE_REC_FIELD(data64), + .offset_words = 40, + .offset_bits = 0, + .size_bits = 2*64 }, +}; + static void free_sm_ah(struct kref *kref) { struct ib_sa_sm_ah *sm_ah = container_of(kref, struct ib_sa_sm_ah, ref); @@ -443,7 +498,6 @@ .remote_qpn = 1, .remote_qkey = IB_QP1_QKEY, .timeout_ms = timeout_ms, - .retries = 0 } } }; @@ -596,6 +650,114 @@ } EXPORT_SYMBOL(ib_sa_path_rec_get); +static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query, + int status, + struct ib_sa_mad *mad) +{ + struct ib_sa_srvrec_query *query = + container_of(sa_query, struct ib_sa_srvrec_query, sa_query); + + if (mad) { + struct ib_sa_service_rec rec; + + ib_unpack(service_rec_table, ARRAY_SIZE(service_rec_table), + mad->data, &rec); + query->callback(status, &rec, query->context); + } else + query->callback(status, NULL, query->context); +} + +static void ib_sa_service_rec_release(struct ib_sa_query *sa_query) +{ + kfree(sa_query->mad); + kfree(container_of(sa_query, struct ib_sa_srvrec_query, sa_query)); +} + +/** + * ib_sa_service_rec_query - Start Service Record operation + * @device:device to send request on + * @port_num: port number to send request on + * @method:SA method - should be get, set, or delete + * @rec:Service Record to send in request + * @comp_mask:component mask to send in request + * @timeout_ms:time to wait for response + * @gfp_mask:GFP mask to use for internal allocations + * @callback:function called when request completes, times out or is + * canceled + * @context:opaque user context passed to callback + * @sa_query:request context, used to cancel request + * + * Send a Service Record set/get/delete to the SA to register, + * unregister or query a service record. + * The callback function will be called when the request completes (or + * fails); status is 0 for a successful response, -EINTR if the query + * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error + * occurred sending the query. The resp parameter of the callback is + * only valid if status is 0. + * + * If the return value of ib_sa_service_rec_query() is negative, it is an + * error code. Otherwise it is a request ID that can be used to cancel + * the query. + */ +int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method, + struct ib_sa_service_rec *rec, + ib_sa_comp_mask comp_mask, + int timeout_ms, int gfp_mask, + void (*callback)(int status, + struct ib_sa_service_rec *resp, + void *context), + void *context, + struct ib_sa_query **sa_query) +{ + struct ib_sa_srvrec_query *query; + struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client); + struct ib_sa_port *port = &sa_dev->port[port_num - sa_dev->start_port]; + struct ib_mad_agent *agent = port->agent; + int ret; + + if (method != IB_MGMT_METHOD_GET && + method != IB_MGMT_METHOD_SET && + method != IB_SA_METHOD_DELETE) + return -EINVAL; + + query = kmalloc(sizeof *query, gfp_mask); + if (!query) + return -ENOMEM; + query->sa_query.mad = kmalloc(sizeof *query->sa_query.mad, gfp_mask); + if (!query->sa_query.mad) { + kfree(query); + return -ENOMEM; + } + + query->callback = callback; + query->context = context; + + init_mad(query->sa_query.mad, agent); + + query->sa_query.callback = ib_sa_service_rec_callback; + query->sa_query.release = ib_sa_service_rec_release; + query->sa_query.port = port; + query->sa_query.mad->mad_hdr.method = method; + query->sa_query.mad->mad_hdr.attr_id = + cpu_to_be16(IB_SA_ATTR_SERVICE_REC); + query->sa_query.mad->sa_hdr.comp_mask = comp_mask; + + ib_pack(service_rec_table, ARRAY_SIZE(service_rec_table), + rec, query->sa_query.mad->data); + + *sa_query = &query->sa_query; + + ret = send_mad(&query->sa_query, timeout_ms); + if (ret < 0) { + *sa_query = NULL; + kfree(query->sa_query.mad); + kfree(query); + } + + return ret; +} +EXPORT_SYMBOL(ib_sa_service_rec_query); + static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query, int status, struct ib_sa_mad *mad) From mst at mellanox.co.il Tue Jun 28 14:30:18 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 29 Jun 2005 00:30:18 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <52hdfith08.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> <52psu6uzkv.fsf@topspin.com> <20050628191710.GC6442@mellanox.co.il> <52hdfith08.fsf@topspin.com> Message-ID: <20050628213018.GD5987@mellanox.co.il> Quoting r. Roland Dreier : > OK, this looks fine. Applied (with slight renames of function names). > > Thanks, > Roland > Now that ib_umem_release is always called in the context of a system call, we dont have to get_task_mm, since current->mm is not going anywere. Signed-off-by: Michael S. Tsirkin Index: infiniband/core/uverbs_mem.c =================================================================== --- infiniband/core/uverbs_mem.c (revision 2740) +++ infiniband/core/uverbs_mem.c (working copy) @@ -174,14 +174,10 @@ void ib_umem_release(struct ib_device *d __ib_umem_release(dev, umem, 1); - mm = get_task_mm(current); - if (!mm) - return; - + mm = current->mm; down_write(&mm->mmap_sem); mm->locked_vm -= PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; up_write(&mm->mmap_sem); - mmput(mm); } static void ib_umem_account(void *work_ptr) -- MST From rolandd at cisco.com Tue Jun 28 14:35:54 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 14:35:54 -0700 Subject: [openib-general] Re: [openib-commits] r2736 - gen2/trunk/src/linux-kernel/infiniband/hw/mthca In-Reply-To: <1119992976.4848.2652.camel@hal.voltaire.com> (Hal Rosenstock's message of "28 Jun 2005 17:09:36 -0400") References: <20050628211705.410AD22834D@openib.ca.sandia.gov> <1119992976.4848.2652.camel@hal.voltaire.com> Message-ID: <52zmtanozp.fsf@topspin.com> Hal> Should SMR support be added to this list ? Sure, I just did that. - R. From mst at mellanox.co.il Tue Jun 28 14:34:37 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 29 Jun 2005 00:34:37 +0300 Subject: [openib-general] Re: [PATCH] sa_query: Add service record support In-Reply-To: <1119993488.4848.2676.camel@hal.voltaire.com> References: <1119993488.4848.2676.camel@hal.voltaire.com> Message-ID: <20050628213437.GE5987@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: [PATCH] sa_query: Add service record support > > sa_query: Add service record support > > Signed-off-by: Hal Rosenstock > > ... > > +struct ib_sa_srvrec_query { > + void (*callback)(int, struct ib_sa_service_rec *, void *); > + void *context; > + struct ib_sa_query sa_query; > +}; > + Can this be renamed to struct ib_sa_service_rec_query, please? Or does it mean something else? -- MST From jlentini at netapp.com Tue Jun 28 14:40:45 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 17:40:45 -0400 (EDT) Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <20050628192649.GA9340@lst.de> References: <20050628192649.GA9340@lst.de> Message-ID: That would require systems to be configured using IB addresses. Can that be made feasible in non-trivial configurations? For example, how could name resolution be made scalable? If you want to resolve a hostname like foo.bar.com to an IB GID, then information about IB GIDs needs to added to the various name resolution systems (/etc/hosts, NIS, DNS,...). Adding GIDs to these databases doesn't seem like a prudent thing to do. GIDs come in different flavors (adapter specific and subnet manager assigned) none of which appear to be good candidates for large scale management. On Tue, 28 Jun 2005, Christoph Hellwig wrote: hch> On Tue, Jun 28, 2005 at 03:24:35PM -0400, James Lentini wrote: hch> > hch> > I'd like to summarize the discussion we've been having around hch> > addressing and start a new email thread with a more appropriate title. hch> > hch> > First off, here is there requirement we are trying to satisfy: hch> > hch> > kDAPL consumers use an Internet Protocol (IP) addresses to hch> > identify remote nodes in an interoperable way. hch> hch> I don't think that does belong into the kernel. It's fine to do that in hch> userspace if you want. hch> From jlentini at netapp.com Tue Jun 28 14:43:53 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 17:43:53 -0400 (EDT) Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <52br5qtgk9.fsf@topspin.com> References: <52br5qtgk9.fsf@topspin.com> Message-ID: On Tue, 28 Jun 2005, Roland Dreier wrote: > James> First off, here [are the] requirement we are trying to satisfy: > > James> On the passive side of a connection, a InfiniBand kDAPL > James> provider must determine a source IB address for an > James> InfiniBand connection request. This information can be > James> obtain by a kDAPL consumer either in the > James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or > James> dat_cr_query()'s dat_cr_param structure. > > James> By interoperable, we mean that the solution must not > James> introduce a non-standard protocol or force ULPs using kDAPL > James> to perform special operations when using an InfiniBand > James> network. > > Since these two points are mutually contradictory Which two points? The ability to determine a source IB address for an InfiniBand connection request and interoperability? I don't think those two are contradictory by definition. > -- the IB communication management protocol does not carry enough > information for a connection request to be mapped uniquely back to a > source address -- we need to figure out which one to drop. > > I would argue in favor of the solution selected by SDP: when defining > the binding of an abstract protocol to the IB transport, put the > source and destination IP addresses in the IB-specific connection > setup messages. > > - R. > From halr at voltaire.com Tue Jun 28 14:42:33 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 Jun 2005 17:42:33 -0400 Subject: [openib-general] Re: [PATCH] sa_query: Add service record support In-Reply-To: <20050628213437.GE5987@mellanox.co.il> References: <1119993488.4848.2676.camel@hal.voltaire.com> <20050628213437.GE5987@mellanox.co.il> Message-ID: <1119994950.4848.2707.camel@hal.voltaire.com> On Tue, 2005-06-28 at 17:34, Michael S. Tsirkin wrote: > Quoting r. Hal Rosenstock : > > Subject: [PATCH] sa_query: Add service record support > > > > sa_query: Add service record support > > > > Signed-off-by: Hal Rosenstock > > > > ... > > > > +struct ib_sa_srvrec_query { > > + void (*callback)(int, struct ib_sa_service_rec *, void *); > > + void *context; > > + struct ib_sa_query sa_query; > > +}; > > + > > Can this be renamed to struct ib_sa_service_rec_query, please? > Or does it mean something else? Sure but ib_sa_service_query would be more in keeping. I'll wait to see what other comments come back before posting an updated patch. -- Hal From rolandd at cisco.com Tue Jun 28 14:51:20 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 14:51:20 -0700 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <20050628213018.GD5987@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 29 Jun 2005 00:30:18 +0300") References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> <52psu6uzkv.fsf@topspin.com> <20050628191710.GC6442@mellanox.co.il> <52hdfith08.fsf@topspin.com> <20050628213018.GD5987@mellanox.co.il> Message-ID: <52ll4uno9z.fsf@topspin.com> Thanks, committed. From mst at mellanox.co.il Tue Jun 28 14:50:40 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 29 Jun 2005 00:50:40 +0300 Subject: [openib-general] Re: [PATCH] sa_query: Add service record support In-Reply-To: <1119994950.4848.2707.camel@hal.voltaire.com> References: <1119993488.4848.2676.camel@hal.voltaire.com> <20050628213437.GE5987@mellanox.co.il> <1119994950.4848.2707.camel@hal.voltaire.com> Message-ID: <20050628215040.GG5987@mellanox.co.il> Quoting r. Hal Rosenstock : > > Can this be renamed to struct ib_sa_service_rec_query, please? > > Or does it mean something else? > > Sure but ib_sa_service_query would be more in keeping. The functions are called "ib_sa_service_rec_query", I guess you'll rename that too? -- MST From rolandd at cisco.com Tue Jun 28 14:52:41 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 14:52:41 -0700 Subject: [openib-general] Re: [PATCH] sa_query: Add service record support In-Reply-To: <20050628213437.GE5987@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 29 Jun 2005 00:34:37 +0300") References: <1119993488.4848.2676.camel@hal.voltaire.com> <20050628213437.GE5987@mellanox.co.il> Message-ID: <52hdfino7q.fsf@topspin.com> Michael> Can this be renamed to struct ib_sa_service_rec_query, Michael> please? Or does it mean something else? Makes sense. The style I tried to follow in the SA client module was that everything was named the same was as the IBA spec, except that StudlyCaps are replaced with underscores (ie StudlyCaps -> studly_caps). - R. From rolandd at cisco.com Tue Jun 28 14:54:37 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 14:54:37 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: (James Lentini's message of "Tue, 28 Jun 2005 17:43:53 -0400 (EDT)") References: <52br5qtgk9.fsf@topspin.com> Message-ID: <52aclano4i.fsf@topspin.com> James> Which two points? The ability to determine a source IB James> address for an InfiniBand connection request and James> interoperability? James> I don't think those two are contradictory by definition. Yes, those two points I quoted. As I said, "the IB communication management protocol does not carry enough information for a connection request to be mapped uniquely back to a source address." So if you're not allowed to add anything to the connection establishment protocol, then you don't have enough information to find a source address. Something has to give. - R. From jlentini at netapp.com Tue Jun 28 14:57:14 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 17:57:14 -0400 (EDT) Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <1119989672.4848.2607.camel@hal.voltaire.com> References: <1119989672.4848.2607.camel@hal.voltaire.com> Message-ID: On Tue, 28 Jun 2005, Hal Rosenstock wrote: > On Tue, 2005-06-28 at 15:24, James Lentini wrote: >> + IPoIB >> >> IPoIB encapsulates IP packets in InfiniBand messages. There have been >> proposals to use the address resolution mechanisms in IPoIB to >> implement these features. IPv4 subnets use ARP and IPv6 subnets use >> Neighbor Discovery. >> >> Analysis: >> >> IPoIB is not free. All nodes would be need to implement it for >> this to work. >> >> The IB address -> IP address mapping on the passive side is >> problematic. If a reverse lookup were available, IPoIB would require >> both a GID and QP number as input. The passive side would know the GID >> but the QP number. >> >> Further more, reverse lookup is not well supported. On IPv4 subnets, >> RARP is quickly becoming (already?) obsolete. > > The IPoIB HW address includes the QPN (in addition to the GID). This is > also problematic. > >> Neighbor Discovery >> doesn't support reverse lookup at all. [RFC 2461] >> >> In addition to all this, IPoIB restricts an IP subnet to the same scope >> as an IB subnet. > > IPoIB does not limit an IP subnet to an IB subnet. It can span IB > subnets. However, IB routers were not completed in the IB architecture. I found this limitation in section 3.3 of the "IP over InfiniBand(IPoIB) Architecture" draft (April, 2004 version, http://www.ietf.org/internet-drafts/draft-ietf-ipoib-architecture-04.txt) Until the above conditions are met it is not possible to implement IPoIB subnets that span IB subnets. The IPoIB standards have however been defined with this possibility in mind. Am I looking at an old version of the spec? > >> If a kDAPL consumer desired to communicate between >> IB subnet's, IPoIB may not be sufficient. > > Are you referring to 2 disjoint IB subnets ? Yes. Your point is valid. If there are no routers, there is no reason to worry about it. > What about IB <-> iWARP ? I hadn't been considering that kind of communication. My assumption is that if a translator was created for IB <-> iWARP it would solve this issue...actually, a hypothetical translator is likely to use the solution we choose. >> + GID as an IPv6 Address >> >> See the attachment to Caitlin Bestler's email: >> >> http://openib.org/pipermail/openib-general/2005-June/008104.html >> >> Analysis: >> >> This has been the least discussed option. One issue is >> that GIDs may not be easy to administer. GIDs can be specific >> to a particular channel adapter since they can contain EUI-64 >> identifiers. Administrators avoid using Ethernet MAC addresses >> in configuration files and they should be able to avoid using >> adapter specific IB addresses as well. > > If they don't like ethernet MACs, they really won't like GUIDs/GIDs > as they are even longer. Length aside, GUIDs/GIDs are not manageable like IP addresses. > >> Another issue is how >> dynamically assigned SM GIDs would be managed. > > Do you mean SM (assigned additional) GUIDs ? No, I was referring to the SM assigned GIDs described in property 3c of section 4.1.1 of the IB spec. > > -- Hal > From jlentini at netapp.com Tue Jun 28 15:03:24 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 18:03:24 -0400 (EDT) Subject: [openib-general] Re: IP addressing on InfiniBand networks In-Reply-To: <20050628204646.GB5987@mellanox.co.il> References: <20050628204646.GB5987@mellanox.co.il> Message-ID: On Tue, 28 Jun 2005, Michael S. Tsirkin wrote: > Hi, James! > > I dont know much about dapl, so forgive me if the question is naive: > > Quoting r. James Lentini : >> >> + CM Private Data >> >> The active side of an IB connection could place its source IP >> address in the CM's private data. The passive side would retrieve >> the source IP from this location. >> >> ... >> >> The security of this is very week. An end node could easily present >> a false IP address. > > Once you have the IP from CM private data, what prevents you from resolving it > back to hardware address (by sending an ARP request with the IP address that > you got)? > > You get back the IPoIB hardware address: GID+QPN, and can verify that > the GID matches the GID that you got from CM. > > The security of this seems to be at least as good as the one you get on > regular IP networks. > > Does this make sense at all? It makes total sense. Your the first person to have pointed out this possibility. I agree with you that the security would be comparable to IP networks given IPoIB. From mst at mellanox.co.il Tue Jun 28 15:06:33 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 29 Jun 2005 01:06:33 +0300 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <52ll4uno9z.fsf@topspin.com> References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> <52psu6uzkv.fsf@topspin.com> <20050628191710.GC6442@mellanox.co.il> <52hdfith08.fsf@topspin.com> <20050628213018.GD5987@mellanox.co.il> <52ll4uno9z.fsf@topspin.com> Message-ID: <20050628220633.GA9216@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH] process locked in D state. > > Thanks, committed. > Hmm, but why are we still testing if (current->mm) ? I think current->mm is never NULL when we are inside a system call, except sys_exit. Am I wrong? -- MST From caitlin.bestler at gmail.com Tue Jun 28 15:06:38 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 28 Jun 2005 15:06:38 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: References: <20050628192649.GA9340@lst.de> Message-ID: <469958e00506281506350b5a88@mail.gmail.com> Site-local GIDs are just as manageable as site-local IPv6 addresses. In fact they look just like them to /etc/hosts, NIS, DNS, etc. The only real work required of the local network administrator(s) is that they assign *different* site-local network IDs for IPv6 and IB. There are 64K total available, so I doubt anyone is going to run out of available network IDs. On 6/28/05, James Lentini wrote: > > That would require systems to be configured using IB addresses. Can > that be made feasible in non-trivial configurations? > > For example, how could name resolution be made scalable? If you want > to resolve a hostname like foo.bar.com to an IB GID, then information > about IB GIDs needs to added to the various name resolution systems > (/etc/hosts, NIS, DNS,...). Adding GIDs to these databases doesn't > seem like a prudent thing to do. GIDs come in different flavors > (adapter specific and subnet manager assigned) none of which appear to > be good candidates for large scale management. > > On Tue, 28 Jun 2005, Christoph Hellwig wrote: > > hch> On Tue, Jun 28, 2005 at 03:24:35PM -0400, James Lentini wrote: > hch> > > hch> > I'd like to summarize the discussion we've been having around > hch> > addressing and start a new email thread with a more appropriate title. > hch> > > hch> > First off, here is there requirement we are trying to satisfy: > hch> > > hch> > kDAPL consumers use an Internet Protocol (IP) addresses to > hch> > identify remote nodes in an interoperable way. > hch> > hch> I don't think that does belong into the kernel. It's fine to do that in > hch> userspace if you want. > hch> > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From jlentini at netapp.com Tue Jun 28 15:10:49 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 18:10:49 -0400 (EDT) Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <469958e00506281506350b5a88@mail.gmail.com> References: <20050628192649.GA9340@lst.de> <469958e00506281506350b5a88@mail.gmail.com> Message-ID: Are there any tools available for a network administrator to assign a GID? Does OpenSM provider this capability? On Tue, 28 Jun 2005, Caitlin Bestler wrote: > Site-local GIDs are just as manageable as site-local IPv6 addresses. > In fact they look just like them to /etc/hosts, NIS, DNS, etc. > > The only real work required of the local network administrator(s) > is that they assign *different* site-local network IDs for IPv6 and IB. > There are 64K total available, so I doubt anyone is going to run out > of available network IDs. > > > On 6/28/05, James Lentini wrote: >> >> That would require systems to be configured using IB addresses. Can >> that be made feasible in non-trivial configurations? >> >> For example, how could name resolution be made scalable? If you want >> to resolve a hostname like foo.bar.com to an IB GID, then information >> about IB GIDs needs to added to the various name resolution systems >> (/etc/hosts, NIS, DNS,...). Adding GIDs to these databases doesn't >> seem like a prudent thing to do. GIDs come in different flavors >> (adapter specific and subnet manager assigned) none of which appear to >> be good candidates for large scale management. >> >> On Tue, 28 Jun 2005, Christoph Hellwig wrote: >> >> hch> On Tue, Jun 28, 2005 at 03:24:35PM -0400, James Lentini wrote: >> hch> > >> hch> > I'd like to summarize the discussion we've been having around >> hch> > addressing and start a new email thread with a more appropriate title. >> hch> > >> hch> > First off, here is there requirement we are trying to satisfy: >> hch> > >> hch> > kDAPL consumers use an Internet Protocol (IP) addresses to >> hch> > identify remote nodes in an interoperable way. >> hch> >> hch> I don't think that does belong into the kernel. It's fine to do that in >> hch> userspace if you want. >> hch> >> _______________________________________________ >> openib-general mailing list >> openib-general at openib.org >> http://openib.org/mailman/listinfo/openib-general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> > From jlentini at netapp.com Tue Jun 28 15:14:32 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 28 Jun 2005 18:14:32 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [uDAPL] Fix compile of dapl_bpool.c on x86_64 In-Reply-To: <1119972645.4848.2434.camel@hal.voltaire.com> References: <1119972645.4848.2434.camel@hal.voltaire.com> Message-ID: Hal, This brings up a good point. I need to get the uDAPL dapltest into the OpenIB tree. I'll work on doing that tomorrow. james On Tue, 28 Jun 2005, Hal Rosenstock wrote: halr> I was able to build udapltest for both x86 and x86-64 with the following halr> change: halr> halr> uDAPL: Fix compile of dapl_bpool.c on x86_64 halr> halr> Signed-off-by: Hal Rosenstock halr> halr> --- userspace/test/dapltest/test/dapl_bpool.c.orig 2005-01-18 halr> 13:20:31.000000000 -0500 halr> +++ userspace/test/dapltest/test/dapl_bpool.c 2005-06-24 halr> 23:47:43.000000000 -0400 halr> @@ -356,8 +356,8 @@ halr> "BPOOL alloc_size %x\n", halr> (int) bpool_ptr->alloc_size); halr> DT_Tdep_PT_Printf (phead, halr> - "BPOOL pz_handle %x\n", halr> - (int) bpool_ptr->pz_handle); halr> + "BPOOL pz_handle %p\n", halr> + bpool_ptr->pz_handle); halr> DT_Tdep_PT_Printf (phead, halr> "BPOOL num_segs %x\n", halr> (int) bpool_ptr->num_segs); halr> halr> From rolandd at cisco.com Tue Jun 28 15:15:57 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 15:15:57 -0700 Subject: [openib-general] Re: [PATCH] process locked in D state. In-Reply-To: <20050628220633.GA9216@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 29 Jun 2005 01:06:33 +0300") References: <20050627082745.GJ15312@minantech.com> <52fyv3z6yt.fsf@topspin.com> <20050628072222.GE25304@mellanox.co.il> <20050628075057.GM15312@minantech.com> <20050628092521.GG25304@mellanox.co.il> <52psu6uzkv.fsf@topspin.com> <20050628191710.GC6442@mellanox.co.il> <52hdfith08.fsf@topspin.com> <20050628213018.GD5987@mellanox.co.il> <52ll4uno9z.fsf@topspin.com> <20050628220633.GA9216@mellanox.co.il> Message-ID: <52mzpam8ki.fsf@topspin.com> Michael> Hmm, but why are we still testing if (current->mm) ? Michael> I think current->mm is never NULL when we are inside a Michael> system call, except sys_exit. Am I wrong? Good question. I don't think we need this. - R. From libor at topspin.com Tue Jun 28 15:32:38 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 28 Jun 2005 15:32:38 -0700 Subject: [openib-general] SDP sk_data_ready() callback In-Reply-To: <1119942921.8348.21.camel@confield>; from arne.redlich@xiranet.com on Tue, Jun 28, 2005 at 09:15:21AM +0200 References: <1119429583.8208.28.camel@confield> <20050627113521.B26240@topspin.com> <1119942921.8348.21.camel@confield> Message-ID: <20050628153238.F26240@topspin.com> On Tue, Jun 28, 2005 at 09:15:21AM +0200, Arne Redlich wrote: > Am Montag, den 27.06.2005, 11:35 -0700 schrieb Libor Michalek: > > On Wed, Jun 22, 2005 at 10:39:43AM +0200, Arne Redlich wrote: > > > Hi, > > > > > > I'm trying to use SDP from within the kernel. My problem is that the > > > code relies on sk_data_ready() (this callback is modified to wake up a > > > Rx thread before executing the original function), but sk_data_ready() > > > apparently never gets called. Is there any way to fix this? > > > > You mean that you replace sk->sk_data_ready, with a similar but > > slightly modified version and so rely on sk->sk_data_ready() being > > called? > > Yes, that's right. Actually the modifications are just minor as I > already pointed out: the replacement function just wakes the thread and > then calls the original sk_data_ready(). > > > You're right it's currently not being called, instead we're calling > > the function to which it's pointing directly, and for no real reason. > > Also, the function that's being called is a duplicate of the function > > sock_def_readable() in net/sock.c. I'll look at correcting these > > issues. Here's a patch which uses the socket readiness function pointers, as well as using the default readiness functions. Except for write readiness, (sk->sk_write_space) which currently uses different info to determine write space. -Libor Index: sdp_rcvd.c =================================================================== --- sdp_rcvd.c (revision 2731) +++ sdp_rcvd.c (working copy) @@ -40,6 +40,7 @@ */ static int sdp_rcvd_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff) { + struct sock *sk; int result = 0; int band; @@ -91,9 +92,11 @@ /* * async notification. POLL_HUP on full duplex close only. */ - sdp_inet_wake_generic(sk_sdp(conn)); - sk_wake_async(sk_sdp(conn), 1, band); + sk = sk_sdp(conn); + sk->sk_state_change(sk); + sk_wake_async(sk, 1, band); + return 0; error: return result; @@ -1027,6 +1030,7 @@ { sdp_event_cb_func dispatch_func; struct sdpc_buff *buff; + struct sock *sk; u32 offset; int result; @@ -1164,12 +1168,15 @@ result); goto drop; - } else + } + else { /* * If data was consumed by the protocol, signal * the user. */ - sdp_inet_wake_recv(sk_sdp(conn), conn->byte_strm); + sk = sk_sdp(conn); + sk->sk_data_ready(sk, conn->byte_strm); + } /* * It's possible that a new recv buffer advertisment opened up the * recv window and we can flush buffered send data Index: sdp_inet.c =================================================================== --- sdp_inet.c (revision 2732) +++ sdp_inet.c (working copy) @@ -132,9 +132,6 @@ { struct sdp_opt *conn = sdp_sk(sk); - if (!sk) - return; - if (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) && sdp_inet_writable(conn)) { read_lock(&sk->sk_callback_lock); @@ -152,64 +149,6 @@ } /* - * sdp_inet_wake_generic - wake up a socket - */ -void sdp_inet_wake_generic(struct sock *sk) -{ - if (sk) { - read_lock(&sk->sk_callback_lock); - - if (sk->sk_sleep && waitqueue_active(sk->sk_sleep)) - wake_up_interruptible_all(sk->sk_sleep); - - read_unlock(&sk->sk_callback_lock); - } -} - -/* - * sdp_inet_wake_recv - wake up a socket for read - */ -void sdp_inet_wake_recv(struct sock *sk, int len) -{ - if (sk) { - read_lock(&sk->sk_callback_lock); - if (sk->sk_sleep) - wake_up_interruptible(sk->sk_sleep); - - sk_wake_async(sk, 1, POLL_IN); - read_unlock(&sk->sk_callback_lock); - } -} - -/* - * sdp_inet_wake_error - wake up a socket for error - */ -void sdp_inet_wake_error(struct sock *sk) -{ - if (sk) { - read_lock(&sk->sk_callback_lock); - if (sk->sk_sleep) - wake_up_interruptible(sk->sk_sleep); - - sk_wake_async(sk, 0, POLL_ERR); - read_unlock(&sk->sk_callback_lock); - } -} - -/* - * sdp_inet_wake_urg - wake up a socket for urgent data - */ -void sdp_inet_wake_urg(struct sock *sk) -{ - /* - * pid for SIGURG/SIGIO has been set. On positive send signal to - * process, on negative send signal to processes group. - */ - if (sk) - sk_send_sigurg(sk); -} - -/* * sdp_inet_disconnect - disconnect a connection */ static int sdp_inet_disconnect(struct sdp_opt *conn) @@ -813,7 +752,7 @@ accept_conn->pid = current->pid; accept_sock->state = SS_CONNECTED; - sdp_inet_wake_send(accept_sk); + accept_sk->sk_write_space(accept_sk); break; default: @@ -1294,7 +1233,7 @@ break; } - sdp_inet_wake_generic(sock->sk); + sock->sk->sk_state_change(sock->sk); sdp_conn_unlock(conn); return result; } Index: sdp_proto.h =================================================================== --- sdp_proto.h (revision 2735) +++ sdp_proto.h (working copy) @@ -271,14 +271,6 @@ void sdp_inet_wake_send(struct sock *sk); -void sdp_inet_wake_generic(struct sock *sk); - -void sdp_inet_wake_recv(struct sock *sk, int len); - -void sdp_inet_wake_error(struct sock *sk); - -void sdp_inet_wake_urg(struct sock *sk); - /* * port/queue managment */ Index: sdp_read.c =================================================================== --- sdp_read.c (revision 2731) +++ sdp_read.c (working copy) @@ -110,6 +110,7 @@ { struct sdpc_iocb *iocb; struct sdpc_buff *buff; + struct sock *sk; s32 result; s32 type; @@ -183,7 +184,8 @@ */ conn->byte_strm += result; - sdp_inet_wake_recv(sk_sdp(conn), conn->byte_strm); + sk = sk_sdp(conn); + sk->sk_data_ready(sk, conn->byte_strm); } else { if (result < 0) sdp_dbg_warn(conn, "Error <%d> receiving buff", Index: sdp_send.c =================================================================== --- sdp_send.c (revision 2732) +++ sdp_send.c (working copy) @@ -1852,6 +1852,7 @@ */ int sdp_send_flush(struct sdp_opt *conn) { + struct sock *sk; int result = 0; /* @@ -1898,7 +1899,8 @@ /* * see if there is enough buffer to wake/notify writers */ - sdp_inet_wake_send(sk_sdp(conn)); + sk = sk_sdp(conn); + sk->sk_write_space(sk); return 0; done: Index: sdp_actv.c =================================================================== --- sdp_actv.c (revision 2732) +++ sdp_actv.c (working copy) @@ -98,7 +98,7 @@ sk->sk_socket->state = SS_UNCONNECTED; sdp_iocb_q_cancel_all(conn, error); - sdp_inet_wake_error(sk); + sk->sk_error_report(sk); } /* @@ -195,8 +195,8 @@ /* * write/read ready. (for those waiting on just one...) */ - sdp_inet_wake_send(sk); - sdp_inet_wake_recv(sk, 0); + sk->sk_write_space(sk); + sk->sk_data_ready(sk, 0); result = 0; done: Index: sdp_conn.c =================================================================== --- sdp_conn.c (revision 2735) +++ sdp_conn.c (working copy) @@ -97,7 +97,7 @@ sk->sk_socket->state = SS_UNCONNECTED; sdp_iocb_q_cancel_all(conn, error); - sdp_inet_wake_error(sk); + sk->sk_error_report(sk); } void sdp_conn_abort(struct sdp_opt *conn) @@ -1205,14 +1205,10 @@ */ sk->sk_protocol = IPPROTO_TCP; /* - * some callbacks. higher levels of linux (e.g. setsockopt) call - * these functions so they should be shim's to our functions... + * replace some callbacks from the standard functions. */ sk->sk_destruct = NULL; sk->sk_write_space = sdp_inet_wake_send; - sk->sk_state_change = sdp_inet_wake_generic; - sk->sk_data_ready = sdp_inet_wake_recv; - sk->sk_error_report = sdp_inet_wake_error; conn = sdp_sk(sk); SDP_CONN_ST_INIT(conn); Index: sdp_recv.c =================================================================== --- sdp_recv.c (revision 2732) +++ sdp_recv.c (working copy) @@ -755,7 +755,7 @@ */ if (buff->flags & SDP_BUFF_F_OOB_PEND) { conn->rcv_urg_cnt++; - sdp_inet_wake_urg(sk_sdp(conn)); + sk_send_sigurg(sk_sdp(conn)); } /* * loop while there are available IOCB's, break if there is no Index: sdp_pass.c =================================================================== --- sdp_pass.c (revision 2735) +++ sdp_pass.c (working copy) @@ -42,6 +42,7 @@ { struct ib_qp_attr *qp_attr; int attr_mask = 0; + struct sock *sk; int result; sdp_dbg_ctrl(conn, "Passive Establish src <%08x:%04x> dst <%08x:%04x>", @@ -87,7 +88,8 @@ goto error; } - sdp_inet_wake_send(sk_sdp(conn)); + sk = sk_sdp(conn); + sk->sk_write_space(sk); kfree(qp_attr); return 0; @@ -350,6 +352,7 @@ struct sdp_msg_hello *msg_hello = event->private_data; struct sdp_opt *listen_conn; struct sdp_opt *conn; + struct sock *sk; int result; u16 port; u32 addr; @@ -478,7 +481,9 @@ * place connection into the listen connections accept queue. */ sdp_inet_accept_q_put(listen_conn, conn); - sdp_inet_wake_recv(sk_sdp(listen_conn), 0); + + sk = sk_sdp(listen_conn); + sk->sk_data_ready(sk, 0); /* * unlock */ From rolandd at cisco.com Tue Jun 28 15:57:10 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 15:57:10 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: (James Lentini's message of "Tue, 28 Jun 2005 17:57:14 -0400 (EDT)") References: <1119989672.4848.2607.camel@hal.voltaire.com> Message-ID: <52psu6ks3d.fsf@topspin.com> James> I found this limitation in section 3.3 of the "IP over James> InfiniBand(IPoIB) Architecture" draft (April, 2004 version, James> http://www.ietf.org/internet-drafts/draft-ietf-ipoib-architecture-04.txt) James> Until the above conditions are met it is not James> possible to implement IPoIB subnets that span IB James> subnets. The IPoIB standards have however been defined with James> this possibility in mind. James> Am I looking at an old version of the spec? Those conditions are pretty much things that would have to be fixed for it to be possible to communicate between IB subnets at all. - R. From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 00/16] Add InfiniBand userspace verbs (direct userspace access) Message-ID: <2005628163.H103BaveG8tYGzYQ@cisco.com> Here is a series of patches that adds support for direct userspace access to InfiniBand hardware -- so-called "userspace verbs." I believe these patches are ready to merge, but a final review would be useful. These patches should incorporate all of the feedback from the discussion when I posted an earlier version back in April (see http://lkml.org/lkml/2005/4/4/267 for the start of the thread). In particular, memory pinned for use by userspace is accounted for in current->mm->vm_locked and requests to pin memory are checked against RLIMIT_MEMLOCK. Thanks, Roland From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 01/16] IB uverbs: core API extensions In-Reply-To: <2005628163.H103BaveG8tYGzYQ@cisco.com> Message-ID: <2005628163.dSvxYW7XXf2cvPdt@cisco.com> Modify the ib_verbs.h header file with changes required for InfiniBand userspace verbs support. We add a few structures to keep track of userspace context, and extend the driver API so that low-level drivers know when they're creating resources that will be used from userspace. Signed-off-by: Roland Dreier --- drivers/infiniband/include/ib_verbs.h | 124 +++++++++++++++++++++++++++++----- 1 files changed, 106 insertions(+), 18 deletions(-) --- linux.orig/drivers/infiniband/include/ib_verbs.h 2005-06-28 15:19:55.956779699 -0700 +++ linux/drivers/infiniband/include/ib_verbs.h 2005-06-28 15:19:57.390470064 -0700 @@ -4,6 +4,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -41,7 +42,10 @@ #include #include + #include +#include +#include union ib_gid { u8 raw[16]; @@ -544,7 +548,7 @@ struct ib_send_wr { int num_sge; enum ib_wr_opcode opcode; int send_flags; - u32 imm_data; + __be32 imm_data; union { struct { u64 remote_addr; @@ -618,29 +622,86 @@ struct ib_fmr_attr { u8 page_size; }; +struct ib_ucontext { + struct ib_device *device; + struct list_head pd_list; + struct list_head mr_list; + struct list_head mw_list; + struct list_head cq_list; + struct list_head qp_list; + struct list_head srq_list; + struct list_head ah_list; + spinlock_t lock; +}; + +struct ib_uobject { + u64 user_handle; /* handle given to us by userspace */ + struct ib_ucontext *context; /* associated user context */ + struct list_head list; /* link to context's list */ + u32 id; /* index into kernel idr */ +}; + +struct ib_umem { + unsigned long user_base; + unsigned long virt_base; + size_t length; + int offset; + int page_size; + int writable; + struct list_head chunk_list; +}; + +struct ib_umem_chunk { + struct list_head list; + int nents; + int nmap; + struct scatterlist page_list[0]; +}; + +struct ib_udata { + void __user *inbuf; + void __user *outbuf; + size_t inlen; + size_t outlen; +}; + +#define IB_UMEM_MAX_PAGE_CHUNK \ + ((PAGE_SIZE - offsetof(struct ib_umem_chunk, page_list)) / \ + ((void *) &((struct ib_umem_chunk *) 0)->page_list[1] - \ + (void *) &((struct ib_umem_chunk *) 0)->page_list[0])) + +struct ib_umem_object { + struct ib_uobject uobject; + struct ib_umem umem; +}; + struct ib_pd { - struct ib_device *device; - atomic_t usecnt; /* count all resources */ + struct ib_device *device; + struct ib_uobject *uobject; + atomic_t usecnt; /* count all resources */ }; struct ib_ah { struct ib_device *device; struct ib_pd *pd; + struct ib_uobject *uobject; }; typedef void (*ib_comp_handler)(struct ib_cq *cq, void *cq_context); struct ib_cq { - struct ib_device *device; - ib_comp_handler comp_handler; - void (*event_handler)(struct ib_event *, void *); - void * cq_context; - int cqe; - atomic_t usecnt; /* count number of work queues */ + struct ib_device *device; + struct ib_uobject *uobject; + ib_comp_handler comp_handler; + void (*event_handler)(struct ib_event *, void *); + void * cq_context; + int cqe; + atomic_t usecnt; /* count number of work queues */ }; struct ib_srq { struct ib_device *device; + struct ib_uobject *uobject; struct ib_pd *pd; void *srq_context; atomic_t usecnt; @@ -652,6 +713,7 @@ struct ib_qp { struct ib_cq *send_cq; struct ib_cq *recv_cq; struct ib_srq *srq; + struct ib_uobject *uobject; void (*event_handler)(struct ib_event *, void *); void *qp_context; u32 qp_num; @@ -659,16 +721,18 @@ struct ib_qp { }; struct ib_mr { - struct ib_device *device; - struct ib_pd *pd; - u32 lkey; - u32 rkey; - atomic_t usecnt; /* count number of MWs */ + struct ib_device *device; + struct ib_pd *pd; + struct ib_uobject *uobject; + u32 lkey; + u32 rkey; + atomic_t usecnt; /* count number of MWs */ }; struct ib_mw { struct ib_device *device; struct ib_pd *pd; + struct ib_uobject *uobject; u32 rkey; }; @@ -737,7 +801,14 @@ struct ib_device { int (*modify_port)(struct ib_device *device, u8 port_num, int port_modify_mask, struct ib_port_modify *port_modify); - struct ib_pd * (*alloc_pd)(struct ib_device *device); + struct ib_ucontext * (*alloc_ucontext)(struct ib_device *device, + struct ib_udata *udata); + int (*dealloc_ucontext)(struct ib_ucontext *context); + int (*mmap)(struct ib_ucontext *context, + struct vm_area_struct *vma); + struct ib_pd * (*alloc_pd)(struct ib_device *device, + struct ib_ucontext *context, + struct ib_udata *udata); int (*dealloc_pd)(struct ib_pd *pd); struct ib_ah * (*create_ah)(struct ib_pd *pd, struct ib_ah_attr *ah_attr); @@ -747,7 +818,8 @@ struct ib_device { struct ib_ah_attr *ah_attr); int (*destroy_ah)(struct ib_ah *ah); struct ib_qp * (*create_qp)(struct ib_pd *pd, - struct ib_qp_init_attr *qp_init_attr); + struct ib_qp_init_attr *qp_init_attr, + struct ib_udata *udata); int (*modify_qp)(struct ib_qp *qp, struct ib_qp_attr *qp_attr, int qp_attr_mask); @@ -762,8 +834,9 @@ struct ib_device { int (*post_recv)(struct ib_qp *qp, struct ib_recv_wr *recv_wr, struct ib_recv_wr **bad_recv_wr); - struct ib_cq * (*create_cq)(struct ib_device *device, - int cqe); + struct ib_cq * (*create_cq)(struct ib_device *device, int cqe, + struct ib_ucontext *context, + struct ib_udata *udata); int (*destroy_cq)(struct ib_cq *cq); int (*resize_cq)(struct ib_cq *cq, int *cqe); int (*poll_cq)(struct ib_cq *cq, int num_entries, @@ -780,6 +853,10 @@ struct ib_device { int num_phys_buf, int mr_access_flags, u64 *iova_start); + struct ib_mr * (*reg_user_mr)(struct ib_pd *pd, + struct ib_umem *region, + int mr_access_flags, + struct ib_udata *udata); int (*query_mr)(struct ib_mr *mr, struct ib_mr_attr *mr_attr); int (*dereg_mr)(struct ib_mr *mr); @@ -817,6 +894,7 @@ struct ib_device { struct ib_mad *in_mad, struct ib_mad *out_mad); + struct module *owner; struct class_device class_dev; struct kobject ports_parent; struct list_head port_list; @@ -852,6 +930,16 @@ void *ib_get_client_data(struct ib_devic void ib_set_client_data(struct ib_device *device, struct ib_client *client, void *data); +static inline int ib_copy_from_udata(void *dest, struct ib_udata *udata, size_t len) +{ + return copy_from_user(dest, udata->inbuf, len) ? -EFAULT : 0; +} + +static inline int ib_copy_to_udata(struct ib_udata *udata, void *src, size_t len) +{ + return copy_to_user(udata->outbuf, src, len) ? -EFAULT : 0; +} + int ib_register_event_handler (struct ib_event_handler *event_handler); int ib_unregister_event_handler(struct ib_event_handler *event_handler); void ib_dispatch_event(struct ib_event *event); From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 12/16] IB uverbs: add mthca user PD support In-Reply-To: <2005628163.gtJFW6uLUrGQteys@cisco.com> Message-ID: <2005628163.px5sYyzsYWf21dJY@cisco.com> Add support for userspace protection domains (PDs) to mthca. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_dev.h | 3 ++- drivers/infiniband/hw/mthca/mthca_main.c | 2 +- drivers/infiniband/hw/mthca/mthca_pd.c | 24 +++++++++++++++--------- drivers/infiniband/hw/mthca/mthca_provider.c | 10 +++++++++- drivers/infiniband/hw/mthca/mthca_provider.h | 1 + 5 files changed, 28 insertions(+), 12 deletions(-) --- linux.orig/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-28 15:19:20.685397179 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-28 15:20:20.514467380 -0700 @@ -1,6 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -378,7 +379,7 @@ void mthca_unregister_device(struct mthc int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar); void mthca_uar_free(struct mthca_dev *dev, struct mthca_uar *uar); -int mthca_pd_alloc(struct mthca_dev *dev, struct mthca_pd *pd); +int mthca_pd_alloc(struct mthca_dev *dev, int privileged, struct mthca_pd *pd); void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd); struct mthca_mtt *mthca_alloc_mtt(struct mthca_dev *dev, int size); --- linux.orig/drivers/infiniband/hw/mthca/mthca_main.c 2005-06-28 15:19:20.685397179 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_main.c 2005-06-28 15:20:20.515467163 -0700 @@ -665,7 +665,7 @@ static int __devinit mthca_setup_hca(str goto err_pd_table_free; } - err = mthca_pd_alloc(dev, &dev->driver_pd); + err = mthca_pd_alloc(dev, 1, &dev->driver_pd); if (err) { mthca_err(dev, "Failed to create driver PD, " "aborting.\n"); --- linux.orig/drivers/infiniband/hw/mthca/mthca_pd.c 2005-06-28 15:19:20.684397395 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_pd.c 2005-06-28 15:20:20.513467597 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -37,23 +38,27 @@ #include "mthca_dev.h" -int mthca_pd_alloc(struct mthca_dev *dev, struct mthca_pd *pd) +int mthca_pd_alloc(struct mthca_dev *dev, int privileged, struct mthca_pd *pd) { - int err; + int err = 0; might_sleep(); + pd->privileged = privileged; + atomic_set(&pd->sqp_count, 0); pd->pd_num = mthca_alloc(&dev->pd_table.alloc); if (pd->pd_num == -1) return -ENOMEM; - err = mthca_mr_alloc_notrans(dev, pd->pd_num, - MTHCA_MPT_FLAG_LOCAL_READ | - MTHCA_MPT_FLAG_LOCAL_WRITE, - &pd->ntmr); - if (err) - mthca_free(&dev->pd_table.alloc, pd->pd_num); + if (privileged) { + err = mthca_mr_alloc_notrans(dev, pd->pd_num, + MTHCA_MPT_FLAG_LOCAL_READ | + MTHCA_MPT_FLAG_LOCAL_WRITE, + &pd->ntmr); + if (err) + mthca_free(&dev->pd_table.alloc, pd->pd_num); + } return err; } @@ -61,7 +66,8 @@ int mthca_pd_alloc(struct mthca_dev *dev void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd) { might_sleep(); - mthca_free_mr(dev, &pd->ntmr); + if (pd->privileged) + mthca_free_mr(dev, &pd->ntmr); mthca_free(&dev->pd_table.alloc, pd->pd_num); } --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:18.380930082 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:20.513467597 -0700 @@ -368,12 +368,20 @@ static struct ib_pd *mthca_alloc_pd(stru if (!pd) return ERR_PTR(-ENOMEM); - err = mthca_pd_alloc(to_mdev(ibdev), pd); + err = mthca_pd_alloc(to_mdev(ibdev), !context, pd); if (err) { kfree(pd); return ERR_PTR(err); } + if (context) { + if (ib_copy_to_udata(udata, &pd->pd_num, sizeof (__u32))) { + mthca_pd_free(to_mdev(ibdev), pd); + kfree(pd); + return ERR_PTR(-EFAULT); + } + } + return &pd->ibpd; } --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.h 2005-06-28 15:20:16.612313643 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.h 2005-06-28 15:20:20.514467380 -0700 @@ -92,6 +92,7 @@ struct mthca_pd { u32 pd_num; atomic_t sqp_count; struct mthca_mr ntmr; + int privileged; }; struct mthca_eq { From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 08/16] IB uverbs: add mthca ABI header In-Reply-To: <2005628163.3Q9FD2m0Zd9kmlVn@cisco.com> Message-ID: <2005628163.kTUk6BiHHfEXJhoz@cisco.com> Add the mthca_user.h header file, which defines the device-specific ABI used by the mthca low-level driver for kernel/user communication. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_user.h | 81 +++++++++++++++++++++++++++++++ 1 files changed, 81 insertions(+) --- /dev/null 2005-06-23 14:14:38.423479552 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_user.h 2005-06-28 15:20:10.937544281 -0700 @@ -0,0 +1,81 @@ +/* + * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#ifndef MTHCA_USER_H +#define MTHCA_USER_H + +#include + +/* + * Make sure that all structs defined in this file remain laid out so + * that they pack the same way on 32-bit and 64-bit architectures (to + * avoid incompatibility between 32-bit userspace and 64-bit kernels). + * In particular do not use pointer types -- pass pointers in __u64 + * instead. + */ + +struct mthca_alloc_ucontext_resp { + __u32 qp_tab_size; + __u32 uarc_size; +}; + +struct mthca_alloc_pd_resp { + __u32 pdn; + __u32 reserved; +}; + +struct mthca_create_cq { + __u32 lkey; + __u32 pdn; + __u64 arm_db_page; + __u64 set_db_page; + __u32 arm_db_index; + __u32 set_db_index; +}; + +struct mthca_create_cq_resp { + __u32 cqn; + __u32 reserved; +}; + +struct mthca_create_qp { + __u32 lkey; + __u32 reserved; + __u64 sq_db_page; + __u64 rq_db_page; + __u32 sq_db_index; + __u32 rq_db_index; +}; + +#endif /* MTHCA_USER_H */ From rolandd at cisco.com Tue Jun 28 16:03:44 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:44 -0700 Subject: [openib-general] [PATCH 13/16] IB uverbs: add mthca user MR support In-Reply-To: <2005628163.px5sYyzsYWf21dJY@cisco.com> Message-ID: <2005628163.XfYnZThlsTGb61Td@cisco.com> Add support for userspace memory regions (MRs) to mthca. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_provider.c | 82 +++++++++++++++++++++++++++ 1 files changed, 82 insertions(+) --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:20.513467597 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:23.354851384 -0700 @@ -654,6 +654,87 @@ static struct ib_mr *mthca_reg_phys_mr(s return &mr->ibmr; } +static struct ib_mr *mthca_reg_user_mr(struct ib_pd *pd, struct ib_umem *region, + int acc, struct ib_udata *udata) +{ + struct mthca_dev *dev = to_mdev(pd->device); + struct ib_umem_chunk *chunk; + struct mthca_mr *mr; + u64 *pages; + int shift, n, len; + int i, j, k; + int err = 0; + + shift = ffs(region->page_size) - 1; + + mr = kmalloc(sizeof *mr, GFP_KERNEL); + if (!mr) + return ERR_PTR(-ENOMEM); + + n = 0; + list_for_each_entry(chunk, ®ion->chunk_list, list) + n += chunk->nents; + + mr->mtt = mthca_alloc_mtt(dev, n); + if (IS_ERR(mr->mtt)) { + err = PTR_ERR(mr->mtt); + goto err; + } + + pages = (u64 *) __get_free_page(GFP_KERNEL); + if (!pages) { + err = -ENOMEM; + goto err_mtt; + } + + i = n = 0; + + list_for_each_entry(chunk, ®ion->chunk_list, list) + for (j = 0; j < chunk->nmap; ++j) { + len = sg_dma_len(&chunk->page_list[j]) >> shift; + for (k = 0; k < len; ++k) { + pages[i++] = sg_dma_address(&chunk->page_list[j]) + + region->page_size * k; + /* + * Be friendly to WRITE_MTT command + * and leave two empty slots for the + * index and reserved fields of the + * mailbox. + */ + if (i == PAGE_SIZE / sizeof (u64) - 2) { + err = mthca_write_mtt(dev, mr->mtt, + n, pages, i); + if (err) + goto mtt_done; + n += i; + i = 0; + } + } + } + + if (i) + err = mthca_write_mtt(dev, mr->mtt, n, pages, i); +mtt_done: + free_page((unsigned long) pages); + if (err) + goto err_mtt; + + err = mthca_mr_alloc(dev, to_mpd(pd)->pd_num, shift, region->virt_base, + region->length, convert_access(acc), mr); + + if (err) + goto err_mtt; + + return &mr->ibmr; + +err_mtt: + mthca_free_mtt(dev, mr->mtt); + +err: + kfree(mr); + return ERR_PTR(err); +} + static int mthca_dereg_mr(struct ib_mr *mr) { struct mthca_mr *mmr = to_mmr(mr); @@ -804,6 +885,7 @@ int mthca_register_device(struct mthca_d dev->ib_dev.poll_cq = mthca_poll_cq; dev->ib_dev.get_dma_mr = mthca_get_dma_mr; dev->ib_dev.reg_phys_mr = mthca_reg_phys_mr; + dev->ib_dev.reg_user_mr = mthca_reg_user_mr; dev->ib_dev.dereg_mr = mthca_dereg_mr; if (dev->mthca_flags & MTHCA_FLAG_FMR) { From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 02/16] IB uverbs: update kernel midlayer for new API In-Reply-To: <2005628163.dSvxYW7XXf2cvPdt@cisco.com> Message-ID: <2005628163.TnJHv5fpNVcYsu6I@cisco.com> Update kernel InfiniBand midlayer to compile against the updated API for low-level drivers. This just amounts to passing NULL for all userspace-related parameters, and setting userspace-related structure members to NULL. Signed-off-by: Roland Dreier --- drivers/infiniband/core/verbs.c | 32 ++++++++++++++++++++------------ 1 files changed, 20 insertions(+), 12 deletions(-) --- linux.orig/drivers/infiniband/core/verbs.c 2005-06-28 15:19:55.267928471 -0700 +++ linux/drivers/infiniband/core/verbs.c 2005-06-28 15:19:59.462022670 -0700 @@ -4,6 +4,7 @@ * Copyright (c) 2004 Intel Corporation. All rights reserved. * Copyright (c) 2004 Topspin Corporation. All rights reserved. * Copyright (c) 2004 Voltaire Corporation. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -47,10 +48,11 @@ struct ib_pd *ib_alloc_pd(struct ib_devi { struct ib_pd *pd; - pd = device->alloc_pd(device); + pd = device->alloc_pd(device, NULL, NULL); if (!IS_ERR(pd)) { - pd->device = device; + pd->device = device; + pd->uobject = NULL; atomic_set(&pd->usecnt, 0); } @@ -76,8 +78,9 @@ struct ib_ah *ib_create_ah(struct ib_pd ah = pd->device->create_ah(pd, ah_attr); if (!IS_ERR(ah)) { - ah->device = pd->device; - ah->pd = pd; + ah->device = pd->device; + ah->pd = pd; + ah->uobject = NULL; atomic_inc(&pd->usecnt); } @@ -122,7 +125,7 @@ struct ib_qp *ib_create_qp(struct ib_pd { struct ib_qp *qp; - qp = pd->device->create_qp(pd, qp_init_attr); + qp = pd->device->create_qp(pd, qp_init_attr, NULL); if (!IS_ERR(qp)) { qp->device = pd->device; @@ -130,6 +133,7 @@ struct ib_qp *ib_create_qp(struct ib_pd qp->send_cq = qp_init_attr->send_cq; qp->recv_cq = qp_init_attr->recv_cq; qp->srq = qp_init_attr->srq; + qp->uobject = NULL; qp->event_handler = qp_init_attr->event_handler; qp->qp_context = qp_init_attr->qp_context; qp->qp_type = qp_init_attr->qp_type; @@ -197,10 +201,11 @@ struct ib_cq *ib_create_cq(struct ib_dev { struct ib_cq *cq; - cq = device->create_cq(device, cqe); + cq = device->create_cq(device, cqe, NULL, NULL); if (!IS_ERR(cq)) { cq->device = device; + cq->uobject = NULL; cq->comp_handler = comp_handler; cq->event_handler = event_handler; cq->cq_context = cq_context; @@ -245,8 +250,9 @@ struct ib_mr *ib_get_dma_mr(struct ib_pd mr = pd->device->get_dma_mr(pd, mr_access_flags); if (!IS_ERR(mr)) { - mr->device = pd->device; - mr->pd = pd; + mr->device = pd->device; + mr->pd = pd; + mr->uobject = NULL; atomic_inc(&pd->usecnt); atomic_set(&mr->usecnt, 0); } @@ -267,8 +273,9 @@ struct ib_mr *ib_reg_phys_mr(struct ib_p mr_access_flags, iova_start); if (!IS_ERR(mr)) { - mr->device = pd->device; - mr->pd = pd; + mr->device = pd->device; + mr->pd = pd; + mr->uobject = NULL; atomic_inc(&pd->usecnt); atomic_set(&mr->usecnt, 0); } @@ -344,8 +351,9 @@ struct ib_mw *ib_alloc_mw(struct ib_pd * mw = pd->device->alloc_mw(pd); if (!IS_ERR(mw)) { - mw->device = pd->device; - mw->pd = pd; + mw->device = pd->device; + mw->pd = pd; + mw->uobject = NULL; atomic_inc(&pd->usecnt); } From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 09/16] IB uverbs: add mthca user doorbell record support In-Reply-To: <2005628163.kTUk6BiHHfEXJhoz@cisco.com> Message-ID: <2005628163.UOpVQTxEtDs4UiFi@cisco.com> Add support for userspace doorbell records to mthca. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_memfree.c | 141 +++++++++++++++++++++++++++- drivers/infiniband/hw/mthca/mthca_memfree.h | 14 ++ 2 files changed, 149 insertions(+), 6 deletions(-) --- linux.orig/drivers/infiniband/hw/mthca/mthca_memfree.c 2005-06-28 15:19:22.793941807 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_memfree.c 2005-06-28 15:20:12.995098113 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -47,6 +48,15 @@ enum { MTHCA_TABLE_CHUNK_SIZE = 1 << 18 }; +struct mthca_user_db_table { + struct semaphore mutex; + struct { + u64 uvirt; + struct scatterlist mem; + int refcount; + } page[0]; +}; + void mthca_free_icm(struct mthca_dev *dev, struct mthca_icm *icm) { struct mthca_icm_chunk *chunk, *tmp; @@ -344,13 +354,133 @@ void mthca_free_icm_table(struct mthca_d kfree(table); } -static u64 mthca_uarc_virt(struct mthca_dev *dev, int page) +static u64 mthca_uarc_virt(struct mthca_dev *dev, struct mthca_uar *uar, int page) { return dev->uar_table.uarc_base + - dev->driver_uar.index * dev->uar_table.uarc_size + + uar->index * dev->uar_table.uarc_size + page * 4096; } +int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, + struct mthca_user_db_table *db_tab, int index, u64 uaddr) +{ + int ret = 0; + u8 status; + int i; + + if (!mthca_is_memfree(dev)) + return 0; + + if (index < 0 || index > dev->uar_table.uarc_size / 8) + return -EINVAL; + + down(&db_tab->mutex); + + i = index / MTHCA_DB_REC_PER_PAGE; + + if ((db_tab->page[i].refcount >= MTHCA_DB_REC_PER_PAGE) || + (db_tab->page[i].uvirt && db_tab->page[i].uvirt != uaddr) || + (uaddr & 4095)) { + ret = -EINVAL; + goto out; + } + + if (db_tab->page[i].refcount) { + ++db_tab->page[i].refcount; + goto out; + } + + ret = get_user_pages(current, current->mm, uaddr & PAGE_MASK, 1, 1, 0, + &db_tab->page[i].mem.page, NULL); + if (ret < 0) + goto out; + + db_tab->page[i].mem.length = 4096; + db_tab->page[i].mem.offset = uaddr & ~PAGE_MASK; + + ret = pci_map_sg(dev->pdev, &db_tab->page[i].mem, 1, PCI_DMA_TODEVICE); + if (ret < 0) { + put_page(db_tab->page[i].mem.page); + goto out; + } + + ret = mthca_MAP_ICM_page(dev, sg_dma_address(&db_tab->page[i].mem), + mthca_uarc_virt(dev, uar, i), &status); + if (!ret && status) + ret = -EINVAL; + if (ret) { + pci_unmap_sg(dev->pdev, &db_tab->page[i].mem, 1, PCI_DMA_TODEVICE); + put_page(db_tab->page[i].mem.page); + goto out; + } + + db_tab->page[i].uvirt = uaddr; + db_tab->page[i].refcount = 1; + +out: + up(&db_tab->mutex); + return ret; +} + +void mthca_unmap_user_db(struct mthca_dev *dev, struct mthca_uar *uar, + struct mthca_user_db_table *db_tab, int index) +{ + if (!mthca_is_memfree(dev)) + return; + + /* + * To make our bookkeeping simpler, we don't unmap DB + * pages until we clean up the whole db table. + */ + + down(&db_tab->mutex); + + --db_tab->page[index / MTHCA_DB_REC_PER_PAGE].refcount; + + up(&db_tab->mutex); +} + +struct mthca_user_db_table *mthca_init_user_db_tab(struct mthca_dev *dev) +{ + struct mthca_user_db_table *db_tab; + int npages; + int i; + + if (!mthca_is_memfree(dev)) + return NULL; + + npages = dev->uar_table.uarc_size / 4096; + db_tab = kmalloc(sizeof *db_tab + npages * sizeof *db_tab->page, GFP_KERNEL); + if (!db_tab) + return ERR_PTR(-ENOMEM); + + init_MUTEX(&db_tab->mutex); + for (i = 0; i < npages; ++i) { + db_tab->page[i].refcount = 0; + db_tab->page[i].uvirt = 0; + } + + return db_tab; +} + +void mthca_cleanup_user_db_tab(struct mthca_dev *dev, struct mthca_uar *uar, + struct mthca_user_db_table *db_tab) +{ + int i; + u8 status; + + if (!mthca_is_memfree(dev)) + return; + + for (i = 0; i < dev->uar_table.uarc_size / 4096; ++i) { + if (db_tab->page[i].uvirt) { + mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, uar, i), 1, &status); + pci_unmap_sg(dev->pdev, &db_tab->page[i].mem, 1, PCI_DMA_TODEVICE); + put_page(db_tab->page[i].mem.page); + } + } +} + int mthca_alloc_db(struct mthca_dev *dev, int type, u32 qn, u32 **db) { int group; @@ -407,7 +537,8 @@ int mthca_alloc_db(struct mthca_dev *dev } memset(page->db_rec, 0, 4096); - ret = mthca_MAP_ICM_page(dev, page->mapping, mthca_uarc_virt(dev, i), &status); + ret = mthca_MAP_ICM_page(dev, page->mapping, + mthca_uarc_virt(dev, &dev->driver_uar, i), &status); if (!ret && status) ret = -EINVAL; if (ret) { @@ -461,7 +592,7 @@ void mthca_free_db(struct mthca_dev *dev if (bitmap_empty(page->used, MTHCA_DB_REC_PER_PAGE) && i >= dev->db_tab->max_group1 - 1) { - mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, i), 1, &status); + mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, &dev->driver_uar, i), 1, &status); dma_free_coherent(&dev->pdev->dev, 4096, page->db_rec, page->mapping); @@ -530,7 +661,7 @@ void mthca_cleanup_db_tab(struct mthca_d if (!bitmap_empty(dev->db_tab->page[i].used, MTHCA_DB_REC_PER_PAGE)) mthca_warn(dev, "Kernel UARC page %d not empty\n", i); - mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, i), 1, &status); + mthca_UNMAP_ICM(dev, mthca_uarc_virt(dev, &dev->driver_uar, i), 1, &status); dma_free_coherent(&dev->pdev->dev, 4096, dev->db_tab->page[i].db_rec, --- linux.orig/drivers/infiniband/hw/mthca/mthca_memfree.h 2005-06-28 15:19:22.793941807 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_memfree.h 2005-06-28 15:20:12.995098113 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -148,7 +149,7 @@ struct mthca_db_table { struct semaphore mutex; }; -enum { +enum mthca_db_type { MTHCA_DB_TYPE_INVALID = 0x0, MTHCA_DB_TYPE_CQ_SET_CI = 0x1, MTHCA_DB_TYPE_CQ_ARM = 0x2, @@ -158,6 +159,17 @@ enum { MTHCA_DB_TYPE_GROUP_SEP = 0x7 }; +struct mthca_user_db_table; +struct mthca_uar; + +int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, + struct mthca_user_db_table *db_tab, int index, u64 uaddr); +void mthca_unmap_user_db(struct mthca_dev *dev, struct mthca_uar *uar, + struct mthca_user_db_table *db_tab, int index); +struct mthca_user_db_table *mthca_init_user_db_tab(struct mthca_dev *dev); +void mthca_cleanup_user_db_tab(struct mthca_dev *dev, struct mthca_uar *uar, + struct mthca_user_db_table *db_tab); + int mthca_init_db_tab(struct mthca_dev *dev); void mthca_cleanup_db_tab(struct mthca_dev *dev); int mthca_alloc_db(struct mthca_dev *dev, int type, u32 qn, u32 **db); From rolandd at cisco.com Tue Jun 28 16:03:44 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:44 -0700 Subject: [openib-general] [PATCH 14/16] IB uverbs: add mthca user CQ support In-Reply-To: <2005628163.XfYnZThlsTGb61Td@cisco.com> Message-ID: <2005628163.jWivrrVepuPgy40z@cisco.com> Add support for userspace completion queues (CQs) to mthca. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_cq.c | 78 +++++++++++++++------------ drivers/infiniband/hw/mthca/mthca_dev.h | 1 drivers/infiniband/hw/mthca/mthca_provider.c | 69 +++++++++++++++++++++-- drivers/infiniband/hw/mthca/mthca_provider.h | 1 4 files changed, 111 insertions(+), 38 deletions(-) --- linux.orig/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-28 15:19:19.183721488 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_cq.c 2005-06-28 15:20:25.680347052 -0700 @@ -1,6 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Cisco Systems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -742,6 +743,7 @@ err_out: } int mthca_init_cq(struct mthca_dev *dev, int nent, + struct mthca_ucontext *ctx, u32 pdn, struct mthca_cq *cq) { int size = nent * MTHCA_CQ_ENTRY_SIZE; @@ -753,30 +755,33 @@ int mthca_init_cq(struct mthca_dev *dev, might_sleep(); - cq->ibcq.cqe = nent - 1; + cq->ibcq.cqe = nent - 1; + cq->is_kernel = !ctx; cq->cqn = mthca_alloc(&dev->cq_table.alloc); if (cq->cqn == -1) return -ENOMEM; if (mthca_is_memfree(dev)) { - cq->arm_sn = 1; - err = mthca_table_get(dev, dev->cq_table.table, cq->cqn); if (err) goto err_out; - err = -ENOMEM; + if (cq->is_kernel) { + cq->arm_sn = 1; + + err = -ENOMEM; - cq->set_ci_db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_CQ_SET_CI, - cq->cqn, &cq->set_ci_db); - if (cq->set_ci_db_index < 0) - goto err_out_icm; - - cq->arm_db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_CQ_ARM, - cq->cqn, &cq->arm_db); - if (cq->arm_db_index < 0) - goto err_out_ci; + cq->set_ci_db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_CQ_SET_CI, + cq->cqn, &cq->set_ci_db); + if (cq->set_ci_db_index < 0) + goto err_out_icm; + + cq->arm_db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_CQ_ARM, + cq->cqn, &cq->arm_db); + if (cq->arm_db_index < 0) + goto err_out_ci; + } } mailbox = mthca_alloc_mailbox(dev, GFP_KERNEL); @@ -785,12 +790,14 @@ int mthca_init_cq(struct mthca_dev *dev, cq_context = mailbox->buf; - err = mthca_alloc_cq_buf(dev, size, cq); - if (err) - goto err_out_mailbox; + if (cq->is_kernel) { + err = mthca_alloc_cq_buf(dev, size, cq); + if (err) + goto err_out_mailbox; - for (i = 0; i < nent; ++i) - set_cqe_hw(get_cqe(cq, i)); + for (i = 0; i < nent; ++i) + set_cqe_hw(get_cqe(cq, i)); + } spin_lock_init(&cq->lock); atomic_set(&cq->refcount, 1); @@ -801,11 +808,14 @@ int mthca_init_cq(struct mthca_dev *dev, MTHCA_CQ_STATE_DISARMED | MTHCA_CQ_FLAG_TR); cq_context->start = cpu_to_be64(0); - cq_context->logsize_usrpage = cpu_to_be32((ffs(nent) - 1) << 24 | - dev->driver_uar.index); + cq_context->logsize_usrpage = cpu_to_be32((ffs(nent) - 1) << 24); + if (ctx) + cq_context->logsize_usrpage |= cpu_to_be32(ctx->uar.index); + else + cq_context->logsize_usrpage |= cpu_to_be32(dev->driver_uar.index); cq_context->error_eqn = cpu_to_be32(dev->eq_table.eq[MTHCA_EQ_ASYNC].eqn); cq_context->comp_eqn = cpu_to_be32(dev->eq_table.eq[MTHCA_EQ_COMP].eqn); - cq_context->pd = cpu_to_be32(dev->driver_pd.pd_num); + cq_context->pd = cpu_to_be32(pdn); cq_context->lkey = cpu_to_be32(cq->mr.ibmr.lkey); cq_context->cqn = cpu_to_be32(cq->cqn); @@ -843,18 +853,20 @@ int mthca_init_cq(struct mthca_dev *dev, return 0; err_out_free_mr: - mthca_free_mr(dev, &cq->mr); - mthca_free_cq_buf(dev, cq); + if (cq->is_kernel) { + mthca_free_mr(dev, &cq->mr); + mthca_free_cq_buf(dev, cq); + } err_out_mailbox: mthca_free_mailbox(dev, mailbox); err_out_arm: - if (mthca_is_memfree(dev)) + if (cq->is_kernel && mthca_is_memfree(dev)) mthca_free_db(dev, MTHCA_DB_TYPE_CQ_ARM, cq->arm_db_index); err_out_ci: - if (mthca_is_memfree(dev)) + if (cq->is_kernel && mthca_is_memfree(dev)) mthca_free_db(dev, MTHCA_DB_TYPE_CQ_SET_CI, cq->set_ci_db_index); err_out_icm: @@ -892,7 +904,8 @@ void mthca_free_cq(struct mthca_dev *dev int j; printk(KERN_ERR "context for CQN %x (cons index %x, next sw %d)\n", - cq->cqn, cq->cons_index, !!next_cqe_sw(cq)); + cq->cqn, cq->cons_index, + cq->is_kernel ? !!next_cqe_sw(cq) : 0); for (j = 0; j < 16; ++j) printk(KERN_ERR "[%2x] %08x\n", j * 4, be32_to_cpu(ctx[j])); } @@ -910,12 +923,13 @@ void mthca_free_cq(struct mthca_dev *dev atomic_dec(&cq->refcount); wait_event(cq->wait, !atomic_read(&cq->refcount)); - mthca_free_mr(dev, &cq->mr); - mthca_free_cq_buf(dev, cq); - - if (mthca_is_memfree(dev)) { - mthca_free_db(dev, MTHCA_DB_TYPE_CQ_ARM, cq->arm_db_index); - mthca_free_db(dev, MTHCA_DB_TYPE_CQ_SET_CI, cq->set_ci_db_index); + if (cq->is_kernel) { + mthca_free_mr(dev, &cq->mr); + mthca_free_cq_buf(dev, cq); + if (mthca_is_memfree(dev)) { + mthca_free_db(dev, MTHCA_DB_TYPE_CQ_ARM, cq->arm_db_index); + mthca_free_db(dev, MTHCA_DB_TYPE_CQ_SET_CI, cq->set_ci_db_index); + } } mthca_table_put(dev, dev->cq_table.table, cq->cqn); --- linux.orig/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-28 15:20:20.514467380 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-28 15:20:25.681346835 -0700 @@ -414,6 +414,7 @@ int mthca_poll_cq(struct ib_cq *ibcq, in int mthca_tavor_arm_cq(struct ib_cq *cq, enum ib_cq_notify notify); int mthca_arbel_arm_cq(struct ib_cq *cq, enum ib_cq_notify notify); int mthca_init_cq(struct mthca_dev *dev, int nent, + struct mthca_ucontext *ctx, u32 pdn, struct mthca_cq *cq); void mthca_free_cq(struct mthca_dev *dev, struct mthca_cq *cq); --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:23.354851384 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:25.681346835 -0700 @@ -497,28 +497,85 @@ static struct ib_cq *mthca_create_cq(str struct ib_ucontext *context, struct ib_udata *udata) { + struct mthca_create_cq ucmd; struct mthca_cq *cq; int nent; int err; + if (context) { + if (ib_copy_from_udata(&ucmd, udata, sizeof ucmd)) + return ERR_PTR(-EFAULT); + + err = mthca_map_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, + to_mucontext(context)->db_tab, + ucmd.set_db_index, ucmd.set_db_page); + if (err) + return ERR_PTR(err); + + err = mthca_map_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, + to_mucontext(context)->db_tab, + ucmd.arm_db_index, ucmd.arm_db_page); + if (err) + goto err_unmap_set; + } + cq = kmalloc(sizeof *cq, GFP_KERNEL); - if (!cq) - return ERR_PTR(-ENOMEM); + if (!cq) { + err = -ENOMEM; + goto err_unmap_arm; + } + + if (context) { + cq->mr.ibmr.lkey = ucmd.lkey; + cq->set_ci_db_index = ucmd.set_db_index; + cq->arm_db_index = ucmd.arm_db_index; + } for (nent = 1; nent <= entries; nent <<= 1) ; /* nothing */ - err = mthca_init_cq(to_mdev(ibdev), nent, cq); - if (err) { - kfree(cq); - cq = ERR_PTR(err); + err = mthca_init_cq(to_mdev(ibdev), nent, + context ? to_mucontext(context) : NULL, + context ? ucmd.pdn : to_mdev(ibdev)->driver_pd.pd_num, + cq); + if (err) + goto err_free; + + if (context && ib_copy_to_udata(udata, &cq->cqn, sizeof (__u32))) { + mthca_free_cq(to_mdev(ibdev), cq); + goto err_free; } return &cq->ibcq; + +err_free: + kfree(cq); + +err_unmap_arm: + if (context) + mthca_unmap_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, + to_mucontext(context)->db_tab, ucmd.arm_db_index); + +err_unmap_set: + if (context) + mthca_unmap_user_db(to_mdev(ibdev), &to_mucontext(context)->uar, + to_mucontext(context)->db_tab, ucmd.set_db_index); + + return ERR_PTR(err); } static int mthca_destroy_cq(struct ib_cq *cq) { + if (cq->uobject) { + mthca_unmap_user_db(to_mdev(cq->device), + &to_mucontext(cq->uobject->context)->uar, + to_mucontext(cq->uobject->context)->db_tab, + to_mcq(cq)->arm_db_index); + mthca_unmap_user_db(to_mdev(cq->device), + &to_mucontext(cq->uobject->context)->uar, + to_mucontext(cq->uobject->context)->db_tab, + to_mcq(cq)->set_ci_db_index); + } mthca_free_cq(to_mdev(cq->device), to_mcq(cq)); kfree(cq); --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.h 2005-06-28 15:20:20.514467380 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.h 2005-06-28 15:20:25.682346619 -0700 @@ -177,6 +177,7 @@ struct mthca_cq { int cqn; u32 cons_index; int is_direct; + int is_kernel; /* Next fields are Arbel only */ int set_ci_db_index; From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 10/16] IB uverbs: add mthca user context support In-Reply-To: <2005628163.UOpVQTxEtDs4UiFi@cisco.com> Message-ID: <2005628163.o84QGfsM7oMSy0oU@cisco.com> Add support for managing userspace contexts to mthca. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_provider.c | 58 +++++++++++++++++++++++++++ drivers/infiniband/hw/mthca/mthca_provider.h | 14 ++++++ 2 files changed, 72 insertions(+) --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:00.882715841 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:16.611313860 -0700 @@ -1,6 +1,7 @@ /* * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Sun Microsystems, Inc. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -37,6 +38,8 @@ #include "mthca_dev.h" #include "mthca_cmd.h" +#include "mthca_user.h" +#include "mthca_memfree.h" static int mthca_query_device(struct ib_device *ibdev, struct ib_device_attr *props) @@ -284,6 +287,59 @@ static int mthca_query_gid(struct ib_dev return err; } +static struct ib_ucontext *mthca_alloc_ucontext(struct ib_device *ibdev, + struct ib_udata *udata) +{ + struct mthca_alloc_ucontext_resp uresp; + struct mthca_ucontext *context; + int err; + + memset(&uresp, 0, sizeof uresp); + + uresp.qp_tab_size = to_mdev(ibdev)->limits.num_qps; + if (mthca_is_memfree(to_mdev(ibdev))) + uresp.uarc_size = to_mdev(ibdev)->uar_table.uarc_size; + else + uresp.uarc_size = 0; + + context = kmalloc(sizeof *context, GFP_KERNEL); + if (!context) + return ERR_PTR(-ENOMEM); + + err = mthca_uar_alloc(to_mdev(ibdev), &context->uar); + if (err) { + kfree(context); + return ERR_PTR(err); + } + + context->db_tab = mthca_init_user_db_tab(to_mdev(ibdev)); + if (IS_ERR(context->db_tab)) { + err = PTR_ERR(context->db_tab); + mthca_uar_free(to_mdev(ibdev), &context->uar); + kfree(context); + return ERR_PTR(err); + } + + if (ib_copy_to_udata(udata, &uresp, sizeof uresp)) { + mthca_cleanup_user_db_tab(to_mdev(ibdev), &context->uar, context->db_tab); + mthca_uar_free(to_mdev(ibdev), &context->uar); + kfree(context); + return ERR_PTR(-EFAULT); + } + + return &context->ibucontext; +} + +static int mthca_dealloc_ucontext(struct ib_ucontext *context) +{ + mthca_cleanup_user_db_tab(to_mdev(context->device), &to_mucontext(context)->uar, + to_mucontext(context)->db_tab); + mthca_uar_free(to_mdev(context->device), &to_mucontext(context)->uar); + kfree(to_mucontext(context)); + + return 0; +} + static struct ib_pd *mthca_alloc_pd(struct ib_device *ibdev, struct ib_ucontext *context, struct ib_udata *udata) @@ -708,6 +764,8 @@ int mthca_register_device(struct mthca_d dev->ib_dev.modify_port = mthca_modify_port; dev->ib_dev.query_pkey = mthca_query_pkey; dev->ib_dev.query_gid = mthca_query_gid; + dev->ib_dev.alloc_ucontext = mthca_alloc_ucontext; + dev->ib_dev.dealloc_ucontext = mthca_dealloc_ucontext; dev->ib_dev.alloc_pd = mthca_alloc_pd; dev->ib_dev.dealloc_pd = mthca_dealloc_pd; dev->ib_dev.create_ah = mthca_ah_create; --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.h 2005-06-28 15:19:22.365034436 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.h 2005-06-28 15:20:16.612313643 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -54,6 +55,14 @@ struct mthca_uar { int index; }; +struct mthca_user_db_table; + +struct mthca_ucontext { + struct ib_ucontext ibucontext; + struct mthca_uar uar; + struct mthca_user_db_table *db_tab; +}; + struct mthca_mtt; struct mthca_mr { @@ -236,6 +245,11 @@ struct mthca_sqp { dma_addr_t header_dma; }; +static inline struct mthca_ucontext *to_mucontext(struct ib_ucontext *ibucontext) +{ + return container_of(ibucontext, struct mthca_ucontext, ibucontext); +} + static inline struct mthca_fmr *to_mfmr(struct ib_fmr *ibmr) { return container_of(ibmr, struct mthca_fmr, ibmr); From rolandd at cisco.com Tue Jun 28 16:03:44 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:44 -0700 Subject: [openib-general] [PATCH 15/16] IB uverbs: add mthca user QP support In-Reply-To: <2005628163.jWivrrVepuPgy40z@cisco.com> Message-ID: <2005628163.VUsohVlfE72duiiM@cisco.com> Add support for userspace queue pairs (QPs) to mthca. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_dev.h | 2 drivers/infiniband/hw/mthca/mthca_provider.c | 80 ++++++++-- drivers/infiniband/hw/mthca/mthca_qp.c | 215 +++++++++++++++++---------- 3 files changed, 212 insertions(+), 85 deletions(-) --- linux.orig/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-28 15:20:25.681346835 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_dev.h 2005-06-28 15:20:28.954636956 -0700 @@ -440,12 +440,14 @@ int mthca_alloc_qp(struct mthca_dev *dev struct mthca_cq *recv_cq, enum ib_qp_type type, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, struct mthca_qp *qp); int mthca_alloc_sqp(struct mthca_dev *dev, struct mthca_pd *pd, struct mthca_cq *send_cq, struct mthca_cq *recv_cq, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, int qpn, int port, struct mthca_sqp *sqp); --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:25.681346835 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:28.953637173 -0700 @@ -424,6 +424,7 @@ static struct ib_qp *mthca_create_qp(str struct ib_qp_init_attr *init_attr, struct ib_udata *udata) { + struct mthca_create_qp ucmd; struct mthca_qp *qp; int err; @@ -432,41 +433,82 @@ static struct ib_qp *mthca_create_qp(str case IB_QPT_UC: case IB_QPT_UD: { + struct mthca_ucontext *context; + qp = kmalloc(sizeof *qp, GFP_KERNEL); if (!qp) return ERR_PTR(-ENOMEM); - qp->sq.max = init_attr->cap.max_send_wr; - qp->rq.max = init_attr->cap.max_recv_wr; - qp->sq.max_gs = init_attr->cap.max_send_sge; - qp->rq.max_gs = init_attr->cap.max_recv_sge; + if (pd->uobject) { + context = to_mucontext(pd->uobject->context); + + if (ib_copy_from_udata(&ucmd, udata, sizeof ucmd)) + return ERR_PTR(-EFAULT); + + err = mthca_map_user_db(to_mdev(pd->device), &context->uar, + context->db_tab, + ucmd.sq_db_index, ucmd.sq_db_page); + if (err) { + kfree(qp); + return ERR_PTR(err); + } + + err = mthca_map_user_db(to_mdev(pd->device), &context->uar, + context->db_tab, + ucmd.rq_db_index, ucmd.rq_db_page); + if (err) { + mthca_unmap_user_db(to_mdev(pd->device), + &context->uar, + context->db_tab, + ucmd.sq_db_index); + kfree(qp); + return ERR_PTR(err); + } + + qp->mr.ibmr.lkey = ucmd.lkey; + qp->sq.db_index = ucmd.sq_db_index; + qp->rq.db_index = ucmd.rq_db_index; + } err = mthca_alloc_qp(to_mdev(pd->device), to_mpd(pd), to_mcq(init_attr->send_cq), to_mcq(init_attr->recv_cq), init_attr->qp_type, init_attr->sq_sig_type, - qp); + &init_attr->cap, qp); + + if (err && pd->uobject) { + context = to_mucontext(pd->uobject->context); + + mthca_unmap_user_db(to_mdev(pd->device), + &context->uar, + context->db_tab, + ucmd.sq_db_index); + mthca_unmap_user_db(to_mdev(pd->device), + &context->uar, + context->db_tab, + ucmd.rq_db_index); + } + qp->ibqp.qp_num = qp->qpn; break; } case IB_QPT_SMI: case IB_QPT_GSI: { + /* Don't allow userspace to create special QPs */ + if (pd->uobject) + return ERR_PTR(-EINVAL); + qp = kmalloc(sizeof (struct mthca_sqp), GFP_KERNEL); if (!qp) return ERR_PTR(-ENOMEM); - qp->sq.max = init_attr->cap.max_send_wr; - qp->rq.max = init_attr->cap.max_recv_wr; - qp->sq.max_gs = init_attr->cap.max_send_sge; - qp->rq.max_gs = init_attr->cap.max_recv_sge; - qp->ibqp.qp_num = init_attr->qp_type == IB_QPT_SMI ? 0 : 1; err = mthca_alloc_sqp(to_mdev(pd->device), to_mpd(pd), to_mcq(init_attr->send_cq), to_mcq(init_attr->recv_cq), - init_attr->sq_sig_type, + init_attr->sq_sig_type, &init_attr->cap, qp->ibqp.qp_num, init_attr->port_num, to_msqp(qp)); break; @@ -481,13 +523,27 @@ static struct ib_qp *mthca_create_qp(str return ERR_PTR(err); } - init_attr->cap.max_inline_data = 0; + init_attr->cap.max_inline_data = 0; + init_attr->cap.max_send_wr = qp->sq.max; + init_attr->cap.max_recv_wr = qp->rq.max; + init_attr->cap.max_send_sge = qp->sq.max_gs; + init_attr->cap.max_recv_sge = qp->rq.max_gs; return &qp->ibqp; } static int mthca_destroy_qp(struct ib_qp *qp) { + if (qp->uobject) { + mthca_unmap_user_db(to_mdev(qp->device), + &to_mucontext(qp->uobject->context)->uar, + to_mucontext(qp->uobject->context)->db_tab, + to_mqp(qp)->sq.db_index); + mthca_unmap_user_db(to_mdev(qp->device), + &to_mucontext(qp->uobject->context)->uar, + to_mucontext(qp->uobject->context)->db_tab, + to_mqp(qp)->rq.db_index); + } mthca_free_qp(to_mdev(qp->device), to_mqp(qp)); kfree(qp); return 0; --- linux.orig/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-28 15:19:18.256921644 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_qp.c 2005-06-28 15:20:28.952637389 -0700 @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -46,7 +47,9 @@ enum { MTHCA_MAX_DIRECT_QP_SIZE = 4 * PAGE_SIZE, MTHCA_ACK_REQ_FREQ = 10, MTHCA_FLIGHT_LIMIT = 9, - MTHCA_UD_HEADER_SIZE = 72 /* largest UD header possible */ + MTHCA_UD_HEADER_SIZE = 72, /* largest UD header possible */ + MTHCA_INLINE_HEADER_SIZE = 4, /* data segment overhead for inline */ + MTHCA_INLINE_CHUNK_SIZE = 16 /* inline data segment chunk */ }; enum { @@ -689,7 +692,11 @@ int mthca_modify_qp(struct ib_qp *ibqp, /* leave arbel_sched_queue as 0 */ - qp_context->usr_page = cpu_to_be32(dev->driver_uar.index); + if (qp->ibqp.uobject) + qp_context->usr_page = + cpu_to_be32(to_mucontext(qp->ibqp.uobject->context)->uar.index); + else + qp_context->usr_page = cpu_to_be32(dev->driver_uar.index); qp_context->local_qpn = cpu_to_be32(qp->qpn); if (attr_mask & IB_QP_DEST_QPN) { qp_context->remote_qpn = cpu_to_be32(attr->dest_qp_num); @@ -954,6 +961,15 @@ static int mthca_alloc_wqe_buf(struct mt qp->send_wqe_offset = ALIGN(qp->rq.max << qp->rq.wqe_shift, 1 << qp->sq.wqe_shift); + + /* + * If this is a userspace QP, we don't actually have to + * allocate anything. All we need is to calculate the WQE + * sizes and the send_wqe_offset, so we're done now. + */ + if (pd->ibpd.uobject) + return 0; + size = PAGE_ALIGN(qp->send_wqe_offset + (qp->sq.max << qp->sq.wqe_shift)); @@ -1053,10 +1069,32 @@ static int mthca_alloc_wqe_buf(struct mt return err; } -static int mthca_alloc_memfree(struct mthca_dev *dev, +static void mthca_free_wqe_buf(struct mthca_dev *dev, struct mthca_qp *qp) { - int ret = 0; + int i; + int size = PAGE_ALIGN(qp->send_wqe_offset + + (qp->sq.max << qp->sq.wqe_shift)); + + if (qp->is_direct) { + dma_free_coherent(&dev->pdev->dev, size, qp->queue.direct.buf, + pci_unmap_addr(&qp->queue.direct, mapping)); + } else { + for (i = 0; i < size / PAGE_SIZE; ++i) { + dma_free_coherent(&dev->pdev->dev, PAGE_SIZE, + qp->queue.page_list[i].buf, + pci_unmap_addr(&qp->queue.page_list[i], + mapping)); + } + } + + kfree(qp->wrid); +} + +static int mthca_map_memfree(struct mthca_dev *dev, + struct mthca_qp *qp) +{ + int ret; if (mthca_is_memfree(dev)) { ret = mthca_table_get(dev, dev->qp_table.qp_table, qp->qpn); @@ -1067,35 +1105,15 @@ static int mthca_alloc_memfree(struct mt if (ret) goto err_qpc; - ret = mthca_table_get(dev, dev->qp_table.rdb_table, - qp->qpn << dev->qp_table.rdb_shift); - if (ret) - goto err_eqpc; - - qp->rq.db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_RQ, - qp->qpn, &qp->rq.db); - if (qp->rq.db_index < 0) { - ret = -ENOMEM; - goto err_rdb; - } + ret = mthca_table_get(dev, dev->qp_table.rdb_table, + qp->qpn << dev->qp_table.rdb_shift); + if (ret) + goto err_eqpc; - qp->sq.db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_SQ, - qp->qpn, &qp->sq.db); - if (qp->sq.db_index < 0) { - ret = -ENOMEM; - goto err_rq_db; - } } return 0; -err_rq_db: - mthca_free_db(dev, MTHCA_DB_TYPE_RQ, qp->rq.db_index); - -err_rdb: - mthca_table_put(dev, dev->qp_table.rdb_table, - qp->qpn << dev->qp_table.rdb_shift); - err_eqpc: mthca_table_put(dev, dev->qp_table.eqp_table, qp->qpn); @@ -1105,6 +1123,35 @@ err_qpc: return ret; } +static void mthca_unmap_memfree(struct mthca_dev *dev, + struct mthca_qp *qp) +{ + mthca_table_put(dev, dev->qp_table.rdb_table, + qp->qpn << dev->qp_table.rdb_shift); + mthca_table_put(dev, dev->qp_table.eqp_table, qp->qpn); + mthca_table_put(dev, dev->qp_table.qp_table, qp->qpn); +} + +static int mthca_alloc_memfree(struct mthca_dev *dev, + struct mthca_qp *qp) +{ + int ret = 0; + + if (mthca_is_memfree(dev)) { + qp->rq.db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_RQ, + qp->qpn, &qp->rq.db); + if (qp->rq.db_index < 0) + return ret; + + qp->sq.db_index = mthca_alloc_db(dev, MTHCA_DB_TYPE_SQ, + qp->qpn, &qp->sq.db); + if (qp->sq.db_index < 0) + mthca_free_db(dev, MTHCA_DB_TYPE_RQ, qp->rq.db_index); + } + + return ret; +} + static void mthca_free_memfree(struct mthca_dev *dev, struct mthca_qp *qp) { @@ -1112,11 +1159,6 @@ static void mthca_free_memfree(struct mt mthca_free_db(dev, MTHCA_DB_TYPE_SQ, qp->sq.db_index); mthca_free_db(dev, MTHCA_DB_TYPE_RQ, qp->rq.db_index); } - - mthca_table_put(dev, dev->qp_table.rdb_table, - qp->qpn << dev->qp_table.rdb_shift); - mthca_table_put(dev, dev->qp_table.eqp_table, qp->qpn); - mthca_table_put(dev, dev->qp_table.qp_table, qp->qpn); } static void mthca_wq_init(struct mthca_wq* wq) @@ -1147,13 +1189,28 @@ static int mthca_alloc_qp_common(struct mthca_wq_init(&qp->sq); mthca_wq_init(&qp->rq); - ret = mthca_alloc_memfree(dev, qp); + ret = mthca_map_memfree(dev, qp); if (ret) return ret; ret = mthca_alloc_wqe_buf(dev, pd, qp); if (ret) { - mthca_free_memfree(dev, qp); + mthca_unmap_memfree(dev, qp); + return ret; + } + + /* + * If this is a userspace QP, we're done now. The doorbells + * will be allocated and buffers will be initialized in + * userspace. + */ + if (pd->ibpd.uobject) + return 0; + + ret = mthca_alloc_memfree(dev, qp); + if (ret) { + mthca_free_wqe_buf(dev, qp); + mthca_unmap_memfree(dev, qp); return ret; } @@ -1186,22 +1243,39 @@ static int mthca_alloc_qp_common(struct return 0; } -static void mthca_align_qp_size(struct mthca_dev *dev, struct mthca_qp *qp) +static int mthca_set_qp_size(struct mthca_dev *dev, struct ib_qp_cap *cap, + struct mthca_qp *qp) { - int i; - - if (!mthca_is_memfree(dev)) - return; + /* Sanity check QP size before proceeding */ + if (cap->max_send_wr > 65536 || cap->max_recv_wr > 65536 || + cap->max_send_sge > 64 || cap->max_recv_sge > 64) + return -EINVAL; - for (i = 0; 1 << i < qp->rq.max; ++i) - ; /* nothing */ + if (mthca_is_memfree(dev)) { + qp->rq.max = cap->max_recv_wr ? + roundup_pow_of_two(cap->max_recv_wr) : 0; + qp->sq.max = cap->max_send_wr ? + roundup_pow_of_two(cap->max_send_wr) : 0; + } else { + qp->rq.max = cap->max_recv_wr; + qp->sq.max = cap->max_send_wr; + } - qp->rq.max = 1 << i; + qp->rq.max_gs = cap->max_recv_sge; + qp->sq.max_gs = max_t(int, cap->max_send_sge, + ALIGN(cap->max_inline_data + MTHCA_INLINE_HEADER_SIZE, + MTHCA_INLINE_CHUNK_SIZE) / + sizeof (struct mthca_data_seg)); - for (i = 0; 1 << i < qp->sq.max; ++i) - ; /* nothing */ + /* + * For MLX transport we need 2 extra S/G entries: + * one for the header and one for the checksum at the end + */ + if ((qp->transport == MLX && qp->sq.max_gs + 2 > dev->limits.max_sg) || + qp->sq.max_gs > dev->limits.max_sg || qp->rq.max_gs > dev->limits.max_sg) + return -EINVAL; - qp->sq.max = 1 << i; + return 0; } int mthca_alloc_qp(struct mthca_dev *dev, @@ -1210,11 +1284,14 @@ int mthca_alloc_qp(struct mthca_dev *dev struct mthca_cq *recv_cq, enum ib_qp_type type, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, struct mthca_qp *qp) { int err; - mthca_align_qp_size(dev, qp); + err = mthca_set_qp_size(dev, cap, qp); + if (err) + return err; switch (type) { case IB_QPT_RC: qp->transport = RC; break; @@ -1247,14 +1324,17 @@ int mthca_alloc_sqp(struct mthca_dev *de struct mthca_cq *send_cq, struct mthca_cq *recv_cq, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, int qpn, int port, struct mthca_sqp *sqp) { - int err = 0; u32 mqpn = qpn * 2 + dev->qp_table.sqp_start + port - 1; + int err; - mthca_align_qp_size(dev, &sqp->qp); + err = mthca_set_qp_size(dev, cap, &sqp->qp); + if (err) + return err; sqp->header_buf_size = sqp->qp.sq.max * MTHCA_UD_HEADER_SIZE; sqp->header_buf = dma_alloc_coherent(&dev->pdev->dev, sqp->header_buf_size, @@ -1313,8 +1393,6 @@ void mthca_free_qp(struct mthca_dev *dev struct mthca_qp *qp) { u8 status; - int size; - int i; struct mthca_cq *send_cq; struct mthca_cq *recv_cq; @@ -1344,31 +1422,22 @@ void mthca_free_qp(struct mthca_dev *dev if (qp->state != IB_QPS_RESET) mthca_MODIFY_QP(dev, MTHCA_TRANS_ANY2RST, qp->qpn, 0, NULL, 0, &status); - mthca_cq_clean(dev, to_mcq(qp->ibqp.send_cq)->cqn, qp->qpn); - if (qp->ibqp.send_cq != qp->ibqp.recv_cq) - mthca_cq_clean(dev, to_mcq(qp->ibqp.recv_cq)->cqn, qp->qpn); - - mthca_free_mr(dev, &qp->mr); - - size = PAGE_ALIGN(qp->send_wqe_offset + - (qp->sq.max << qp->sq.wqe_shift)); + /* + * If this is a userspace QP, the buffers, MR, CQs and so on + * will be cleaned up in userspace, so all we have to do is + * unref the mem-free tables and free the QPN in our table. + */ + if (!qp->ibqp.uobject) { + mthca_cq_clean(dev, to_mcq(qp->ibqp.send_cq)->cqn, qp->qpn); + if (qp->ibqp.send_cq != qp->ibqp.recv_cq) + mthca_cq_clean(dev, to_mcq(qp->ibqp.recv_cq)->cqn, qp->qpn); - if (qp->is_direct) { - pci_free_consistent(dev->pdev, size, - qp->queue.direct.buf, - pci_unmap_addr(&qp->queue.direct, mapping)); - } else { - for (i = 0; i < size / PAGE_SIZE; ++i) { - pci_free_consistent(dev->pdev, PAGE_SIZE, - qp->queue.page_list[i].buf, - pci_unmap_addr(&qp->queue.page_list[i], - mapping)); - } + mthca_free_mr(dev, &qp->mr); + mthca_free_memfree(dev, qp); + mthca_free_wqe_buf(dev, qp); } - kfree(qp->wrid); - - mthca_free_memfree(dev, qp); + mthca_unmap_memfree(dev, qp); if (is_sqp(dev, qp)) { atomic_dec(&(to_mpd(qp->ibqp.pd)->sqp_count)); From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 11/16] IB uverbs: add mthca mmap support In-Reply-To: <2005628163.o84QGfsM7oMSy0oU@cisco.com> Message-ID: <2005628163.gtJFW6uLUrGQteys@cisco.com> Add support for mmap() method to mthca, so that userspace can get access to doorbell registers. This allows userspace to get direct access to the HCA for data path operations. Each userspace context gets its own copy of the doorbell registers and is only allowed to use resources that the kernel has given it access to. In other words, this is safe. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_provider.c | 18 ++++++++++++++++++ 1 files changed, 18 insertions(+) --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:16.611313860 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:18.380930082 -0700 @@ -340,6 +340,23 @@ static int mthca_dealloc_ucontext(struct return 0; } +static int mthca_mmap_uar(struct ib_ucontext *context, + struct vm_area_struct *vma) +{ + if (vma->vm_end - vma->vm_start != PAGE_SIZE) + return -EINVAL; + + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); + vma->vm_flags |= VM_DONTCOPY; + + if (remap_pfn_range(vma, vma->vm_start, + to_mucontext(context)->uar.pfn, + PAGE_SIZE, vma->vm_page_prot)) + return -EAGAIN; + + return 0; +} + static struct ib_pd *mthca_alloc_pd(struct ib_device *ibdev, struct ib_ucontext *context, struct ib_udata *udata) @@ -766,6 +783,7 @@ int mthca_register_device(struct mthca_d dev->ib_dev.query_gid = mthca_query_gid; dev->ib_dev.alloc_ucontext = mthca_alloc_ucontext; dev->ib_dev.dealloc_ucontext = mthca_dealloc_ucontext; + dev->ib_dev.mmap = mthca_mmap_uar; dev->ib_dev.alloc_pd = mthca_alloc_pd; dev->ib_dev.dealloc_pd = mthca_dealloc_pd; dev->ib_dev.create_ah = mthca_ah_create; From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 03/16] IB uverbs: update mthca for new API In-Reply-To: <2005628163.TnJHv5fpNVcYsu6I@cisco.com> Message-ID: <2005628163.01zcuUvu5mtvwzkX@cisco.com> Update mthca to compile against the updated API for low-level drivers. Signed-off-by: Roland Dreier --- drivers/infiniband/hw/mthca/mthca_provider.c | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-) --- linux.orig/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:19:55.005985043 -0700 +++ linux/drivers/infiniband/hw/mthca/mthca_provider.c 2005-06-28 15:20:00.882715841 -0700 @@ -284,7 +284,9 @@ static int mthca_query_gid(struct ib_dev return err; } -static struct ib_pd *mthca_alloc_pd(struct ib_device *ibdev) +static struct ib_pd *mthca_alloc_pd(struct ib_device *ibdev, + struct ib_ucontext *context, + struct ib_udata *udata) { struct mthca_pd *pd; int err; @@ -338,7 +340,8 @@ static int mthca_ah_destroy(struct ib_ah } static struct ib_qp *mthca_create_qp(struct ib_pd *pd, - struct ib_qp_init_attr *init_attr) + struct ib_qp_init_attr *init_attr, + struct ib_udata *udata) { struct mthca_qp *qp; int err; @@ -409,7 +412,9 @@ static int mthca_destroy_qp(struct ib_qp return 0; } -static struct ib_cq *mthca_create_cq(struct ib_device *ibdev, int entries) +static struct ib_cq *mthca_create_cq(struct ib_device *ibdev, int entries, + struct ib_ucontext *context, + struct ib_udata *udata) { struct mthca_cq *cq; int nent; @@ -692,6 +697,8 @@ int mthca_register_device(struct mthca_d int i; strlcpy(dev->ib_dev.name, "mthca%d", IB_DEVICE_NAME_MAX); + dev->ib_dev.owner = THIS_MODULE; + dev->ib_dev.node_type = IB_NODE_CA; dev->ib_dev.phys_port_cnt = dev->limits.num_ports; dev->ib_dev.dma_device = &dev->pdev->dev; From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> Message-ID: <2005628163.jfSiMqRcI78iLMJP@cisco.com> Add the core of the InfiniBand userspace verbs implementation, including creating character device nodes, dispatching requests from userspace, and passing event notifications back up to userspace. Signed-off-by: Roland Dreier --- drivers/infiniband/core/uverbs.h | 132 ++++ drivers/infiniband/core/uverbs_cmd.c | 1006 ++++++++++++++++++++++++++++++++++ drivers/infiniband/core/uverbs_main.c | 708 +++++++++++++++++++++++ 3 files changed, 1846 insertions(+) --- /dev/null 2005-06-23 14:14:38.423479552 -0700 +++ linux/drivers/infiniband/core/uverbs.h 2005-06-28 15:20:04.361964423 -0700 @@ -0,0 +1,132 @@ +/* + * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id: uverbs.h 2559 2005-06-06 19:43:16Z roland $ + */ + +#ifndef UVERBS_H +#define UVERBS_H + +/* Include device.h and fs.h until cdev.h is self-sufficient */ +#include +#include +#include +#include +#include + +#include +#include + +struct ib_uverbs_device { + int devnum; + struct cdev dev; + struct class_device class_dev; + struct ib_device *ib_dev; + int num_comp; +}; + +struct ib_uverbs_event_file { + struct kref ref; + struct ib_uverbs_file *uverbs_file; + spinlock_t lock; + int fd; + int is_async; + wait_queue_head_t poll_wait; + struct list_head event_list; +}; + +struct ib_uverbs_file { + struct kref ref; + struct ib_uverbs_device *device; + struct ib_ucontext *ucontext; + struct ib_event_handler event_handler; + struct ib_uverbs_event_file async_file; + struct ib_uverbs_event_file comp_file[1]; +}; + +struct ib_uverbs_async_event { + struct ib_uverbs_async_event_desc desc; + struct list_head list; +}; + +struct ib_uverbs_comp_event { + struct ib_uverbs_comp_event_desc desc; + struct list_head list; +}; + +struct ib_uobject_mr { + struct ib_uobject uobj; + struct page *page_list; + struct scatterlist *sg_list; +}; + +extern struct semaphore ib_uverbs_idr_mutex; +extern struct idr ib_uverbs_pd_idr; +extern struct idr ib_uverbs_mr_idr; +extern struct idr ib_uverbs_mw_idr; +extern struct idr ib_uverbs_ah_idr; +extern struct idr ib_uverbs_cq_idr; +extern struct idr ib_uverbs_qp_idr; + +void ib_uverbs_comp_handler(struct ib_cq *cq, void *cq_context); +void ib_uverbs_cq_event_handler(struct ib_event *event, void *context_ptr); +void ib_uverbs_qp_event_handler(struct ib_event *event, void *context_ptr); + +int ib_umem_get(struct ib_device *dev, struct ib_umem *mem, + void *addr, size_t size, int write); +void ib_umem_release(struct ib_device *dev, struct ib_umem *umem); +void ib_umem_release_on_close(struct ib_device *dev, struct ib_umem *umem); + +#define IB_UVERBS_DECLARE_CMD(name) \ + ssize_t ib_uverbs_##name(struct ib_uverbs_file *file, \ + const char __user *buf, int in_len, \ + int out_len) + +IB_UVERBS_DECLARE_CMD(query_params); +IB_UVERBS_DECLARE_CMD(get_context); +IB_UVERBS_DECLARE_CMD(query_device); +IB_UVERBS_DECLARE_CMD(query_port); +IB_UVERBS_DECLARE_CMD(query_gid); +IB_UVERBS_DECLARE_CMD(query_pkey); +IB_UVERBS_DECLARE_CMD(alloc_pd); +IB_UVERBS_DECLARE_CMD(dealloc_pd); +IB_UVERBS_DECLARE_CMD(reg_mr); +IB_UVERBS_DECLARE_CMD(dereg_mr); +IB_UVERBS_DECLARE_CMD(create_cq); +IB_UVERBS_DECLARE_CMD(destroy_cq); +IB_UVERBS_DECLARE_CMD(create_qp); +IB_UVERBS_DECLARE_CMD(modify_qp); +IB_UVERBS_DECLARE_CMD(destroy_qp); +IB_UVERBS_DECLARE_CMD(attach_mcast); +IB_UVERBS_DECLARE_CMD(detach_mcast); + +#endif /* UVERBS_H */ --- /dev/null 2005-06-23 14:14:38.423479552 -0700 +++ linux/drivers/infiniband/core/uverbs_cmd.c 2005-06-28 15:20:04.365963559 -0700 @@ -0,0 +1,1006 @@ +/* + * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id: uverbs_cmd.c 2708 2005-06-24 17:27:21Z roland $ + */ + +#include + +#include "uverbs.h" + +#define INIT_UDATA(udata, ibuf, obuf, ilen, olen) \ + do { \ + (udata)->inbuf = (void __user *) (ibuf); \ + (udata)->outbuf = (void __user *) (obuf); \ + (udata)->inlen = (ilen); \ + (udata)->outlen = (olen); \ + } while (0) + +ssize_t ib_uverbs_query_params(struct ib_uverbs_file *file, + const char __user *buf, + int in_len, int out_len) +{ + struct ib_uverbs_query_params cmd; + struct ib_uverbs_query_params_resp resp; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + memset(&resp, 0, sizeof resp); + + resp.num_cq_events = file->device->num_comp; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, &resp, sizeof resp)) + return -EFAULT; + + return in_len; +} + +ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file, + const char __user *buf, + int in_len, int out_len) +{ + struct ib_uverbs_get_context cmd; + struct ib_uverbs_get_context_resp resp; + struct ib_udata udata; + struct ib_device *ibdev = file->device->ib_dev; + int i; + int ret = in_len; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + INIT_UDATA(&udata, buf + sizeof cmd, + (unsigned long) cmd.response + sizeof resp, + in_len - sizeof cmd, out_len - sizeof resp); + + file->ucontext = ibdev->alloc_ucontext(ibdev, &udata); + if (IS_ERR(file->ucontext)) { + ret = PTR_ERR(file->ucontext); + file->ucontext = NULL; + return ret; + } + + file->ucontext->device = ibdev; + INIT_LIST_HEAD(&file->ucontext->pd_list); + INIT_LIST_HEAD(&file->ucontext->mr_list); + INIT_LIST_HEAD(&file->ucontext->mw_list); + INIT_LIST_HEAD(&file->ucontext->cq_list); + INIT_LIST_HEAD(&file->ucontext->qp_list); + INIT_LIST_HEAD(&file->ucontext->srq_list); + INIT_LIST_HEAD(&file->ucontext->ah_list); + spin_lock_init(&file->ucontext->lock); + + resp.async_fd = file->async_file.fd; + for (i = 0; i < file->device->num_comp; ++i) + if (copy_to_user((void __user *) (unsigned long) cmd.cq_fd_tab + + i * sizeof (__u32), + &file->comp_file[i].fd, sizeof (__u32))) + goto err; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) + goto err; + + return in_len; + +err: + ibdev->dealloc_ucontext(file->ucontext); + file->ucontext = NULL; + + return -EFAULT; +} + +ssize_t ib_uverbs_query_device(struct ib_uverbs_file *file, + const char __user *buf, + int in_len, int out_len) +{ + struct ib_uverbs_query_device cmd; + struct ib_uverbs_query_device_resp resp; + struct ib_device_attr attr; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + ret = ib_query_device(file->device->ib_dev, &attr); + if (ret) + return ret; + + memset(&resp, 0, sizeof resp); + + resp.fw_ver = attr.fw_ver; + resp.node_guid = attr.node_guid; + resp.sys_image_guid = attr.sys_image_guid; + resp.max_mr_size = attr.max_mr_size; + resp.page_size_cap = attr.page_size_cap; + resp.vendor_id = attr.vendor_id; + resp.vendor_part_id = attr.vendor_part_id; + resp.hw_ver = attr.hw_ver; + resp.max_qp = attr.max_qp; + resp.max_qp_wr = attr.max_qp_wr; + resp.device_cap_flags = attr.device_cap_flags; + resp.max_sge = attr.max_sge; + resp.max_sge_rd = attr.max_sge_rd; + resp.max_cq = attr.max_cq; + resp.max_cqe = attr.max_cqe; + resp.max_mr = attr.max_mr; + resp.max_pd = attr.max_pd; + resp.max_qp_rd_atom = attr.max_qp_rd_atom; + resp.max_ee_rd_atom = attr.max_ee_rd_atom; + resp.max_res_rd_atom = attr.max_res_rd_atom; + resp.max_qp_init_rd_atom = attr.max_qp_init_rd_atom; + resp.max_ee_init_rd_atom = attr.max_ee_init_rd_atom; + resp.atomic_cap = attr.atomic_cap; + resp.max_ee = attr.max_ee; + resp.max_rdd = attr.max_rdd; + resp.max_mw = attr.max_mw; + resp.max_raw_ipv6_qp = attr.max_raw_ipv6_qp; + resp.max_raw_ethy_qp = attr.max_raw_ethy_qp; + resp.max_mcast_grp = attr.max_mcast_grp; + resp.max_mcast_qp_attach = attr.max_mcast_qp_attach; + resp.max_total_mcast_qp_attach = attr.max_total_mcast_qp_attach; + resp.max_ah = attr.max_ah; + resp.max_fmr = attr.max_fmr; + resp.max_map_per_fmr = attr.max_map_per_fmr; + resp.max_srq = attr.max_srq; + resp.max_srq_wr = attr.max_srq_wr; + resp.max_srq_sge = attr.max_srq_sge; + resp.max_pkeys = attr.max_pkeys; + resp.local_ca_ack_delay = attr.local_ca_ack_delay; + resp.phys_port_cnt = file->device->ib_dev->phys_port_cnt; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) + return -EFAULT; + + return in_len; +} + +ssize_t ib_uverbs_query_port(struct ib_uverbs_file *file, + const char __user *buf, + int in_len, int out_len) +{ + struct ib_uverbs_query_port cmd; + struct ib_uverbs_query_port_resp resp; + struct ib_port_attr attr; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + ret = ib_query_port(file->device->ib_dev, cmd.port_num, &attr); + if (ret) + return ret; + + memset(&resp, 0, sizeof resp); + + resp.state = attr.state; + resp.max_mtu = attr.max_mtu; + resp.active_mtu = attr.active_mtu; + resp.gid_tbl_len = attr.gid_tbl_len; + resp.port_cap_flags = attr.port_cap_flags; + resp.max_msg_sz = attr.max_msg_sz; + resp.bad_pkey_cntr = attr.bad_pkey_cntr; + resp.qkey_viol_cntr = attr.qkey_viol_cntr; + resp.pkey_tbl_len = attr.pkey_tbl_len; + resp.lid = attr.lid; + resp.sm_lid = attr.sm_lid; + resp.lmc = attr.lmc; + resp.max_vl_num = attr.max_vl_num; + resp.sm_sl = attr.sm_sl; + resp.subnet_timeout = attr.subnet_timeout; + resp.init_type_reply = attr.init_type_reply; + resp.active_width = attr.active_width; + resp.active_speed = attr.active_speed; + resp.phys_state = attr.phys_state; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) + return -EFAULT; + + return in_len; +} + +ssize_t ib_uverbs_query_gid(struct ib_uverbs_file *file, + const char __user *buf, + int in_len, int out_len) +{ + struct ib_uverbs_query_gid cmd; + struct ib_uverbs_query_gid_resp resp; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + memset(&resp, 0, sizeof resp); + + ret = ib_query_gid(file->device->ib_dev, cmd.port_num, cmd.index, + (union ib_gid *) resp.gid); + if (ret) + return ret; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) + return -EFAULT; + + return in_len; +} + +ssize_t ib_uverbs_query_pkey(struct ib_uverbs_file *file, + const char __user *buf, + int in_len, int out_len) +{ + struct ib_uverbs_query_pkey cmd; + struct ib_uverbs_query_pkey_resp resp; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + memset(&resp, 0, sizeof resp); + + ret = ib_query_pkey(file->device->ib_dev, cmd.port_num, cmd.index, + &resp.pkey); + if (ret) + return ret; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) + return -EFAULT; + + return in_len; +} + +ssize_t ib_uverbs_alloc_pd(struct ib_uverbs_file *file, + const char __user *buf, + int in_len, int out_len) +{ + struct ib_uverbs_alloc_pd cmd; + struct ib_uverbs_alloc_pd_resp resp; + struct ib_udata udata; + struct ib_uobject *uobj; + struct ib_pd *pd; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + INIT_UDATA(&udata, buf + sizeof cmd, + (unsigned long) cmd.response + sizeof resp, + in_len - sizeof cmd, out_len - sizeof resp); + + uobj = kmalloc(sizeof *uobj, GFP_KERNEL); + if (!uobj) + return -ENOMEM; + + uobj->context = file->ucontext; + + pd = file->device->ib_dev->alloc_pd(file->device->ib_dev, + file->ucontext, &udata); + if (IS_ERR(pd)) { + ret = PTR_ERR(pd); + goto err; + } + + pd->device = file->device->ib_dev; + pd->uobject = uobj; + atomic_set(&pd->usecnt, 0); + +retry: + if (!idr_pre_get(&ib_uverbs_pd_idr, GFP_KERNEL)) { + ret = -ENOMEM; + goto err_pd; + } + + down(&ib_uverbs_idr_mutex); + ret = idr_get_new(&ib_uverbs_pd_idr, pd, &uobj->id); + up(&ib_uverbs_idr_mutex); + + if (ret == -EAGAIN) + goto retry; + if (ret) + goto err_pd; + + spin_lock_irq(&file->ucontext->lock); + list_add_tail(&uobj->list, &file->ucontext->pd_list); + spin_unlock_irq(&file->ucontext->lock); + + memset(&resp, 0, sizeof resp); + resp.pd_handle = uobj->id; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) { + ret = -EFAULT; + goto err_list; + } + + return in_len; + +err_list: + spin_lock_irq(&file->ucontext->lock); + list_del(&uobj->list); + spin_unlock_irq(&file->ucontext->lock); + + down(&ib_uverbs_idr_mutex); + idr_remove(&ib_uverbs_pd_idr, uobj->id); + up(&ib_uverbs_idr_mutex); + +err_pd: + ib_dealloc_pd(pd); + +err: + kfree(uobj); + return ret; +} + +ssize_t ib_uverbs_dealloc_pd(struct ib_uverbs_file *file, + const char __user *buf, + int in_len, int out_len) +{ + struct ib_uverbs_dealloc_pd cmd; + struct ib_pd *pd; + struct ib_uobject *uobj; + int ret = -EINVAL; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + down(&ib_uverbs_idr_mutex); + + pd = idr_find(&ib_uverbs_pd_idr, cmd.pd_handle); + if (!pd || pd->uobject->context != file->ucontext) + goto out; + + uobj = pd->uobject; + + ret = ib_dealloc_pd(pd); + if (ret) + goto out; + + idr_remove(&ib_uverbs_pd_idr, cmd.pd_handle); + + spin_lock_irq(&file->ucontext->lock); + list_del(&uobj->list); + spin_unlock_irq(&file->ucontext->lock); + + kfree(uobj); + +out: + up(&ib_uverbs_idr_mutex); + + return ret ? ret : in_len; +} + +ssize_t ib_uverbs_reg_mr(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_reg_mr cmd; + struct ib_uverbs_reg_mr_resp resp; + struct ib_udata udata; + struct ib_umem_object *obj; + struct ib_pd *pd; + struct ib_mr *mr; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + INIT_UDATA(&udata, buf + sizeof cmd, + (unsigned long) cmd.response + sizeof resp, + in_len - sizeof cmd, out_len - sizeof resp); + + if ((cmd.start & ~PAGE_MASK) != (cmd.hca_va & ~PAGE_MASK)) + return -EINVAL; + + obj = kmalloc(sizeof *obj, GFP_KERNEL); + if (!obj) + return -ENOMEM; + + obj->uobject.context = file->ucontext; + + /* + * We ask for writable memory if any access flags other than + * "remote read" are set. "Local write" and "remote write" + * obviously require write access. "Remote atomic" can do + * things like fetch and add, which will modify memory, and + * "MW bind" can change permissions by binding a window. + */ + ret = ib_umem_get(file->device->ib_dev, &obj->umem, + (void *) (unsigned long) cmd.start, cmd.length, + !!(cmd.access_flags & ~IB_ACCESS_REMOTE_READ)); + if (ret) + goto err_free; + + obj->umem.virt_base = cmd.hca_va; + + down(&ib_uverbs_idr_mutex); + + pd = idr_find(&ib_uverbs_pd_idr, cmd.pd_handle); + if (!pd || pd->uobject->context != file->ucontext) { + ret = -EINVAL; + goto err_up; + } + + if (!pd->device->reg_user_mr) { + ret = -ENOSYS; + goto err_up; + } + + mr = pd->device->reg_user_mr(pd, &obj->umem, cmd.access_flags, &udata); + if (IS_ERR(mr)) { + ret = PTR_ERR(mr); + goto err_up; + } + + mr->device = pd->device; + mr->pd = pd; + mr->uobject = &obj->uobject; + atomic_inc(&pd->usecnt); + atomic_set(&mr->usecnt, 0); + + memset(&resp, 0, sizeof resp); + resp.lkey = mr->lkey; + resp.rkey = mr->rkey; + +retry: + if (!idr_pre_get(&ib_uverbs_mr_idr, GFP_KERNEL)) { + ret = -ENOMEM; + goto err_unreg; + } + + ret = idr_get_new(&ib_uverbs_mr_idr, mr, &obj->uobject.id); + + if (ret == -EAGAIN) + goto retry; + if (ret) + goto err_unreg; + + resp.mr_handle = obj->uobject.id; + + spin_lock_irq(&file->ucontext->lock); + list_add_tail(&obj->uobject.list, &file->ucontext->mr_list); + spin_unlock_irq(&file->ucontext->lock); + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) { + ret = -EFAULT; + goto err_list; + } + + up(&ib_uverbs_idr_mutex); + + return in_len; + +err_list: + spin_lock_irq(&file->ucontext->lock); + list_del(&obj->uobject.list); + spin_unlock_irq(&file->ucontext->lock); + +err_unreg: + ib_dereg_mr(mr); + +err_up: + up(&ib_uverbs_idr_mutex); + + ib_umem_release(file->device->ib_dev, &obj->umem); + +err_free: + kfree(obj); + return ret; +} + +ssize_t ib_uverbs_dereg_mr(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_dereg_mr cmd; + struct ib_mr *mr; + struct ib_umem_object *memobj; + int ret = -EINVAL; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + down(&ib_uverbs_idr_mutex); + + mr = idr_find(&ib_uverbs_mr_idr, cmd.mr_handle); + if (!mr || mr->uobject->context != file->ucontext) + goto out; + + memobj = container_of(mr->uobject, struct ib_umem_object, uobject); + + ret = ib_dereg_mr(mr); + if (ret) + goto out; + + idr_remove(&ib_uverbs_mr_idr, cmd.mr_handle); + + spin_lock_irq(&file->ucontext->lock); + list_del(&memobj->uobject.list); + spin_unlock_irq(&file->ucontext->lock); + + ib_umem_release(file->device->ib_dev, &memobj->umem); + kfree(memobj); + +out: + up(&ib_uverbs_idr_mutex); + + return ret ? ret : in_len; +} + +ssize_t ib_uverbs_create_cq(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_create_cq cmd; + struct ib_uverbs_create_cq_resp resp; + struct ib_udata udata; + struct ib_uobject *uobj; + struct ib_cq *cq; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + INIT_UDATA(&udata, buf + sizeof cmd, + (unsigned long) cmd.response + sizeof resp, + in_len - sizeof cmd, out_len - sizeof resp); + + if (cmd.event_handler >= file->device->num_comp) + return -EINVAL; + + uobj = kmalloc(sizeof *uobj, GFP_KERNEL); + if (!uobj) + return -ENOMEM; + + uobj->user_handle = cmd.user_handle; + uobj->context = file->ucontext; + + cq = file->device->ib_dev->create_cq(file->device->ib_dev, cmd.cqe, + file->ucontext, &udata); + if (IS_ERR(cq)) { + ret = PTR_ERR(cq); + goto err; + } + + cq->device = file->device->ib_dev; + cq->uobject = uobj; + cq->comp_handler = ib_uverbs_comp_handler; + cq->event_handler = ib_uverbs_cq_event_handler; + cq->cq_context = file; + atomic_set(&cq->usecnt, 0); + +retry: + if (!idr_pre_get(&ib_uverbs_cq_idr, GFP_KERNEL)) { + ret = -ENOMEM; + goto err_cq; + } + + down(&ib_uverbs_idr_mutex); + ret = idr_get_new(&ib_uverbs_cq_idr, cq, &uobj->id); + up(&ib_uverbs_idr_mutex); + + if (ret == -EAGAIN) + goto retry; + if (ret) + goto err_cq; + + spin_lock_irq(&file->ucontext->lock); + list_add_tail(&uobj->list, &file->ucontext->cq_list); + spin_unlock_irq(&file->ucontext->lock); + + memset(&resp, 0, sizeof resp); + resp.cq_handle = uobj->id; + resp.cqe = cq->cqe; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) { + ret = -EFAULT; + goto err_list; + } + + return in_len; + +err_list: + spin_lock_irq(&file->ucontext->lock); + list_del(&uobj->list); + spin_unlock_irq(&file->ucontext->lock); + + down(&ib_uverbs_idr_mutex); + idr_remove(&ib_uverbs_cq_idr, uobj->id); + up(&ib_uverbs_idr_mutex); + +err_cq: + ib_destroy_cq(cq); + +err: + kfree(uobj); + return ret; +} + +ssize_t ib_uverbs_destroy_cq(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_destroy_cq cmd; + struct ib_cq *cq; + struct ib_uobject *uobj; + int ret = -EINVAL; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + down(&ib_uverbs_idr_mutex); + + cq = idr_find(&ib_uverbs_cq_idr, cmd.cq_handle); + if (!cq || cq->uobject->context != file->ucontext) + goto out; + + uobj = cq->uobject; + + ret = ib_destroy_cq(cq); + if (ret) + goto out; + + idr_remove(&ib_uverbs_cq_idr, cmd.cq_handle); + + spin_lock_irq(&file->ucontext->lock); + list_del(&uobj->list); + spin_unlock_irq(&file->ucontext->lock); + + kfree(uobj); + +out: + up(&ib_uverbs_idr_mutex); + + return ret ? ret : in_len; +} + +ssize_t ib_uverbs_create_qp(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_create_qp cmd; + struct ib_uverbs_create_qp_resp resp; + struct ib_udata udata; + struct ib_uobject *uobj; + struct ib_pd *pd; + struct ib_cq *scq, *rcq; + struct ib_qp *qp; + struct ib_qp_init_attr attr; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + INIT_UDATA(&udata, buf + sizeof cmd, + (unsigned long) cmd.response + sizeof resp, + in_len - sizeof cmd, out_len - sizeof resp); + + uobj = kmalloc(sizeof *uobj, GFP_KERNEL); + if (!uobj) + return -ENOMEM; + + down(&ib_uverbs_idr_mutex); + + pd = idr_find(&ib_uverbs_pd_idr, cmd.pd_handle); + scq = idr_find(&ib_uverbs_cq_idr, cmd.send_cq_handle); + rcq = idr_find(&ib_uverbs_cq_idr, cmd.recv_cq_handle); + + if (!pd || pd->uobject->context != file->ucontext || + !scq || scq->uobject->context != file->ucontext || + !rcq || rcq->uobject->context != file->ucontext) { + ret = -EINVAL; + goto err_up; + } + + attr.event_handler = ib_uverbs_qp_event_handler; + attr.qp_context = file; + attr.send_cq = scq; + attr.recv_cq = rcq; + attr.srq = NULL; + attr.sq_sig_type = cmd.sq_sig_all ? IB_SIGNAL_ALL_WR : IB_SIGNAL_REQ_WR; + attr.qp_type = cmd.qp_type; + + attr.cap.max_send_wr = cmd.max_send_wr; + attr.cap.max_recv_wr = cmd.max_recv_wr; + attr.cap.max_send_sge = cmd.max_send_sge; + attr.cap.max_recv_sge = cmd.max_recv_sge; + attr.cap.max_inline_data = cmd.max_inline_data; + + uobj->user_handle = cmd.user_handle; + uobj->context = file->ucontext; + + qp = pd->device->create_qp(pd, &attr, &udata); + if (IS_ERR(qp)) { + ret = PTR_ERR(qp); + goto err_up; + } + + qp->device = pd->device; + qp->pd = pd; + qp->send_cq = attr.send_cq; + qp->recv_cq = attr.recv_cq; + qp->srq = attr.srq; + qp->uobject = uobj; + qp->event_handler = attr.event_handler; + qp->qp_context = attr.qp_context; + qp->qp_type = attr.qp_type; + atomic_inc(&pd->usecnt); + atomic_inc(&attr.send_cq->usecnt); + atomic_inc(&attr.recv_cq->usecnt); + if (attr.srq) + atomic_inc(&attr.srq->usecnt); + + memset(&resp, 0, sizeof resp); + resp.qpn = qp->qp_num; + +retry: + if (!idr_pre_get(&ib_uverbs_qp_idr, GFP_KERNEL)) { + ret = -ENOMEM; + goto err_destroy; + } + + ret = idr_get_new(&ib_uverbs_qp_idr, qp, &uobj->id); + + if (ret == -EAGAIN) + goto retry; + if (ret) + goto err_destroy; + + resp.qp_handle = uobj->id; + + spin_lock_irq(&file->ucontext->lock); + list_add_tail(&uobj->list, &file->ucontext->qp_list); + spin_unlock_irq(&file->ucontext->lock); + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) { + ret = -EFAULT; + goto err_list; + } + + up(&ib_uverbs_idr_mutex); + + return in_len; + +err_list: + spin_lock_irq(&file->ucontext->lock); + list_del(&uobj->list); + spin_unlock_irq(&file->ucontext->lock); + +err_destroy: + ib_destroy_qp(qp); + +err_up: + up(&ib_uverbs_idr_mutex); + + kfree(uobj); + return ret; +} + +ssize_t ib_uverbs_modify_qp(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_modify_qp cmd; + struct ib_qp *qp; + struct ib_qp_attr *attr; + int ret; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + attr = kmalloc(sizeof *attr, GFP_KERNEL); + if (!attr) + return -ENOMEM; + + down(&ib_uverbs_idr_mutex); + + qp = idr_find(&ib_uverbs_qp_idr, cmd.qp_handle); + if (!qp || qp->uobject->context != file->ucontext) { + ret = -EINVAL; + goto out; + } + + attr->qp_state = cmd.qp_state; + attr->cur_qp_state = cmd.cur_qp_state; + attr->path_mtu = cmd.path_mtu; + attr->path_mig_state = cmd.path_mig_state; + attr->qkey = cmd.qkey; + attr->rq_psn = cmd.rq_psn; + attr->sq_psn = cmd.sq_psn; + attr->dest_qp_num = cmd.dest_qp_num; + attr->qp_access_flags = cmd.qp_access_flags; + attr->pkey_index = cmd.pkey_index; + attr->alt_pkey_index = cmd.pkey_index; + attr->en_sqd_async_notify = cmd.en_sqd_async_notify; + attr->max_rd_atomic = cmd.max_rd_atomic; + attr->max_dest_rd_atomic = cmd.max_dest_rd_atomic; + attr->min_rnr_timer = cmd.min_rnr_timer; + attr->port_num = cmd.port_num; + attr->timeout = cmd.timeout; + attr->retry_cnt = cmd.retry_cnt; + attr->rnr_retry = cmd.rnr_retry; + attr->alt_port_num = cmd.alt_port_num; + attr->alt_timeout = cmd.alt_timeout; + + memcpy(attr->ah_attr.grh.dgid.raw, cmd.dest.dgid, 16); + attr->ah_attr.grh.flow_label = cmd.dest.flow_label; + attr->ah_attr.grh.sgid_index = cmd.dest.sgid_index; + attr->ah_attr.grh.hop_limit = cmd.dest.hop_limit; + attr->ah_attr.grh.traffic_class = cmd.dest.traffic_class; + attr->ah_attr.dlid = cmd.dest.dlid; + attr->ah_attr.sl = cmd.dest.sl; + attr->ah_attr.src_path_bits = cmd.dest.src_path_bits; + attr->ah_attr.static_rate = cmd.dest.static_rate; + attr->ah_attr.ah_flags = cmd.dest.is_global ? IB_AH_GRH : 0; + attr->ah_attr.port_num = cmd.dest.port_num; + + memcpy(attr->alt_ah_attr.grh.dgid.raw, cmd.alt_dest.dgid, 16); + attr->alt_ah_attr.grh.flow_label = cmd.alt_dest.flow_label; + attr->alt_ah_attr.grh.sgid_index = cmd.alt_dest.sgid_index; + attr->alt_ah_attr.grh.hop_limit = cmd.alt_dest.hop_limit; + attr->alt_ah_attr.grh.traffic_class = cmd.alt_dest.traffic_class; + attr->alt_ah_attr.dlid = cmd.alt_dest.dlid; + attr->alt_ah_attr.sl = cmd.alt_dest.sl; + attr->alt_ah_attr.src_path_bits = cmd.alt_dest.src_path_bits; + attr->alt_ah_attr.static_rate = cmd.alt_dest.static_rate; + attr->alt_ah_attr.ah_flags = cmd.alt_dest.is_global ? IB_AH_GRH : 0; + attr->alt_ah_attr.port_num = cmd.alt_dest.port_num; + + ret = ib_modify_qp(qp, attr, cmd.attr_mask); + if (ret) + goto out; + + ret = in_len; + +out: + up(&ib_uverbs_idr_mutex); + kfree(attr); + + return ret; +} + +ssize_t ib_uverbs_destroy_qp(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_destroy_qp cmd; + struct ib_qp *qp; + struct ib_uobject *uobj; + int ret = -EINVAL; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + down(&ib_uverbs_idr_mutex); + + qp = idr_find(&ib_uverbs_qp_idr, cmd.qp_handle); + if (!qp || qp->uobject->context != file->ucontext) + goto out; + + uobj = qp->uobject; + + ret = ib_destroy_qp(qp); + if (ret) + goto out; + + idr_remove(&ib_uverbs_qp_idr, cmd.qp_handle); + + spin_lock_irq(&file->ucontext->lock); + list_del(&uobj->list); + spin_unlock_irq(&file->ucontext->lock); + + kfree(uobj); + +out: + up(&ib_uverbs_idr_mutex); + + return ret ? ret : in_len; +} + +ssize_t ib_uverbs_attach_mcast(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_attach_mcast cmd; + struct ib_qp *qp; + int ret = -EINVAL; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + down(&ib_uverbs_idr_mutex); + + qp = idr_find(&ib_uverbs_qp_idr, cmd.qp_handle); + if (qp && qp->uobject->context == file->ucontext) + ret = ib_attach_mcast(qp, (union ib_gid *) cmd.gid, cmd.mlid); + + up(&ib_uverbs_idr_mutex); + + return ret ? ret : in_len; +} + +ssize_t ib_uverbs_detach_mcast(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_detach_mcast cmd; + struct ib_qp *qp; + int ret = -EINVAL; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + down(&ib_uverbs_idr_mutex); + + qp = idr_find(&ib_uverbs_qp_idr, cmd.qp_handle); + if (qp && qp->uobject->context == file->ucontext) + ret = ib_detach_mcast(qp, (union ib_gid *) cmd.gid, cmd.mlid); + + up(&ib_uverbs_idr_mutex); + + return ret ? ret : in_len; +} --- /dev/null 2005-06-23 14:14:38.423479552 -0700 +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-28 15:20:04.363963991 -0700 @@ -0,0 +1,708 @@ +/* + * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id: uverbs_main.c 2733 2005-06-28 19:14:34Z roland $ + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "uverbs.h" + +MODULE_AUTHOR("Roland Dreier"); +MODULE_DESCRIPTION("InfiniBand userspace verbs access"); +MODULE_LICENSE("Dual BSD/GPL"); + +#define INFINIBANDEVENTFS_MAGIC 0x49426576 /* "IBev" */ + +enum { + IB_UVERBS_MAJOR = 231, + IB_UVERBS_BASE_MINOR = 192, + IB_UVERBS_MAX_DEVICES = 32 +}; + +#define IB_UVERBS_BASE_DEV MKDEV(IB_UVERBS_MAJOR, IB_UVERBS_BASE_MINOR) + +DECLARE_MUTEX(ib_uverbs_idr_mutex); +DEFINE_IDR(ib_uverbs_pd_idr); +DEFINE_IDR(ib_uverbs_mr_idr); +DEFINE_IDR(ib_uverbs_mw_idr); +DEFINE_IDR(ib_uverbs_ah_idr); +DEFINE_IDR(ib_uverbs_cq_idr); +DEFINE_IDR(ib_uverbs_qp_idr); + +static spinlock_t map_lock; +static DECLARE_BITMAP(dev_map, IB_UVERBS_MAX_DEVICES); + +static ssize_t (*uverbs_cmd_table[])(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) = { + [IB_USER_VERBS_CMD_QUERY_PARAMS] = ib_uverbs_query_params, + [IB_USER_VERBS_CMD_GET_CONTEXT] = ib_uverbs_get_context, + [IB_USER_VERBS_CMD_QUERY_DEVICE] = ib_uverbs_query_device, + [IB_USER_VERBS_CMD_QUERY_PORT] = ib_uverbs_query_port, + [IB_USER_VERBS_CMD_QUERY_GID] = ib_uverbs_query_gid, + [IB_USER_VERBS_CMD_QUERY_PKEY] = ib_uverbs_query_pkey, + [IB_USER_VERBS_CMD_ALLOC_PD] = ib_uverbs_alloc_pd, + [IB_USER_VERBS_CMD_DEALLOC_PD] = ib_uverbs_dealloc_pd, + [IB_USER_VERBS_CMD_REG_MR] = ib_uverbs_reg_mr, + [IB_USER_VERBS_CMD_DEREG_MR] = ib_uverbs_dereg_mr, + [IB_USER_VERBS_CMD_CREATE_CQ] = ib_uverbs_create_cq, + [IB_USER_VERBS_CMD_DESTROY_CQ] = ib_uverbs_destroy_cq, + [IB_USER_VERBS_CMD_CREATE_QP] = ib_uverbs_create_qp, + [IB_USER_VERBS_CMD_MODIFY_QP] = ib_uverbs_modify_qp, + [IB_USER_VERBS_CMD_DESTROY_QP] = ib_uverbs_destroy_qp, + [IB_USER_VERBS_CMD_ATTACH_MCAST] = ib_uverbs_attach_mcast, + [IB_USER_VERBS_CMD_DETACH_MCAST] = ib_uverbs_detach_mcast, +}; + +static struct vfsmount *uverbs_event_mnt; + +static void ib_uverbs_add_one(struct ib_device *device); +static void ib_uverbs_remove_one(struct ib_device *device); + +static int ib_dealloc_ucontext(struct ib_ucontext *context) +{ + struct ib_uobject *uobj, *tmp; + + if (!context) + return 0; + + down(&ib_uverbs_idr_mutex); + + /* XXX Free AHs */ + + list_for_each_entry_safe(uobj, tmp, &context->qp_list, list) { + struct ib_qp *qp = idr_find(&ib_uverbs_qp_idr, uobj->id); + idr_remove(&ib_uverbs_qp_idr, uobj->id); + ib_destroy_qp(qp); + list_del(&uobj->list); + kfree(uobj); + } + + list_for_each_entry_safe(uobj, tmp, &context->cq_list, list) { + struct ib_cq *cq = idr_find(&ib_uverbs_cq_idr, uobj->id); + idr_remove(&ib_uverbs_cq_idr, uobj->id); + ib_destroy_cq(cq); + list_del(&uobj->list); + kfree(uobj); + } + + /* XXX Free SRQs */ + /* XXX Free MWs */ + + list_for_each_entry_safe(uobj, tmp, &context->mr_list, list) { + struct ib_mr *mr = idr_find(&ib_uverbs_mr_idr, uobj->id); + struct ib_umem_object *memobj; + + idr_remove(&ib_uverbs_mr_idr, uobj->id); + ib_dereg_mr(mr); + + memobj = container_of(uobj, struct ib_umem_object, uobject); + ib_umem_release_on_close(mr->device, &memobj->umem); + + list_del(&uobj->list); + kfree(memobj); + } + + list_for_each_entry_safe(uobj, tmp, &context->pd_list, list) { + struct ib_pd *pd = idr_find(&ib_uverbs_pd_idr, uobj->id); + idr_remove(&ib_uverbs_pd_idr, uobj->id); + ib_dealloc_pd(pd); + list_del(&uobj->list); + kfree(uobj); + } + + up(&ib_uverbs_idr_mutex); + + return context->device->dealloc_ucontext(context); +} + +static void ib_uverbs_release_file(struct kref *ref) +{ + struct ib_uverbs_file *file = + container_of(ref, struct ib_uverbs_file, ref); + + module_put(file->device->ib_dev->owner); + kfree(file); +} + +static ssize_t ib_uverbs_event_read(struct file *filp, char __user *buf, + size_t count, loff_t *pos) +{ + struct ib_uverbs_event_file *file = filp->private_data; + void *event; + int eventsz; + int ret = 0; + + spin_lock_irq(&file->lock); + + while (list_empty(&file->event_list) && file->fd >= 0) { + spin_unlock_irq(&file->lock); + + if (filp->f_flags & O_NONBLOCK) + return -EAGAIN; + + if (wait_event_interruptible(file->poll_wait, + !list_empty(&file->event_list) || + file->fd < 0)) + return -ERESTARTSYS; + + spin_lock_irq(&file->lock); + } + + if (file->fd < 0) { + spin_unlock_irq(&file->lock); + return -ENODEV; + } + + if (file->is_async) { + event = list_entry(file->event_list.next, + struct ib_uverbs_async_event, list); + eventsz = sizeof (struct ib_uverbs_async_event_desc); + } else { + event = list_entry(file->event_list.next, + struct ib_uverbs_comp_event, list); + eventsz = sizeof (struct ib_uverbs_comp_event_desc); + } + + if (eventsz > count) { + ret = -EINVAL; + event = NULL; + } else + list_del(file->event_list.next); + + spin_unlock_irq(&file->lock); + + if (event) { + if (copy_to_user(buf, event, eventsz)) + ret = -EFAULT; + else + ret = eventsz; + } + + kfree(event); + + return ret; +} + +static unsigned int ib_uverbs_event_poll(struct file *filp, + struct poll_table_struct *wait) +{ + unsigned int pollflags = 0; + struct ib_uverbs_event_file *file = filp->private_data; + + poll_wait(filp, &file->poll_wait, wait); + + spin_lock_irq(&file->lock); + if (file->fd < 0) + pollflags = POLLERR; + else if (!list_empty(&file->event_list)) + pollflags = POLLIN | POLLRDNORM; + spin_unlock_irq(&file->lock); + + return pollflags; +} + +static void ib_uverbs_event_release(struct ib_uverbs_event_file *file) +{ + struct list_head *entry, *tmp; + + spin_lock_irq(&file->lock); + if (file->fd != -1) { + file->fd = -1; + list_for_each_safe(entry, tmp, &file->event_list) + if (file->is_async) + kfree(list_entry(entry, struct ib_uverbs_async_event, list)); + else + kfree(list_entry(entry, struct ib_uverbs_comp_event, list)); + } + spin_unlock_irq(&file->lock); +} + +static int ib_uverbs_event_close(struct inode *inode, struct file *filp) +{ + struct ib_uverbs_event_file *file = filp->private_data; + + ib_uverbs_event_release(file); + kref_put(&file->uverbs_file->ref, ib_uverbs_release_file); + + return 0; +} + +static struct file_operations uverbs_event_fops = { + /* + * No .owner field since we artificially create event files, + * so there is no increment to the module reference count in + * the open path. All event files come from a uverbs command + * file, which already takes a module reference, so this is OK. + */ + .read = ib_uverbs_event_read, + .poll = ib_uverbs_event_poll, + .release = ib_uverbs_event_close +}; + +void ib_uverbs_comp_handler(struct ib_cq *cq, void *cq_context) +{ + struct ib_uverbs_file *file = cq_context; + struct ib_uverbs_comp_event *entry; + unsigned long flags; + + entry = kmalloc(sizeof *entry, GFP_ATOMIC); + if (!entry) + return; + + entry->desc.cq_handle = cq->uobject->user_handle; + + spin_lock_irqsave(&file->comp_file[0].lock, flags); + list_add_tail(&entry->list, &file->comp_file[0].event_list); + spin_unlock_irqrestore(&file->comp_file[0].lock, flags); + + wake_up_interruptible(&file->comp_file[0].poll_wait); +} + +static void ib_uverbs_async_handler(struct ib_uverbs_file *file, + __u64 element, __u64 event) +{ + struct ib_uverbs_async_event *entry; + unsigned long flags; + + entry = kmalloc(sizeof *entry, GFP_ATOMIC); + if (!entry) + return; + + entry->desc.element = element; + entry->desc.event_type = event; + + spin_lock_irqsave(&file->async_file.lock, flags); + list_add_tail(&entry->list, &file->async_file.event_list); + spin_unlock_irqrestore(&file->async_file.lock, flags); + + wake_up_interruptible(&file->async_file.poll_wait); +} + +void ib_uverbs_cq_event_handler(struct ib_event *event, void *context_ptr) +{ + ib_uverbs_async_handler(context_ptr, + event->element.cq->uobject->user_handle, + event->event); +} + +void ib_uverbs_qp_event_handler(struct ib_event *event, void *context_ptr) +{ + ib_uverbs_async_handler(context_ptr, + event->element.qp->uobject->user_handle, + event->event); +} + +static void ib_uverbs_event_handler(struct ib_event_handler *handler, + struct ib_event *event) +{ + struct ib_uverbs_file *file = + container_of(handler, struct ib_uverbs_file, event_handler); + + ib_uverbs_async_handler(file, event->element.port_num, event->event); +} + +static int ib_uverbs_event_init(struct ib_uverbs_event_file *file, + struct ib_uverbs_file *uverbs_file) +{ + struct file *filp; + + spin_lock_init(&file->lock); + INIT_LIST_HEAD(&file->event_list); + init_waitqueue_head(&file->poll_wait); + file->uverbs_file = uverbs_file; + + file->fd = get_unused_fd(); + if (file->fd < 0) + return file->fd; + + filp = get_empty_filp(); + if (!filp) { + put_unused_fd(file->fd); + return -ENFILE; + } + + filp->f_op = &uverbs_event_fops; + filp->f_vfsmnt = mntget(uverbs_event_mnt); + filp->f_dentry = dget(uverbs_event_mnt->mnt_root); + filp->f_mapping = filp->f_dentry->d_inode->i_mapping; + filp->f_flags = O_RDONLY; + filp->f_mode = FMODE_READ; + filp->private_data = file; + + fd_install(file->fd, filp); + + return 0; +} + +static ssize_t ib_uverbs_write(struct file *filp, const char __user *buf, + size_t count, loff_t *pos) +{ + struct ib_uverbs_file *file = filp->private_data; + struct ib_uverbs_cmd_hdr hdr; + + if (count < sizeof hdr) + return -EINVAL; + + if (copy_from_user(&hdr, buf, sizeof hdr)) + return -EFAULT; + + if (hdr.in_words * 4 != count) + return -EINVAL; + + if (hdr.command < 0 || hdr.command >= ARRAY_SIZE(uverbs_cmd_table)) + return -EINVAL; + + if (!file->ucontext && + hdr.command != IB_USER_VERBS_CMD_QUERY_PARAMS && + hdr.command != IB_USER_VERBS_CMD_GET_CONTEXT) + return -EINVAL; + + return uverbs_cmd_table[hdr.command](file, buf + sizeof hdr, + hdr.in_words * 4, hdr.out_words * 4); +} + +static int ib_uverbs_mmap(struct file *filp, struct vm_area_struct *vma) +{ + struct ib_uverbs_file *file = filp->private_data; + + if (!file->ucontext) + return -ENODEV; + else + return file->device->ib_dev->mmap(file->ucontext, vma); +} + +static int ib_uverbs_open(struct inode *inode, struct file *filp) +{ + struct ib_uverbs_device *dev = + container_of(inode->i_cdev, struct ib_uverbs_device, dev); + struct ib_uverbs_file *file; + int i = 0; + int ret; + + if (!try_module_get(dev->ib_dev->owner)) + return -ENODEV; + + file = kmalloc(sizeof *file + + (dev->num_comp - 1) * sizeof (struct ib_uverbs_event_file), + GFP_KERNEL); + if (!file) + return -ENOMEM; + + file->device = dev; + kref_init(&file->ref); + + file->ucontext = NULL; + + ret = ib_uverbs_event_init(&file->async_file, file); + if (ret) + goto err; + + file->async_file.is_async = 1; + + kref_get(&file->ref); + + for (i = 0; i < dev->num_comp; ++i) { + ret = ib_uverbs_event_init(&file->comp_file[i], file); + if (ret) + goto err_async; + kref_get(&file->ref); + file->comp_file[i].is_async = 0; + } + + + filp->private_data = file; + + INIT_IB_EVENT_HANDLER(&file->event_handler, dev->ib_dev, + ib_uverbs_event_handler); + if (ib_register_event_handler(&file->event_handler)) + goto err_async; + + return 0; + +err_async: + while (i--) + ib_uverbs_event_release(&file->comp_file[i]); + + ib_uverbs_event_release(&file->async_file); + +err: + kref_put(&file->ref, ib_uverbs_release_file); + + return ret; +} + +static int ib_uverbs_close(struct inode *inode, struct file *filp) +{ + struct ib_uverbs_file *file = filp->private_data; + int i; + + ib_unregister_event_handler(&file->event_handler); + ib_uverbs_event_release(&file->async_file); + ib_dealloc_ucontext(file->ucontext); + + for (i = 0; i < file->device->num_comp; ++i) + ib_uverbs_event_release(&file->comp_file[i]); + + kref_put(&file->ref, ib_uverbs_release_file); + + return 0; +} + +static struct file_operations uverbs_fops = { + .owner = THIS_MODULE, + .write = ib_uverbs_write, + .open = ib_uverbs_open, + .release = ib_uverbs_close +}; + +static struct file_operations uverbs_mmap_fops = { + .owner = THIS_MODULE, + .write = ib_uverbs_write, + .mmap = ib_uverbs_mmap, + .open = ib_uverbs_open, + .release = ib_uverbs_close +}; + +static struct ib_client uverbs_client = { + .name = "uverbs", + .add = ib_uverbs_add_one, + .remove = ib_uverbs_remove_one +}; + +static ssize_t show_dev(struct class_device *class_dev, char *buf) +{ + struct ib_uverbs_device *dev = + container_of(class_dev, struct ib_uverbs_device, class_dev); + + return print_dev_t(buf, dev->dev.dev); +} +static CLASS_DEVICE_ATTR(dev, S_IRUGO, show_dev, NULL); + +static ssize_t show_ibdev(struct class_device *class_dev, char *buf) +{ + struct ib_uverbs_device *dev = + container_of(class_dev, struct ib_uverbs_device, class_dev); + + return sprintf(buf, "%s\n", dev->ib_dev->name); +} +static CLASS_DEVICE_ATTR(ibdev, S_IRUGO, show_ibdev, NULL); + +static void ib_uverbs_release_class_dev(struct class_device *class_dev) +{ + struct ib_uverbs_device *dev = + container_of(class_dev, struct ib_uverbs_device, class_dev); + + cdev_del(&dev->dev); + clear_bit(dev->devnum, dev_map); + kfree(dev); +} + +static struct class uverbs_class = { + .name = "infiniband_verbs", + .release = ib_uverbs_release_class_dev +}; + +static ssize_t show_abi_version(struct class *class, char *buf) +{ + return sprintf(buf, "%d\n", IB_USER_VERBS_ABI_VERSION); +} +static CLASS_ATTR(abi_version, S_IRUGO, show_abi_version, NULL); + +static void ib_uverbs_add_one(struct ib_device *device) +{ + struct ib_uverbs_device *uverbs_dev; + + if (!device->alloc_ucontext) + return; + + uverbs_dev = kmalloc(sizeof *uverbs_dev, GFP_KERNEL); + if (!uverbs_dev) + return; + + memset(uverbs_dev, 0, sizeof *uverbs_dev); + + spin_lock(&map_lock); + uverbs_dev->devnum = find_first_zero_bit(dev_map, IB_UVERBS_MAX_DEVICES); + if (uverbs_dev->devnum >= IB_UVERBS_MAX_DEVICES) { + spin_unlock(&map_lock); + goto err; + } + set_bit(uverbs_dev->devnum, dev_map); + spin_unlock(&map_lock); + + uverbs_dev->ib_dev = device; + uverbs_dev->num_comp = 1; + + if (device->mmap) + cdev_init(&uverbs_dev->dev, &uverbs_mmap_fops); + else + cdev_init(&uverbs_dev->dev, &uverbs_fops); + uverbs_dev->dev.owner = THIS_MODULE; + kobject_set_name(&uverbs_dev->dev.kobj, "uverbs%d", uverbs_dev->devnum); + if (cdev_add(&uverbs_dev->dev, IB_UVERBS_BASE_DEV + uverbs_dev->devnum, 1)) + goto err; + + uverbs_dev->class_dev.class = &uverbs_class; + uverbs_dev->class_dev.dev = device->dma_device; + snprintf(uverbs_dev->class_dev.class_id, BUS_ID_SIZE, "uverbs%d", uverbs_dev->devnum); + if (class_device_register(&uverbs_dev->class_dev)) + goto err_cdev; + + if (class_device_create_file(&uverbs_dev->class_dev, &class_device_attr_dev)) + goto err_class; + if (class_device_create_file(&uverbs_dev->class_dev, &class_device_attr_ibdev)) + goto err_class; + + ib_set_client_data(device, &uverbs_client, uverbs_dev); + + return; + +err_class: + class_device_unregister(&uverbs_dev->class_dev); + +err_cdev: + cdev_del(&uverbs_dev->dev); + clear_bit(uverbs_dev->devnum, dev_map); + +err: + kfree(uverbs_dev); + return; +} + +static void ib_uverbs_remove_one(struct ib_device *device) +{ + struct ib_uverbs_device *uverbs_dev = ib_get_client_data(device, &uverbs_client); + + if (!uverbs_dev) + return; + + class_device_unregister(&uverbs_dev->class_dev); +} + +static struct super_block *uverbs_event_get_sb(struct file_system_type *fs_type, int flags, + const char *dev_name, void *data) +{ + return get_sb_pseudo(fs_type, "infinibandevent:", NULL, + INFINIBANDEVENTFS_MAGIC); +} + +static struct file_system_type uverbs_event_fs = { + /* No owner field so module can be unloaded */ + .name = "infinibandeventfs", + .get_sb = uverbs_event_get_sb, + .kill_sb = kill_litter_super +}; + +static int __init ib_uverbs_init(void) +{ + int ret; + + spin_lock_init(&map_lock); + + ret = register_chrdev_region(IB_UVERBS_BASE_DEV, IB_UVERBS_MAX_DEVICES, + "infiniband_verbs"); + if (ret) { + printk(KERN_ERR "user_verbs: couldn't register device number\n"); + goto out; + } + + ret = class_register(&uverbs_class); + if (ret) { + printk(KERN_ERR "user_verbs: couldn't create class infiniband_verbs\n"); + goto out_chrdev; + } + + ret = class_create_file(&uverbs_class, &class_attr_abi_version); + if (ret) { + printk(KERN_ERR "user_verbs: couldn't create abi_version attribute\n"); + goto out_class; + } + + ret = register_filesystem(&uverbs_event_fs); + if (ret) { + printk(KERN_ERR "user_verbs: couldn't register infinibandeventfs\n"); + goto out_class; + } + + uverbs_event_mnt = kern_mount(&uverbs_event_fs); + if (IS_ERR(uverbs_event_mnt)) { + ret = PTR_ERR(uverbs_event_mnt); + printk(KERN_ERR "user_verbs: couldn't mount infinibandeventfs\n"); + goto out_fs; + } + + ret = ib_register_client(&uverbs_client); + if (ret) { + printk(KERN_ERR "user_verbs: couldn't register client\n"); + goto out_mnt; + } + + return 0; + +out_mnt: + mntput(uverbs_event_mnt); + +out_fs: + unregister_filesystem(&uverbs_event_fs); + +out_class: + class_unregister(&uverbs_class); + +out_chrdev: + unregister_chrdev_region(IB_UVERBS_BASE_DEV, IB_UVERBS_MAX_DEVICES); + +out: + return ret; +} + +static void __exit ib_uverbs_cleanup(void) +{ + ib_unregister_client(&uverbs_client); + mntput(uverbs_event_mnt); + unregister_filesystem(&uverbs_event_fs); + class_unregister(&uverbs_class); + unregister_chrdev_region(IB_UVERBS_BASE_DEV, IB_UVERBS_MAX_DEVICES); +} + +module_init(ib_uverbs_init); +module_exit(ib_uverbs_cleanup); From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 04/16] IB uverbs: add user verbs ABI header In-Reply-To: <2005628163.01zcuUvu5mtvwzkX@cisco.com> Message-ID: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> Add the ib_user_verbs.h header file, which defines the ABI used by InfiniBand userspace verbs for kernel/user communication. Signed-off-by: Roland Dreier --- drivers/infiniband/include/ib_user_verbs.h | 389 +++++++++++++++++++++++++++++ 1 files changed, 389 insertions(+) --- /dev/null 2005-06-23 14:14:38.423479552 -0700 +++ linux/drivers/infiniband/include/ib_user_verbs.h 2005-06-28 15:20:02.710321131 -0700 @@ -0,0 +1,389 @@ +/* + * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id: ib_user_verbs.h 2708 2005-06-24 17:27:21Z roland $ + */ + +#ifndef IB_USER_VERBS_H +#define IB_USER_VERBS_H + +#include + +/* + * Increment this value if any changes that break userspace ABI + * compatibility are made. + */ +#define IB_USER_VERBS_ABI_VERSION 1 + +enum { + IB_USER_VERBS_CMD_QUERY_PARAMS, + IB_USER_VERBS_CMD_GET_CONTEXT, + IB_USER_VERBS_CMD_QUERY_DEVICE, + IB_USER_VERBS_CMD_QUERY_PORT, + IB_USER_VERBS_CMD_QUERY_GID, + IB_USER_VERBS_CMD_QUERY_PKEY, + IB_USER_VERBS_CMD_ALLOC_PD, + IB_USER_VERBS_CMD_DEALLOC_PD, + IB_USER_VERBS_CMD_CREATE_AH, + IB_USER_VERBS_CMD_MODIFY_AH, + IB_USER_VERBS_CMD_QUERY_AH, + IB_USER_VERBS_CMD_DESTROY_AH, + IB_USER_VERBS_CMD_REG_MR, + IB_USER_VERBS_CMD_REG_SMR, + IB_USER_VERBS_CMD_REREG_MR, + IB_USER_VERBS_CMD_QUERY_MR, + IB_USER_VERBS_CMD_DEREG_MR, + IB_USER_VERBS_CMD_ALLOC_MW, + IB_USER_VERBS_CMD_BIND_MW, + IB_USER_VERBS_CMD_DEALLOC_MW, + IB_USER_VERBS_CMD_CREATE_CQ, + IB_USER_VERBS_CMD_RESIZE_CQ, + IB_USER_VERBS_CMD_DESTROY_CQ, + IB_USER_VERBS_CMD_POLL_CQ, + IB_USER_VERBS_CMD_PEEK_CQ, + IB_USER_VERBS_CMD_REQ_NOTIFY_CQ, + IB_USER_VERBS_CMD_CREATE_QP, + IB_USER_VERBS_CMD_QUERY_QP, + IB_USER_VERBS_CMD_MODIFY_QP, + IB_USER_VERBS_CMD_DESTROY_QP, + IB_USER_VERBS_CMD_POST_SEND, + IB_USER_VERBS_CMD_POST_RECV, + IB_USER_VERBS_CMD_ATTACH_MCAST, + IB_USER_VERBS_CMD_DETACH_MCAST +}; + +/* + * Make sure that all structs defined in this file remain laid out so + * that they pack the same way on 32-bit and 64-bit architectures (to + * avoid incompatibility between 32-bit userspace and 64-bit kernels). + * In particular do not use pointer types -- pass pointers in __u64 + * instead. + */ + +struct ib_uverbs_async_event_desc { + __u64 element; + __u32 event_type; /* enum ib_event_type */ + __u32 reserved; +}; + +struct ib_uverbs_comp_event_desc { + __u64 cq_handle; +}; + +/* + * All commands from userspace should start with a __u32 command field + * followed by __u16 in_words and out_words fields (which give the + * length of the command block and response buffer if any in 32-bit + * words). The kernel driver will read these fields first and read + * the rest of the command struct based on these value. + */ + +struct ib_uverbs_cmd_hdr { + __u32 command; + __u16 in_words; + __u16 out_words; +}; + +/* + * No driver_data for "query params" command, since this is intended + * to be a core function with no possible device dependence. + */ +struct ib_uverbs_query_params { + __u64 response; +}; + +struct ib_uverbs_query_params_resp { + __u32 num_cq_events; +}; + +struct ib_uverbs_get_context { + __u64 response; + __u64 cq_fd_tab; + __u64 driver_data[0]; +}; + +struct ib_uverbs_get_context_resp { + __u32 async_fd; + __u32 reserved; +}; + +struct ib_uverbs_query_device { + __u64 response; + __u64 driver_data[0]; +}; + +struct ib_uverbs_query_device_resp { + __u64 fw_ver; + __u64 node_guid; + __u64 sys_image_guid; + __u64 max_mr_size; + __u64 page_size_cap; + __u32 vendor_id; + __u32 vendor_part_id; + __u32 hw_ver; + __u32 max_qp; + __u32 max_qp_wr; + __u32 device_cap_flags; + __u32 max_sge; + __u32 max_sge_rd; + __u32 max_cq; + __u32 max_cqe; + __u32 max_mr; + __u32 max_pd; + __u32 max_qp_rd_atom; + __u32 max_ee_rd_atom; + __u32 max_res_rd_atom; + __u32 max_qp_init_rd_atom; + __u32 max_ee_init_rd_atom; + __u32 atomic_cap; + __u32 max_ee; + __u32 max_rdd; + __u32 max_mw; + __u32 max_raw_ipv6_qp; + __u32 max_raw_ethy_qp; + __u32 max_mcast_grp; + __u32 max_mcast_qp_attach; + __u32 max_total_mcast_qp_attach; + __u32 max_ah; + __u32 max_fmr; + __u32 max_map_per_fmr; + __u32 max_srq; + __u32 max_srq_wr; + __u32 max_srq_sge; + __u16 max_pkeys; + __u8 local_ca_ack_delay; + __u8 phys_port_cnt; + __u8 reserved[4]; +}; + +struct ib_uverbs_query_port { + __u64 response; + __u8 port_num; + __u8 reserved[7]; + __u64 driver_data[0]; +}; + +struct ib_uverbs_query_port_resp { + __u32 port_cap_flags; + __u32 max_msg_sz; + __u32 bad_pkey_cntr; + __u32 qkey_viol_cntr; + __u32 gid_tbl_len; + __u16 pkey_tbl_len; + __u16 lid; + __u16 sm_lid; + __u8 state; + __u8 max_mtu; + __u8 active_mtu; + __u8 lmc; + __u8 max_vl_num; + __u8 sm_sl; + __u8 subnet_timeout; + __u8 init_type_reply; + __u8 active_width; + __u8 active_speed; + __u8 phys_state; + __u8 reserved[3]; +}; + +struct ib_uverbs_query_gid { + __u64 response; + __u8 port_num; + __u8 index; + __u8 reserved[6]; + __u64 driver_data[0]; +}; + +struct ib_uverbs_query_gid_resp { + __u8 gid[16]; +}; + +struct ib_uverbs_query_pkey { + __u64 response; + __u8 port_num; + __u8 index; + __u8 reserved[6]; + __u64 driver_data[0]; +}; + +struct ib_uverbs_query_pkey_resp { + __u16 pkey; + __u16 reserved; +}; + +struct ib_uverbs_alloc_pd { + __u64 response; + __u64 driver_data[0]; +}; + +struct ib_uverbs_alloc_pd_resp { + __u32 pd_handle; +}; + +struct ib_uverbs_dealloc_pd { + __u32 pd_handle; +}; + +struct ib_uverbs_reg_mr { + __u64 response; + __u64 start; + __u64 length; + __u64 hca_va; + __u32 pd_handle; + __u32 access_flags; + __u64 driver_data[0]; +}; + +struct ib_uverbs_reg_mr_resp { + __u32 mr_handle; + __u32 lkey; + __u32 rkey; +}; + +struct ib_uverbs_dereg_mr { + __u32 mr_handle; +}; + +struct ib_uverbs_create_cq { + __u64 response; + __u64 user_handle; + __u32 cqe; + __u32 event_handler; + __u64 driver_data[0]; +}; + +struct ib_uverbs_create_cq_resp { + __u32 cq_handle; + __u32 cqe; +}; + +struct ib_uverbs_destroy_cq { + __u32 cq_handle; +}; + +struct ib_uverbs_create_qp { + __u64 response; + __u64 user_handle; + __u32 pd_handle; + __u32 send_cq_handle; + __u32 recv_cq_handle; + __u32 srq_handle; + __u32 max_send_wr; + __u32 max_recv_wr; + __u32 max_send_sge; + __u32 max_recv_sge; + __u32 max_inline_data; + __u8 sq_sig_all; + __u8 qp_type; + __u8 is_srq; + __u8 reserved; + __u64 driver_data[0]; +}; + +struct ib_uverbs_create_qp_resp { + __u32 qp_handle; + __u32 qpn; +}; + +/* + * This struct needs to remain a multiple of 8 bytes to keep the + * alignment of the modify QP parameters. + */ +struct ib_uverbs_qp_dest { + __u8 dgid[16]; + __u32 flow_label; + __u16 dlid; + __u16 reserved; + __u8 sgid_index; + __u8 hop_limit; + __u8 traffic_class; + __u8 sl; + __u8 src_path_bits; + __u8 static_rate; + __u8 is_global; + __u8 port_num; +}; + +struct ib_uverbs_modify_qp { + struct ib_uverbs_qp_dest dest; + struct ib_uverbs_qp_dest alt_dest; + __u32 qp_handle; + __u32 attr_mask; + __u32 qkey; + __u32 rq_psn; + __u32 sq_psn; + __u32 dest_qp_num; + __u32 qp_access_flags; + __u16 pkey_index; + __u16 alt_pkey_index; + __u8 qp_state; + __u8 cur_qp_state; + __u8 path_mtu; + __u8 path_mig_state; + __u8 en_sqd_async_notify; + __u8 max_rd_atomic; + __u8 max_dest_rd_atomic; + __u8 min_rnr_timer; + __u8 port_num; + __u8 timeout; + __u8 retry_cnt; + __u8 rnr_retry; + __u8 alt_port_num; + __u8 alt_timeout; + __u8 reserved[2]; + __u64 driver_data[0]; +}; + +struct ib_uverbs_modify_qp_resp { +}; + +struct ib_uverbs_destroy_qp { + __u32 qp_handle; +}; + +struct ib_uverbs_attach_mcast { + __u8 gid[16]; + __u32 qp_handle; + __u16 mlid; + __u16 reserved; + __u64 driver_data[0]; +}; + +struct ib_uverbs_detach_mcast { + __u8 gid[16]; + __u32 qp_handle; + __u16 mlid; + __u16 reserved; + __u64 driver_data[0]; +}; + +#endif /* IB_USER_VERBS_H */ From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 06/16] IB uverbs: memory pinning implementation In-Reply-To: <2005628163.jfSiMqRcI78iLMJP@cisco.com> Message-ID: <2005628163.qcqYIUxXOrm3IH43@cisco.com> Add support for pinning userspace memory regions and returning a list of pages in the region. This includes tracking pinned memory against vm_locked and preventing unprivileged users from exceeding RLIMIT_MEMLOCK. Signed-off-by: Roland Dreier --- drivers/infiniband/core/uverbs_mem.c | 221 +++++++++++++++++++++++++++++++++++ 1 files changed, 221 insertions(+) --- /dev/null 2005-06-23 14:14:38.423479552 -0700 +++ linux/drivers/infiniband/core/uverbs_mem.c 2005-06-28 15:20:06.718455487 -0700 @@ -0,0 +1,221 @@ +/* + * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Cisco Systems. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id: uverbs_mem.c 2743 2005-06-28 22:27:59Z roland $ + */ + +#include +#include + +#include "uverbs.h" + +struct ib_umem_account_work { + struct work_struct work; + struct mm_struct *mm; + unsigned long diff; +}; + + +static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty) +{ + struct ib_umem_chunk *chunk, *tmp; + int i; + + list_for_each_entry_safe(chunk, tmp, &umem->chunk_list, list) { + dma_unmap_sg(dev->dma_device, chunk->page_list, + chunk->nents, DMA_BIDIRECTIONAL); + for (i = 0; i < chunk->nents; ++i) { + if (umem->writable && dirty) + set_page_dirty_lock(chunk->page_list[i].page); + put_page(chunk->page_list[i].page); + } + + kfree(chunk); + } +} + +int ib_umem_get(struct ib_device *dev, struct ib_umem *mem, + void *addr, size_t size, int write) +{ + struct page **page_list; + struct ib_umem_chunk *chunk; + unsigned long locked; + unsigned long lock_limit; + unsigned long cur_base; + unsigned long npages; + int ret = 0; + int off; + int i; + + if (!can_do_mlock()) + return -EPERM; + + page_list = (struct page **) __get_free_page(GFP_KERNEL); + if (!page_list) + return -ENOMEM; + + mem->user_base = (unsigned long) addr; + mem->length = size; + mem->offset = (unsigned long) addr & ~PAGE_MASK; + mem->page_size = PAGE_SIZE; + mem->writable = write; + + INIT_LIST_HEAD(&mem->chunk_list); + + npages = PAGE_ALIGN(size + mem->offset) >> PAGE_SHIFT; + + down_write(¤t->mm->mmap_sem); + + locked = npages + current->mm->locked_vm; + lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur >> PAGE_SHIFT; + + if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) { + ret = -ENOMEM; + goto out; + } + + cur_base = (unsigned long) addr & PAGE_MASK; + + while (npages) { + ret = get_user_pages(current, current->mm, cur_base, + min_t(int, npages, + PAGE_SIZE / sizeof (struct page *)), + 1, !write, page_list, NULL); + + if (ret < 0) + goto out; + + cur_base += ret * PAGE_SIZE; + npages -= ret; + + off = 0; + + while (ret) { + chunk = kmalloc(sizeof *chunk + sizeof (struct scatterlist) * + min_t(int, ret, IB_UMEM_MAX_PAGE_CHUNK), + GFP_KERNEL); + if (!chunk) { + ret = -ENOMEM; + goto out; + } + + chunk->nents = min_t(int, ret, IB_UMEM_MAX_PAGE_CHUNK); + for (i = 0; i < chunk->nents; ++i) { + chunk->page_list[i].page = page_list[i + off]; + chunk->page_list[i].offset = 0; + chunk->page_list[i].length = PAGE_SIZE; + } + + chunk->nmap = dma_map_sg(dev->dma_device, + &chunk->page_list[0], + chunk->nents, + DMA_BIDIRECTIONAL); + if (chunk->nmap <= 0) { + for (i = 0; i < chunk->nents; ++i) + put_page(chunk->page_list[i].page); + kfree(chunk); + + ret = -ENOMEM; + goto out; + } + + ret -= chunk->nents; + off += chunk->nents; + list_add_tail(&chunk->list, &mem->chunk_list); + } + + ret = 0; + } + +out: + if (ret < 0) + __ib_umem_release(dev, mem, 0); + else + current->mm->locked_vm = locked; + + up_write(¤t->mm->mmap_sem); + free_page((unsigned long) page_list); + + return ret; +} + +void ib_umem_release(struct ib_device *dev, struct ib_umem *umem) +{ + __ib_umem_release(dev, umem, 1); + + down_write(¤t->mm->mmap_sem); + current->mm->locked_vm -= + PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; + up_write(¤t->mm->mmap_sem); +} + +static void ib_umem_account(void *work_ptr) +{ + struct ib_umem_account_work *work = work_ptr; + + down_write(&work->mm->mmap_sem); + work->mm->locked_vm -= work->diff; + up_write(&work->mm->mmap_sem); + mmput(work->mm); + kfree(work); +} + +void ib_umem_release_on_close(struct ib_device *dev, struct ib_umem *umem) +{ + struct ib_umem_account_work *work; + struct mm_struct *mm; + + __ib_umem_release(dev, umem, 1); + + mm = get_task_mm(current); + if (!mm) + return; + + /* + * We may be called with the mm's mmap_sem already held. This + * can happen when a userspace munmap() is the call that drops + * the last reference to our file and calls our release + * method. If there are memory regions to destroy, we'll end + * up here and not be able to take the mmap_sem. Therefore we + * defer the vm_locked accounting to the system workqueue. + */ + + work = kmalloc(sizeof *work, GFP_KERNEL); + if (!work) + return; + + INIT_WORK(&work->work, ib_umem_account, work); + work->mm = mm; + work->diff = PAGE_ALIGN(umem->length + umem->offset) >> PAGE_SHIFT; + + schedule_work(&work->work); +} From rolandd at cisco.com Tue Jun 28 16:03:44 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:44 -0700 Subject: [openib-general] [PATCH 16/16] IB uverbs: add documentation file In-Reply-To: <2005628163.VUsohVlfE72duiiM@cisco.com> Message-ID: <2005628163.mVVkhZjoFQLjYM7E@cisco.com> Add documentation for InfiniBand userspace verbs. Signed-off-by: Roland Dreier --- Documentation/infiniband/user_verbs.txt | 69 ++++++++++++++++++++++++++++++++ 1 files changed, 69 insertions(+) --- /dev/null 2005-06-23 14:14:38.423479552 -0700 +++ linux/Documentation/infiniband/user_verbs.txt 2005-06-28 15:20:31.245140214 -0700 @@ -0,0 +1,69 @@ +USERSPACE VERBS ACCESS + + The ib_uverbs module, built by enabling CONFIG_INFINIBAND_USER_VERBS, + enables direct userspace access to IB hardware via "verbs," as + described in chapter 11 of the InfiniBand Architecture Specification. + + To use the verbs, the libibverbs library, available from + , is required. libibverbs contains a + device-independent API for using the ib_uverbs interface. + libibverbs also requires appropriate device-dependent kernel and + userspace driver for your InfiniBand hardware. For example, to use + a Mellanox HCA, you will need the ib_mthca kernel module and the + libmthca userspace driver be installed. + +User-kernel communication + + Userspace communicates with the kernel for slow path, resource + management operations via the /dev/infiniband/uverbsN character + devices. Fast path operations are typically performed by writing + directly to hardware registers mmap()ed into userspace, with no + system call or context switch into the kernel. + + Commands are sent to the kernel via write()s on these device files. + The ABI is defined in drivers/infiniband/include/ib_user_verbs.h. + The structs for commands that require a response from the kernel + contain a 64-bit field used to pass a pointer to an output buffer. + Status is returned to userspace as the return value of the write() + system call. + +Resource management + + Since creation and destruction of all IB resources is done by + commands passed through a file descriptor, the kernel can keep track + of which resources are attached to a given userspace context. The + ib_uverbs module maintains idr tables that are used to translate + between kernel pointers and opaque userspace handles, so that kernel + pointers are never exposed to userspace and userspace cannot trick + the kernel into following a bogus pointer. + + This also allows the kernel to clean up when a process exits and + prevent one process from touching another process's resources. + +Memory pinning + + Direct userspace I/O requires that memory regions that are potential + I/O targets be kept resident at the same physical address. The + ib_uverbs module manages pinning and unpinning memory regions via + get_user_pages() and put_page() calls. It also accounts for the + amount of memory pinned in the process's locked_vm, and checks that + unprivileged processes do not exceed their RLIMIT_MEMLOCK limit. + + Pages that are pinned multiple times are counted each time they are + pinned, so the value of locked_vm may be an overestimate of the + number of pages pinned by a process. + +/dev files + + To create the appropriate character device files automatically with + udev, a rule like + + KERNEL="uverbs*", NAME="infiniband/%k" + + can be used. This will create device nodes named + + /dev/infiniband/uverbs0 + + and so on. Since the InfiniBand userspace verbs should be safe for + use by non-privileged processes, it may be useful to add an + appropriate MODE or GROUP to the udev rule. From rolandd at cisco.com Tue Jun 28 16:03:43 2005 From: rolandd at cisco.com (Roland Dreier) Date: Tue, 28 Jun 2005 16:03:43 -0700 Subject: [openib-general] [PATCH 07/16] IB uverbs: hook up Kconfig/Makefile In-Reply-To: <2005628163.qcqYIUxXOrm3IH43@cisco.com> Message-ID: <2005628163.3Q9FD2m0Zd9kmlVn@cisco.com> Hook up InfiniBand userspace verbs to Kconfig and the make system. Signed-off-by: Roland Dreier --- drivers/infiniband/Kconfig | 10 ++++++++++ drivers/infiniband/core/Makefile | 5 ++++- 2 files changed, 14 insertions(+), 1 deletion(-) --- linux.orig/drivers/infiniband/Kconfig 2005-06-28 15:19:24.783512128 -0700 +++ linux/drivers/infiniband/Kconfig 2005-06-28 15:20:09.143931652 -0700 @@ -7,6 +7,16 @@ config INFINIBAND any protocols you wish to use as well as drivers for your InfiniBand hardware. +config INFINIBAND_USER_VERBS + tristate "InfiniBand userspace verbs support" + depends on INFINIBAND + ---help--- + Userspace InfiniBand verbs support. This is the kernel side + of userspace verbs, which allows userspace processes to + directly access InfiniBand hardware for fast-path + operations. You will also need libibverbs and a hardware + driver library from . + source "drivers/infiniband/hw/mthca/Kconfig" source "drivers/infiniband/ulp/ipoib/Kconfig" --- linux.orig/drivers/infiniband/core/Makefile 2005-06-28 15:19:24.782512344 -0700 +++ linux/drivers/infiniband/core/Makefile 2005-06-28 15:20:09.143931652 -0700 @@ -1,6 +1,7 @@ EXTRA_CFLAGS += -Idrivers/infiniband/include -obj-$(CONFIG_INFINIBAND) += ib_core.o ib_mad.o ib_sa.o ib_umad.o +obj-$(CONFIG_INFINIBAND) += ib_core.o ib_mad.o ib_sa.o ib_umad.o +obj-$(CONFIG_INFINIBAND_USER_VERBS) += ib_uverbs.o ib_core-y := packer.o ud_header.o verbs.o sysfs.o \ device.o fmr_pool.o cache.o @@ -10,3 +11,5 @@ ib_mad-y := mad.o smi.o agent.o ib_sa-y := sa_query.o ib_umad-y := user_mad.o + +ib_uverbs-y := uverbs_main.o uverbs_cmd.o uverbs_mem.o From akpm at osdl.org Tue Jun 28 17:02:12 2005 From: akpm at osdl.org (Andrew Morton) Date: Tue, 28 Jun 2005 17:02:12 -0700 Subject: [openib-general] Re: [PATCH 06/16] IB uverbs: memory pinning implementation In-Reply-To: <2005628163.qcqYIUxXOrm3IH43@cisco.com> References: <2005628163.jfSiMqRcI78iLMJP@cisco.com> <2005628163.qcqYIUxXOrm3IH43@cisco.com> Message-ID: <20050628170212.24623191.akpm@osdl.org> Roland Dreier wrote: > > Add support for pinning userspace memory regions and returning a list > of pages in the region. This includes tracking pinned memory against > vm_locked and preventing unprivileged users from exceeding RLIMIT_MEMLOCK. > Can you tell us a bit more about the design ideas here? What's it doing, how and why? We should look at these things and also decide whether some of this should live in mm/*. > +int ib_umem_get(struct ib_device *dev, struct ib_umem *mem, > + void *addr, size_t size, int write) > +{ > ... > + if (!can_do_mlock()) > + return -EPERM; > + > ... > + if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) { The capable() test is redundant. From akpm at osdl.org Tue Jun 28 17:05:53 2005 From: akpm at osdl.org (Andrew Morton) Date: Tue, 28 Jun 2005 17:05:53 -0700 Subject: [openib-general] Re: [PATCH 11/16] IB uverbs: add mthca mmap support In-Reply-To: <2005628163.gtJFW6uLUrGQteys@cisco.com> References: <2005628163.o84QGfsM7oMSy0oU@cisco.com> <2005628163.gtJFW6uLUrGQteys@cisco.com> Message-ID: <20050628170553.00a14a29.akpm@osdl.org> Roland Dreier wrote: > > Add support for mmap() method to mthca, so that userspace can get > access to doorbell registers. This allows userspace to get direct > access to the HCA for data path operations. > > Each userspace context gets its own copy of the doorbell registers and > is only allowed to use resources that the kernel has given it access > to. In other words, this is safe. > > ... > > +static int mthca_mmap_uar(struct ib_ucontext *context, > + struct vm_area_struct *vma) > +{ > + if (vma->vm_end - vma->vm_start != PAGE_SIZE) > + return -EINVAL; > + > + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > + vma->vm_flags |= VM_DONTCOPY; > + > + if (remap_pfn_range(vma, vma->vm_start, > + to_mucontext(context)->uar.pfn, > + PAGE_SIZE, vma->vm_page_prot)) > + return -EAGAIN; > + > + return 0; > +} What's the thinking behind the VM_DONTCOPY there? What's actually being mapped here? Hardware? If so, is VM_IO not needed? From akpm at osdl.org Tue Jun 28 17:07:18 2005 From: akpm at osdl.org (Andrew Morton) Date: Tue, 28 Jun 2005 17:07:18 -0700 Subject: [openib-general] Re: [PATCH 12/16] IB uverbs: add mthca user PD support In-Reply-To: <2005628163.px5sYyzsYWf21dJY@cisco.com> References: <2005628163.gtJFW6uLUrGQteys@cisco.com> <2005628163.px5sYyzsYWf21dJY@cisco.com> Message-ID: <20050628170718.0b2a9cad.akpm@osdl.org> Roland Dreier wrote: > > Add support for userspace protection domains (PDs) to mthca. What is a userspace protection domain? From akpm at osdl.org Tue Jun 28 17:10:46 2005 From: akpm at osdl.org (Andrew Morton) Date: Tue, 28 Jun 2005 17:10:46 -0700 Subject: [openib-general] Re: [PATCH 14/16] IB uverbs: add mthca user CQ support In-Reply-To: <2005628163.jWivrrVepuPgy40z@cisco.com> References: <2005628163.XfYnZThlsTGb61Td@cisco.com> <2005628163.jWivrrVepuPgy40z@cisco.com> Message-ID: <20050628171046.6fa7de0d.akpm@osdl.org> Roland Dreier wrote: > > Add support for userspace completion queues (CQs) to mthca. > There are more interesting things happening here ;) > @@ -177,6 +177,7 @@ struct mthca_cq { > int cqn; > u32 cons_index; > int is_direct; > + int is_kernel; > > /* Next fields are Arbel only */ > int set_ci_db_index; I assume we have one body of code which is capable of handling data structures in either kenrel memory of user memory? (guess). If so, that's a fairly sensitive thing to be doing. Tell us more, please. From greg at kroah.com Tue Jun 28 17:27:09 2005 From: greg at kroah.com (Greg KH) Date: Tue, 28 Jun 2005 17:27:09 -0700 Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <2005628163.jfSiMqRcI78iLMJP@cisco.com> References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> Message-ID: <20050629002709.GB17805@kroah.com> On Tue, Jun 28, 2005 at 04:03:43PM -0700, Roland Dreier wrote: > +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-28 15:20:04.363963991 -0700 > @@ -0,0 +1,708 @@ > +/* > + * Copyright (c) 2005 Topspin Communications. All rights reserved. > + * Copyright (c) 2005 Cisco Systems. All rights reserved. > + * > + * This software is available to you under a choice of one of two > + * licenses. You may choose to be licensed under the terms of the GNU > + * General Public License (GPL) Version 2, available from the file > + * COPYING in the main directory of this source tree, or the > + * OpenIB.org BSD license below: Ok, I've complained about this before, but due to the fact that you are calling EXPORT_SYMBOL_GPL() only functions in this code, the ability for it for someone to use the BSD license on it in the future, is pretty much impossible, right? Wasn't the openib group going to drop this horrible license, or are they still insisting on porting this to other operating systems? > +static ssize_t show_dev(struct class_device *class_dev, char *buf) > +{ > + struct ib_uverbs_device *dev = > + container_of(class_dev, struct ib_uverbs_device, class_dev); > + > + return print_dev_t(buf, dev->dev.dev); > +} > +static CLASS_DEVICE_ATTR(dev, S_IRUGO, show_dev, NULL); This is no longer needed with the class device interface in the kernel today. Please use the new api (basically just set dev_t in the class_device, and you get this for free.) thanks, greg k-h From tduffy at sun.com Tue Jun 28 18:16:49 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 28 Jun 2005 18:16:49 -0700 Subject: [openib-general] [PATCH][RFC] nfsordma: initial port of nfsrdma to 2.6 and james'sss kdapl In-Reply-To: <6.2.1.2.2.20050628111852.0685fc60@exnane01.nane.netapp.com> References: <1119911140.25388.6.camel@duffman> <6.2.1.2.2.20050628094824.04e0dc10@exnane01.nane.netapp.com> <1119971635.31370.3.camel@duffman> <6.2.1.2.2.20050628111852.0685fc60@exnane01.nane.netapp.com> Message-ID: <1120007809.31370.18.camel@duffman> On Tue, 2005-06-28 at 11:28 -0400, Talpey, Thomas wrote: > At 11:13 AM 6/28/2005, Tom Duffy wrote: > >I am sure I got the RPC stuff wrong. I just wanted to make it compile > >against James's kDAPL and inside the drivers/infiniband directory. > > > >Where can I find the 2.6.11 version of the patch? > > Here: Thanks. > This is kind of involved, you need to apply some prepatches as well > as postpatches. The 2.6.12 is significantly cleaner but it implements > a new API, due to Trond's requests. In other words, moving target > alert. > > I have a working tarball against 2.6.11. The two files rdma_transport.c > and rdma_marshal.c plus what you're doing in rdma_kdapl.c might just > work. But it's not GPL yet. Can you wait a couple of days? Of course. Just wanted to jump start the work and integration effort. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Tue Jun 28 18:38:15 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 28 Jun 2005 18:38:15 -0700 Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <20050629002709.GB17805@kroah.com> References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> <20050629002709.GB17805@kroah.com> Message-ID: <1120009095.31370.30.camel@duffman> On Tue, 2005-06-28 at 17:27 -0700, Greg KH wrote: > Ok, I've complained about this before, but due to the fact that you are > calling EXPORT_SYMBOL_GPL() only functions in this code, the ability for > it for someone to use the BSD license on it in the future, is pretty > much impossible, right? No, only to call these functions from BSD-only (or other licensed) modules. > Wasn't the openib group going to drop this horrible license, or are they > still insisting on porting this to other operating systems? I don't think we need to drop this license. What is the harm? At some point, Sun may want OpenSolaris to use OpenIB. Or what if the Darwin folks decide to create a port? Don't worry: the OpenIB Windows work is done in a completely different repository with a completely different code base because Microsoft was scared of code that ever *was* GPL, even if a BSD-only fork was created. The bylaws of OpenIB.org require that all code hosted and developed under our auspices be (at least) BSD. I don't want it to happen, but if the code in Linux chooses one license (GPL) and not both, then we won't be able to accept patches back that come in through the mainline kernel. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From hozer at hozed.org Tue Jun 28 21:13:22 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Tue, 28 Jun 2005 23:13:22 -0500 Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <20050629002709.GB17805@kroah.com> References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> <20050629002709.GB17805@kroah.com> Message-ID: <20050629041321.GM4907@kalmia.hozed.org> On Tue, Jun 28, 2005 at 05:27:09PM -0700, Greg KH wrote: > On Tue, Jun 28, 2005 at 04:03:43PM -0700, Roland Dreier wrote: > > +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-28 15:20:04.363963991 -0700 > > @@ -0,0 +1,708 @@ > > +/* > > + * Copyright (c) 2005 Topspin Communications. All rights reserved. > > + * Copyright (c) 2005 Cisco Systems. All rights reserved. > > + * > > + * This software is available to you under a choice of one of two > > + * licenses. You may choose to be licensed under the terms of the GNU > > + * General Public License (GPL) Version 2, available from the file > > + * COPYING in the main directory of this source tree, or the > > + * OpenIB.org BSD license below: > > Ok, I've complained about this before, but due to the fact that you are > calling EXPORT_SYMBOL_GPL() only functions in this code, the ability for > it for someone to use the BSD license on it in the future, is pretty > much impossible, right? Only if someone tries to use it under a BSD license, strips off the GPL notices, and then builds it against *Linux*. If linux-kernel is going to be that fascist about licensing, let's please clean up all the binary firmware blobs in header files first. It seems reasonable to me that distribution and modification of the *source code* can be under either license. But as soon as you build a binary agaist the linux kernel, the binary is irrevocably GPL licensed. -------------------------------------------------------------------------- Troy Benjegerdes 'da hozer' hozer at hozed.org Somone asked my why I work on this free (http://www.fsf.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer: "Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life." -- Charles Shultz From halr at voltaire.com Wed Jun 29 04:09:13 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 29 Jun 2005 07:09:13 -0400 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: References: <20050628192649.GA9340@lst.de> <469958e00506281506350b5a88@mail.gmail.com> Message-ID: <1120043352.4848.4556.camel@hal.voltaire.com> On Tue, 2005-06-28 at 18:10, James Lentini wrote: > Are there any tools available for a network administrator to assign a > GID? Does OpenSM provider this capability? As far as OpenIB OpenSM, there is currrently no capability to set SM GUIDs. (Note that the SM assigned GID is comprised of the SM supplied subnet prefix in PortInfo and these additional GUIDs). -- Hal From Arkady.Kanevsky at netapp.com Wed Jun 29 07:43:39 2005 From: Arkady.Kanevsky at netapp.com (Kanevsky, Arkady) Date: Wed, 29 Jun 2005 10:43:39 -0400 Subject: [openib-general] IP addressing on InfiniBand networks Message-ID: Most of the existing apps kernel and user space are based on socket addressing nad naming convention including IP addresses and ports. All RDMA APIs made a decision to only deal with IP address and hide IB/MAC addresses under the covers so Consumer do not have to change their existing IP addresses based setup. We will severelly hamper adoption if we request any changes to application addressing scheme. We should strive to the requirement that no ULP changes are needed in order to use RDMA. James were nicely summirizes the technical requirements derived from this. Second, we have had 2 DAT plugfests were most of IB and iWARP providers demonstrated interoperability including the IP addressing one and satisfying all requirements that James stated. For IB it was done through ATS. Before we adopt another solution we must ensure that it interoperates. Third, kDAPL and uDAPL are APIs for RDMA transports in general not just IB. Obviously the problem of determining remote IP address and port are IB specific. While currently people contemplating on using DAT Registry as iWARP insertion point sooner or later the insertion point will be pushed down the stack. Since this is IB CM specific issue kDAPL and uDAPL implementation should have the proper design on how to switch the code path based on underlying RDMA network type to support this feature. Arkady Arkady Kanevsky email: arkady at netapp.com Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Wednesday, June 29, 2005 7:09 AM > To: Lentini, James > Cc: Caitlin Bestler; Christoph Hellwig; openib-general > Subject: Re: [openib-general] IP addressing on InfiniBand networks > > > On Tue, 2005-06-28 at 18:10, James Lentini wrote: > > Are there any tools available for a network administrator > to assign a > > GID? Does OpenSM provider this capability? > > As far as OpenIB OpenSM, there is currrently no capability to > set SM GUIDs. (Note that the SM assigned GID is comprised of > the SM supplied subnet prefix in PortInfo and these additional GUIDs). > > -- Hal > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From caitlin.bestler at gmail.com Wed Jun 29 08:55:08 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 29 Jun 2005 08:55:08 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: References: Message-ID: <469958e005062908555460deae@mail.gmail.com> On 6/29/05, Kanevsky, Arkady wrote: > Most of the existing apps kernel and user space are based on socket > addressing nad naming convention including IP addresses and ports. > All RDMA APIs made a decision to only deal with IP address and hide > IB/MAC > addresses under the covers so Consumer do not have to change their > existing IP addresses based setup. > We will severelly hamper adoption if we request any changes to > application > addressing scheme. > We should strive to the requirement that no ULP changes are needed > in order to use RDMA. I'm in full agreement, but want to add one clarification. The DAT APIs do not require that the user work with an IP Address. They work with an IA Address that has the same format as an IPv6 address and meets all of the semantics of an IPv6 address. But we never *required* it to be an IPv6 address. The actual distinction has more to do with IANA that anything visible to an application. There is one concrete requirement that the IA Address be unique from the perspective on the host. That is, it cannot be a link dependent address. The consensus was that link dependent addresses were something foreign to current network programming and not something that an application should have to deal with. Link-local addresses are intended for bootstrapping, not applications. An assigned GID meets all of the requirements for an IA Address. I think taking advantage of that existing capability is just one of many options that can be done by the IB CM rather than forcing IB specific changes up to the application layer. From rolandd at cisco.com Wed Jun 29 09:00:37 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 09:00:37 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <469958e005062908555460deae@mail.gmail.com> (Caitlin Bestler's message of "Wed, 29 Jun 2005 08:55:08 -0700") References: <469958e005062908555460deae@mail.gmail.com> Message-ID: <5264vxi256.fsf@topspin.com> Caitlin> An assigned GID meets all of the requirements for an IA Caitlin> Address. I think taking advantage of that existing Caitlin> capability is just one of many options that can be done Caitlin> by the IB CM rather than forcing IB specific changes up Caitlin> to the application layer. Just to be clear, the IBA spec is very clear that a GID _is_ an IPv6 address. - R. From rolandd at cisco.com Wed Jun 29 09:06:15 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 09:06:15 -0700 Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> <20050629002709.GB17805@kroah.com> Message-ID: <52acl9gnbc.fsf@topspin.com> Greg> This is no longer needed with the class device interface in Greg> the kernel today. Please use the new api (basically just Greg> set dev_t in the class_device, and you get this for free.) Thanks, I've killed that code and just set class_dev.devt instead. - R. From rolandd at cisco.com Wed Jun 29 09:06:18 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 09:06:18 -0700 Subject: [openib-general] Re: [PATCH 06/16] IB uverbs: memory pinning implementation References: <2005628163.jfSiMqRcI78iLMJP@cisco.com> <2005628163.qcqYIUxXOrm3IH43@cisco.com> <20050628170212.24623191.akpm@osdl.org> Message-ID: <524qbhgnb9.fsf@topspin.com> Roland> Add support for pinning userspace memory regions and Roland> returning a list of pages in the region. This includes Roland> tracking pinned memory against vm_locked and preventing Roland> unprivileged users from exceeding RLIMIT_MEMLOCK. Andrew> Can you tell us a bit more about the design ideas here? Andrew> What's it doing, how and why? The idea is that allowing userspace to handle initiating IO directly requires the kernel to make sure the memory targeted by that IO is kept pinned. This is done by requiring userspace to register the memory regions it will use for IO in advance. The code in uverbs_mem.c helps handle this by providing a function ib_umem_get(), which wraps up calling get_user_pages() and dma_map_sg() for a given piece of userspace address space, and returns a data structure with DMA addresses for region. Since userspace can potentially register huge chunks of memory, the code breaks up the calls to get_user_pages() and dma_map_sg() into chunks, and the umem data structure has a linked list of these chunks. Andrew> We should look at these things and also decide whether Andrew> some of this should live in mm/*. I thought about that a little while I was writing the code. The only thing that seemed generic enough was the logic for vm_locked accounting and checking against RLIMIT_MEMLOCK. I wasn't smart enough to come up with a way to encapsulate it that seemed any easier to read or maintain than just spelling the logic out. iWARP (basically RDMA over TCP) will also want to use the memory pinning code here, but I think the best plan for handling iWARP is to evolve drivers/infiniband into a more generic drivers/rdma -- in which case, this code is fine where it is. So... no objection to making it generic or putting it somewhere else, but there's not anything deep going on here. - R. From rolandd at cisco.com Wed Jun 29 09:06:20 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 09:06:20 -0700 Subject: [openib-general] Re: [PATCH 11/16] IB uverbs: add mthca mmap support References: <2005628163.o84QGfsM7oMSy0oU@cisco.com> <2005628163.gtJFW6uLUrGQteys@cisco.com> <20050628170553.00a14a29.akpm@osdl.org> Message-ID: <52y88tf8qr.fsf@topspin.com> Andrew> What's the thinking behind the VM_DONTCOPY there? I think that was my paranoia about something like a process doing a fork, the original process exiting, and the new process having page still mapped even though the file has been released. This is bad because then we could map the same page to a different process and have them collide. But it seems that there will always be a reference to the underlying struct file as long as someone has this mapping, so I don't really need to worry about this and the VM_DONTCOPY can go. Andrew> What's actually being mapped here? Hardware? If so, is Andrew> VM_IO not needed? Yes, this is a page from a PCI BAR. However, we use remap_pfn_range() to map the page, which sets VM_IO already. - R. From rolandd at cisco.com Wed Jun 29 09:06:22 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 09:06:22 -0700 Subject: [openib-general] Re: [PATCH 12/16] IB uverbs: add mthca user PD support References: <2005628163.gtJFW6uLUrGQteys@cisco.com> <2005628163.px5sYyzsYWf21dJY@cisco.com> <20050628170718.0b2a9cad.akpm@osdl.org> Message-ID: <52slz1f8qp.fsf@topspin.com> Andrew> What is a userspace protection domain? A protection domain is an abstraction enforced by IB hardware -- loosely put, every resource (work queue, memory region, etc) in put in a PD when it is created, and different resources can only see each other if they belong to the same PD. As an example, PDs are needed because IB allows unprivileged processes to directly post requests to work queues. Work requests refer to memory regions by memory keys (32 bit cookies). Without PDs, a process could get access to another process's memory region if it could guess the 32-bit key -- with PDs, it can't because the other process's memory region will be in a different PD from its work queue. - R. From rolandd at cisco.com Wed Jun 29 09:06:25 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 09:06:25 -0700 Subject: [openib-general] Re: [PATCH 14/16] IB uverbs: add mthca user CQ support References: <2005628163.XfYnZThlsTGb61Td@cisco.com> <2005628163.jWivrrVepuPgy40z@cisco.com> <20050628171046.6fa7de0d.akpm@osdl.org> Message-ID: <52mzp9f8qm.fsf@topspin.com> >> + int is_kernel; Andrew> I assume we have one body of code which is capable of Andrew> handling data structures in either kenrel memory of user Andrew> memory? (guess). Andrew> If so, that's a fairly sensitive thing to be doing. Tell Andrew> us more, please. It's actually not that bad. A completion queue (CQ) is a basically a chunk of memory where completion information is written when a work request completes. The hardware can handle many CQs (64K is not an unreasonable number), and we always do things like allocation of CQ numbers, programming HW for CQ context, etc. in the kernel. Both the kernel and userspace can do data path operations like looking for a new CQ entry. This means that for userspace CQs, the actual memory where entries are written should be in userspace. However the struct mthca_cq will always be in the kernel. If you look at how the is_kernel flag is used, you can see that all it does is control whether we allocate/free the actual buffer and a few other things in the kernel, or just use the stuff that userspace has already allocated. - R. From halr at voltaire.com Wed Jun 29 09:02:38 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 29 Jun 2005 12:02:38 -0400 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <5264vxi256.fsf@topspin.com> References: <469958e005062908555460deae@mail.gmail.com> <5264vxi256.fsf@topspin.com> Message-ID: <1120060958.4848.4749.camel@hal.voltaire.com> On Wed, 2005-06-29 at 12:00, Roland Dreier wrote: > Just to be clear, the IBA spec is very clear that a GID _is_ an IPv6 address. albeit with additional properties/restrictions on IBA which do not apply to IPv6 (IBA 1,2 p, 143 lines 11-16). -- Hal From greg at kroah.com Wed Jun 29 09:12:09 2005 From: greg at kroah.com (Greg KH) Date: Wed, 29 Jun 2005 09:12:09 -0700 Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <20050629041321.GM4907@kalmia.hozed.org> References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> <20050629002709.GB17805@kroah.com> <20050629041321.GM4907@kalmia.hozed.org> Message-ID: <20050629161209.GA23781@kroah.com> On Tue, Jun 28, 2005 at 11:13:22PM -0500, Troy Benjegerdes wrote: > On Tue, Jun 28, 2005 at 05:27:09PM -0700, Greg KH wrote: > > On Tue, Jun 28, 2005 at 04:03:43PM -0700, Roland Dreier wrote: > > > +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-28 15:20:04.363963991 -0700 > > > @@ -0,0 +1,708 @@ > > > +/* > > > + * Copyright (c) 2005 Topspin Communications. All rights reserved. > > > + * Copyright (c) 2005 Cisco Systems. All rights reserved. > > > + * > > > + * This software is available to you under a choice of one of two > > > + * licenses. You may choose to be licensed under the terms of the GNU > > > + * General Public License (GPL) Version 2, available from the file > > > + * COPYING in the main directory of this source tree, or the > > > + * OpenIB.org BSD license below: > > > > Ok, I've complained about this before, but due to the fact that you are > > calling EXPORT_SYMBOL_GPL() only functions in this code, the ability for > > it for someone to use the BSD license on it in the future, is pretty > > much impossible, right? > > Only if someone tries to use it under a BSD license, strips off the GPL > notices, and then builds it against *Linux*. Exactly, that's my point. It's pretty useless, and if you are going to build this code for another OS, well, that's going to be a tough job :) > If linux-kernel is going to be that fascist about licensing, let's > please clean up all the binary firmware blobs in header files first. I'm not being "fascist", I'm just saying it's pretty pointless to try to dual license this code, that's all. thanks, greg k-h From rolandd at cisco.com Wed Jun 29 09:15:50 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 09:15:50 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <1120060958.4848.4749.camel@hal.voltaire.com> (Hal Rosenstock's message of "29 Jun 2005 12:02:38 -0400") References: <469958e005062908555460deae@mail.gmail.com> <5264vxi256.fsf@topspin.com> <1120060958.4848.4749.camel@hal.voltaire.com> Message-ID: <52ekalf8ax.fsf@topspin.com> Roland> Just to be clear, the IBA spec is very clear that a GID Roland> _is_ an IPv6 address. Hal> albeit with additional properties/restrictions on IBA which Hal> do not apply to IPv6 (IBA 1,2 p, 143 lines 11-16). Right: all GIDs are IPv6 addresses. However, as you point out, the converse is not true: all IPv6 addresses are not necessarily GIDs. - R. From hozer at hozed.org Wed Jun 29 09:32:25 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Wed, 29 Jun 2005 11:32:25 -0500 Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <20050629161209.GA23781@kroah.com> References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> <20050629002709.GB17805@kroah.com> <20050629041321.GM4907@kalmia.hozed.org> <20050629161209.GA23781@kroah.com> Message-ID: <20050629163225.GP4907@kalmia.hozed.org> On Wed, Jun 29, 2005 at 09:12:09AM -0700, Greg KH wrote: > On Tue, Jun 28, 2005 at 11:13:22PM -0500, Troy Benjegerdes wrote: > > On Tue, Jun 28, 2005 at 05:27:09PM -0700, Greg KH wrote: > > > On Tue, Jun 28, 2005 at 04:03:43PM -0700, Roland Dreier wrote: > > > > +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-28 15:20:04.363963991 -0700 > > > > @@ -0,0 +1,708 @@ > > > > +/* > > > > + * Copyright (c) 2005 Topspin Communications. All rights reserved. > > > > + * Copyright (c) 2005 Cisco Systems. All rights reserved. > > > > + * > > > > + * This software is available to you under a choice of one of two > > > > + * licenses. You may choose to be licensed under the terms of the GNU > > > > + * General Public License (GPL) Version 2, available from the file > > > > + * COPYING in the main directory of this source tree, or the > > > > + * OpenIB.org BSD license below: > > > > > > Ok, I've complained about this before, but due to the fact that you are > > > calling EXPORT_SYMBOL_GPL() only functions in this code, the ability for > > > it for someone to use the BSD license on it in the future, is pretty > > > much impossible, right? > > > > Only if someone tries to use it under a BSD license, strips off the GPL > > notices, and then builds it against *Linux*. > > Exactly, that's my point. It's pretty useless, and if you are going to > build this code for another OS, well, that's going to be a tough job :) > > > If linux-kernel is going to be that fascist about licensing, let's > > please clean up all the binary firmware blobs in header files first. > > I'm not being "fascist", I'm just saying it's pretty pointless to try to > dual license this code, that's all. Ahh.. I think the point of the dual-license is that there is a lot of non linux-specific Infiniband code that will (hopefully) be usefull on other platforms where a BSD license might be more usefull. If for some reason I decided I wanted to run MacOSX, I would at least want to be running the OpenIB infiniband stack, and not some proprietary module. Does anyone have some nice scripts to audit for useage of EXPORT_SYMBOL_GPL only functions? Maybe it's worth trying to clean up the code to clearly deliniate what depends on GPL functions and what doesn't. From jlentini at netapp.com Wed Jun 29 09:42:03 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 29 Jun 2005 12:42:03 -0400 (EDT) Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <52br5qtgk9.fsf@topspin.com> References: <52br5qtgk9.fsf@topspin.com> Message-ID: On Tue, 28 Jun 2005, Roland Dreier wrote: > James> First off, here [are the] requirement we are trying to satisfy: > > James> On the passive side of a connection, a InfiniBand kDAPL > James> provider must determine a source IB address for an > James> InfiniBand connection request. This information can be > James> obtain by a kDAPL consumer either in the > James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or > James> dat_cr_query()'s dat_cr_param structure. > > James> By interoperable, we mean that the solution must not > James> introduce a non-standard protocol or force ULPs using kDAPL > James> to perform special operations when using an InfiniBand > James> network. > > Since these two points are mutually contradictory -- the IB > communication management protocol does not carry enough information > for a connection request to be mapped uniquely back to a source > address -- we need to figure out which one to drop. > > I would argue in favor of the solution selected by SDP: when defining > the binding of an abstract protocol to the IB transport, put the > source and destination IP addresses in the IB-specific connection > setup messages. I want to make sure I understand your solution. If we choose this option: - IB services (both kernel and user space) will be configured using IP addresses. By IB services, I'm referring to protocols that are layered directly on top of the InfiniBand protocols (e.g. SDP, NFS-RDMA, iSER, etc.). - IB services will resolve IP addresses to IB addresses using IPoIB ARP (all IB nodes would be required to support IPoIB). - each IB service would place appropriate IP address information in its protocol messages Did I get it right? james From rolandd at cisco.com Wed Jun 29 09:46:44 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 09:46:44 -0700 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: (James Lentini's message of "Wed, 29 Jun 2005 12:42:03 -0400 (EDT)") References: <52br5qtgk9.fsf@topspin.com> Message-ID: <521x6lf6vf.fsf@topspin.com> James> I want to make sure I understand your solution. If we choose this James> option: James> - IB services (both kernel and user space) will be configured using James> IP addresses. By IB services, I'm referring to protocols that are James> layered directly on top of the InfiniBand protocols (e.g. SDP, James> NFS-RDMA, iSER, etc.). James> - IB services will resolve IP addresses to IB addresses using IPoIB James> ARP (all IB nodes would be required to support IPoIB). James> - each IB service would place appropriate IP address information in James> its protocol messages That's mostly right, except that I would leave it up to the individual protocols if and how they want to use IP addressing. For example, the SCSI RDMA Protocol (SRP) published by INCITS T10 does not use IP addresses in any way, and it seems that should be allowed and supported. - R. From rolandd at cisco.com Wed Jun 29 10:01:53 2005 From: rolandd at cisco.com (Roland Dreier) Date: Wed, 29 Jun 2005 10:01:53 -0700 Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <20050629002709.GB17805@kroah.com> (Greg KH's message of "Tue, 28 Jun 2005 17:27:09 -0700") References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> <20050629002709.GB17805@kroah.com> Message-ID: <52slz1drlq.fsf@topspin.com> Greg> This is no longer needed with the class device interface in Greg> the kernel today. Please use the new api (basically just Greg> set dev_t in the class_device, and you get this for free.) Here's a patch that applies on top of this patch set that fixes this: Greg KH pointed out that with the new class device code, we can just set class_dev.devt instead of having our own show_dev() function. Signed-off-by: Roland Dreier --- linux.orig/drivers/infiniband/core/uverbs_main.c 2005-06-28 15:20:04.000000000 -0700 +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-29 09:54:26.560138283 -0700 @@ -509,15 +509,6 @@ .remove = ib_uverbs_remove_one }; -static ssize_t show_dev(struct class_device *class_dev, char *buf) -{ - struct ib_uverbs_device *dev = - container_of(class_dev, struct ib_uverbs_device, class_dev); - - return print_dev_t(buf, dev->dev.dev); -} -static CLASS_DEVICE_ATTR(dev, S_IRUGO, show_dev, NULL); - static ssize_t show_ibdev(struct class_device *class_dev, char *buf) { struct ib_uverbs_device *dev = @@ -584,12 +575,11 @@ uverbs_dev->class_dev.class = &uverbs_class; uverbs_dev->class_dev.dev = device->dma_device; + uverbs_dev->class_dev.devt = uverbs_dev->dev.dev; snprintf(uverbs_dev->class_dev.class_id, BUS_ID_SIZE, "uverbs%d", uverbs_dev->devnum); if (class_device_register(&uverbs_dev->class_dev)) goto err_cdev; - if (class_device_create_file(&uverbs_dev->class_dev, &class_device_attr_dev)) - goto err_class; if (class_device_create_file(&uverbs_dev->class_dev, &class_device_attr_ibdev)) goto err_class; From greg at kroah.com Wed Jun 29 11:03:13 2005 From: greg at kroah.com (Greg KH) Date: Wed, 29 Jun 2005 11:03:13 -0700 Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <52slz1drlq.fsf@topspin.com> References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> <20050629002709.GB17805@kroah.com> <52slz1drlq.fsf@topspin.com> Message-ID: <20050629180313.GC25066@kroah.com> On Wed, Jun 29, 2005 at 10:01:53AM -0700, Roland Dreier wrote: > Greg> This is no longer needed with the class device interface in > Greg> the kernel today. Please use the new api (basically just > Greg> set dev_t in the class_device, and you get this for free.) > > Here's a patch that applies on top of this patch set that fixes this: > > > Greg KH pointed out that with the new class device code, we can just > set class_dev.devt instead of having our own show_dev() function. > > Signed-off-by: Roland Dreier Nice, thanks for doing this. You also get better userspace support as now the MAJOR and MINOR environment variables are set for the hotplug event when you create this device. Which, if Kay is correct, will make udev even faster... thanks, greg k-h From libor at topspin.com Wed Jun 29 11:10:38 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 29 Jun 2005 11:10:38 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <42C1BA15.7060205@ichips.intel.com>; from ardavis@ichips.intel.com on Tue, Jun 28, 2005 at 01:59:01PM -0700 References: <42C1BA15.7060205@ichips.intel.com> Message-ID: <20050629111038.G26240@topspin.com> On Tue, Jun 28, 2005 at 01:59:01PM -0700, Arlin Davis wrote: > Hi Libor, > > I have a couple of uCM questions regarding create_id and events... > > Is it possible for a consumer of uCM to provide a context with the > create_id that could be returned with the event? I will have some scale > up issues if I have to walk a list looking for a uCM provided connection > ID instead of a context that could point directly to the appropriate > uDAPL CM object. It would be easy to add in a context variable. I had left it out on purpose, since it's easy to get into a situation where using the context as a pointer you can end up referencing deallocated memory. However, I suppose it should be there for flexability. > It would also be very helpful if the CM event could be woke from user space. You mean break CM event out of it's wait? I would instead recommend that you call poll on the file descriptor for read readiness, and only call for the CM event when there's an event available. Also, if you insist on using threads, tou can break out of poll by sending a signal to the thread that is waiting on the poll, which will return -1 with an errno of EINTR. -Libor From libor at topspin.com Wed Jun 29 11:14:40 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 29 Jun 2005 11:14:40 -0700 Subject: [openib-general] [PATCH] sdp_inet: fix schedule_timeout() usage In-Reply-To: <20050628204855.GA24226@us.ibm.com>; from nacc@us.ibm.com on Tue, Jun 28, 2005 at 01:48:55PM -0700 References: <20050628204855.GA24226@us.ibm.com> Message-ID: <20050629111440.H26240@topspin.com> On Tue, Jun 28, 2005 at 01:48:55PM -0700, Nishanth Aravamudan wrote: > > Using schedule_timeout() without setting the state first is broken and > causes schedule_timeout() to return immediately (effectively you call > schedule() without changing your state and are thus going to run again). > In each of these loops in sdp_inet.c involving schedule_timeout(), the > first iteration is correct, but subsequent ones result in busy-wait. Add > the appropriate set_current_state() call to fix the issue. Nish, Thank you for the patch. I'm wondering if it would be better to just move the existing call to set_current_state(TASK_INTERRUPTIBLE) from outside the loop to inside at the begining, as below? -Libor Index: sdp_inet.c =================================================================== --- sdp_inet.c (revision 2749) +++ sdp_inet.c (working copy) @@ -317,10 +317,11 @@ timeout = sk->sk_lingertime; add_wait_queue(sk->sk_sleep, &wait); - set_current_state(TASK_INTERRUPTIBLE); while (timeout > 0 && !(SDP_ST_MASK_CLOSED & conn->state)) { + + set_current_state(TASK_INTERRUPTIBLE); sdp_conn_unlock(conn); timeout = schedule_timeout(timeout); sdp_conn_lock(conn); @@ -554,10 +555,10 @@ DECLARE_WAITQUEUE(wait, current); add_wait_queue(sk->sk_sleep, &wait); - set_current_state(TASK_INTERRUPTIBLE); while (timeout > 0 && (conn->state & SDP_ST_MASK_CONNECT)) { + set_current_state(TASK_INTERRUPTIBLE); sdp_conn_unlock(conn); timeout = schedule_timeout(timeout); sdp_conn_lock(conn); @@ -710,11 +711,12 @@ if (!accept_conn) { DECLARE_WAITQUEUE(wait, current); add_wait_queue(listen_sk->sk_sleep, &wait); - set_current_state(TASK_INTERRUPTIBLE); while (timeout > 0 && listen_conn->state == SDP_CONN_ST_LISTEN && !listen_conn->backlog_cnt) { + + set_current_state(TASK_INTERRUPTIBLE); sdp_conn_unlock(listen_conn); timeout = schedule_timeout(timeout); sdp_conn_lock(listen_conn); From mshefty at ichips.intel.com Wed Jun 29 11:16:19 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 29 Jun 2005 11:16:19 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <20050629111038.G26240@topspin.com> References: <42C1BA15.7060205@ichips.intel.com> <20050629111038.G26240@topspin.com> Message-ID: <42C2E573.5010304@ichips.intel.com> Libor Michalek wrote: >>Is it possible for a consumer of uCM to provide a context with the >>create_id that could be returned with the event? I will have some scale >>up issues if I have to walk a list looking for a uCM provided connection >>ID instead of a context that could point directly to the appropriate >>uDAPL CM object. > > It would be easy to add in a context variable. I had left it out on > purpose, since it's easy to get into a situation where using the context > as a pointer you can end up referencing deallocated memory. However, I > suppose it should be there for flexability. Can you explain the situation where the application could reference deallocated memory? I would think that the uCM could take steps that would make it impossible for a well written app from doing this. - Sean From mst at mellanox.co.il Wed Jun 29 11:22:18 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 29 Jun 2005 21:22:18 +0300 Subject: [openib-general] Re: [PATCH] sdp_inet: fix schedule_timeout() usage In-Reply-To: <20050629111440.H26240@topspin.com> References: <20050628204855.GA24226@us.ibm.com> <20050629111440.H26240@topspin.com> Message-ID: <20050629182218.GF20102@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: [PATCH] sdp_inet: fix schedule_timeout() usage > > On Tue, Jun 28, 2005 at 01:48:55PM -0700, Nishanth Aravamudan wrote: > > > > Using schedule_timeout() without setting the state first is broken and > > causes schedule_timeout() to return immediately (effectively you call > > schedule() without changing your state and are thus going to run again). > > In each of these loops in sdp_inet.c involving schedule_timeout(), the > > first iteration is correct, but subsequent ones result in busy-wait. Add > > the appropriate set_current_state() call to fix the issue. > > Nish, > > Thank you for the patch. I'm wondering if it would be better to just > move the existing call to set_current_state(TASK_INTERRUPTIBLE) from > outside the loop to inside at the begining, as below? > > -Libor > > Index: sdp_inet.c > =================================================================== > --- sdp_inet.c (revision 2749) > +++ sdp_inet.c (working copy) > @@ -317,10 +317,11 @@ > timeout = sk->sk_lingertime; > > add_wait_queue(sk->sk_sleep, &wait); > - set_current_state(TASK_INTERRUPTIBLE); > > while (timeout > 0 && > !(SDP_ST_MASK_CLOSED & conn->state)) { > + > + set_current_state(TASK_INTERRUPTIBLE); > sdp_conn_unlock(conn); > timeout = schedule_timeout(timeout); > sdp_conn_lock(conn); > @@ -554,10 +555,10 @@ > > DECLARE_WAITQUEUE(wait, current); > add_wait_queue(sk->sk_sleep, &wait); > - set_current_state(TASK_INTERRUPTIBLE); > > while (timeout > 0 && (conn->state & SDP_ST_MASK_CONNECT)) { > > + set_current_state(TASK_INTERRUPTIBLE); > sdp_conn_unlock(conn); > timeout = schedule_timeout(timeout); > sdp_conn_lock(conn); > @@ -710,11 +711,12 @@ > if (!accept_conn) { > DECLARE_WAITQUEUE(wait, current); > add_wait_queue(listen_sk->sk_sleep, &wait); > - set_current_state(TASK_INTERRUPTIBLE); > > while (timeout > 0 && > listen_conn->state == SDP_CONN_ST_LISTEN && > !listen_conn->backlog_cnt) { > + > + set_current_state(TASK_INTERRUPTIBLE); > sdp_conn_unlock(listen_conn); > timeout = schedule_timeout(timeout); > sdp_conn_lock(listen_conn); > Shouldnt we be using msleep_interruptible rather than playing with task states explicitly? -- MST From David.Brean at Sun.COM Wed Jun 29 12:39:03 2005 From: David.Brean at Sun.COM (David M. Brean) Date: Wed, 29 Jun 2005 15:39:03 -0400 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <469958e0050628133348e7b64b@mail.gmail.com> References: <52br5qtgk9.fsf@topspin.com> <469958e0050628133348e7b64b@mail.gmail.com> Message-ID: <42C2F8D7.5020100@sun.com> Hello, Caitlin Bestler wrote: >On 6/28/05, Roland Dreier wrote: > > >> James> First off, here [are the] requirement we are trying to satisfy: >> >> James> On the passive side of a connection, a InfiniBand kDAPL >> James> provider must determine a source IB address for an >> James> InfiniBand connection request. This information can be >> James> obtain by a kDAPL consumer either in the >> James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or >> James> dat_cr_query()'s dat_cr_param structure. >> >> James> By interoperable, we mean that the solution must not >> James> introduce a non-standard protocol or force ULPs using kDAPL >> James> to perform special operations when using an InfiniBand >> James> network. >> >>Since these two points are mutually contradictory -- the IB >>communication management protocol does not carry enough information >>for a connection request to be mapped uniquely back to a source >>address -- we need to figure out which one to drop. >> >>I would argue in favor of the solution selected by SDP: when defining >>the binding of an abstract protocol to the IB transport, put the >>source and destination IP addresses in the IB-specific connection >>setup messages. >> >> - R. >> >> > >The remote identification is a service being provided *to* the ULP. >SDP, or any other application, can provide whatever additional >information desired. The "IA Address" is as authenticated as >the local network management allows it to be. Private data >exchanged by applications is outside the view of the network >administrator and therefore can never be authenticated in >the same way. > >For NFSoRDMA, for example, "authenticating" a remote >peer based upon an application supplied field in the private >data is obviously inappropriate. > > > In proposals that I've seen to use the CM private-data area, the information exchanged for authenticating the remote peer is performed by the provider, not the ULP. The DAPL provider is responsible for figuring out how to handle IA addresses. I think this operation is consistent with your statements above. However, I'm aware of one issue. I'm told that the DAPL implementation already uses the CM private-data area to enable ULPs to exchange information. As a result, there isn't any space left over for a SDP-like Hello messages that could be exchanged by the providers. Reducing the space available to DAPL ULPs to make room for the Hello message may be a change that is visible to ULPs even though this is not an API change. >The GID that InfiniBand does supply *does* fully qualify as >an IP Address, and can be the address supplied as the >"remote address" even *if* there are additional methods >of translating IPv4 addresses to GIDs (or even IPv6, but >why you would want to *translate* an IPv6 address to >a GID rather than just using the GID *as* an IPv6 >address is beyond me). > > > I recommend that the mechanism that is chosen be capable of supporting all three cases. Furthermore, I think the impact on IP administrative tools needs to be assessed when using IB GIDs as IPv6 addresses. For example, should it be possible to use ifconfig(1m) to assign an IPv6 address to the GID on an IB port? What is the observed behavior of IP administrative tools when one of the "additional properties / restrictions defined within IBA" related to GID assignment is encountered? What interfaces are required in the SM to enable coordination of IB GIDs with IP layer infrastructure? I think this is all doable, but a bit of work. I think that we should view the ability to use GIDs as IPv6 addresses as an optimization, not a fundamental function of the mechanism. Deployment of the optimization can then occur later. -David >To clarify: DAT *does* fully support the use of GIDs as >IA Addresses, however it assumes that link local addresses >will not be presented to the Consumer (at least when the >host is attached to more than one subnet). It assumes >that they will be at least upgraded to site-local. > >The IA Address can be easily sub-divided into IPv4 >addresses that need translation, IPv6 addresses that >need translation and IPv6 addresses that are also GIDs >and therefore do not need translation. > >The existence of the alternate solutions was largely >driven by the need to deploy early solutions that were >*not* integrated with the OS naming system, and therefore >could not assume that the OS already knew what to do >with a GID. > >That is no longer a problem. > >Therefore there is no need to change any of the DAT >semantics. They are adequate as they are, and in >particular there is no need to eliminate reporting >of remote peer addresses -- something that is both >easy and useful for IP networks. And a feature that >is already in use. > > From ardavis at ichips.intel.com Wed Jun 29 12:42:09 2005 From: ardavis at ichips.intel.com (Arlin Davis) Date: Wed, 29 Jun 2005 12:42:09 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <20050629111038.G26240@topspin.com> References: <42C1BA15.7060205@ichips.intel.com> <20050629111038.G26240@topspin.com> Message-ID: <42C2F991.60005@ichips.intel.com> Libor Michalek wrote: >On Tue, Jun 28, 2005 at 01:59:01PM -0700, Arlin Davis wrote: > > >>Hi Libor, >> >>I have a couple of uCM questions regarding create_id and events... >> >>Is it possible for a consumer of uCM to provide a context with the >>create_id that could be returned with the event? I will have some scale >>up issues if I have to walk a list looking for a uCM provided connection >>ID instead of a context that could point directly to the appropriate >>uDAPL CM object. >> >> > > It would be easy to add in a context variable. I had left it out on >purpose, since it's easy to get into a situation where using the context >as a pointer you can end up referencing deallocated memory. However, I >suppose it should be there for flexability. > > Thanks. Also, in the case with listen's (conn_req), the event returns the new conn_id but not the original conn_id associated with the listen. I will need the listen conn_id to associate back to the proper listen service point. > > >>It would also be very helpful if the CM event could be woke from user space. >> >> > > You mean break CM event out of it's wait? I would instead recommend that >you call poll on the file descriptor for read readiness, and only call for >the CM event when there's an event available. Also, if you insist on using >threads, tou can break out of poll by sending a signal to the thread that >is waiting on the poll, which will return -1 with an errno of EINTR. > > Fair enough. Have you tried the polling method? I am having problems waking up on the event even though I see the following kernel message from ucm. "kernel: UCM: Event. CM ID <10> event <1>" -arlin > >-Libor > > > From nacc at us.ibm.com Wed Jun 29 14:17:54 2005 From: nacc at us.ibm.com (Nishanth Aravamudan) Date: Wed, 29 Jun 2005 14:17:54 -0700 Subject: [openib-general] [PATCH] sdp_inet: fix schedule_timeout() usage In-Reply-To: <20050629111440.H26240@topspin.com> References: <20050628204855.GA24226@us.ibm.com> <20050629111440.H26240@topspin.com> Message-ID: <20050629211754.GB2615@us.ibm.com> On 29.06.2005 [11:14:40 -0700], Libor Michalek wrote: > On Tue, Jun 28, 2005 at 01:48:55PM -0700, Nishanth Aravamudan wrote: > > > > Using schedule_timeout() without setting the state first is broken and > > causes schedule_timeout() to return immediately (effectively you call > > schedule() without changing your state and are thus going to run again). > > In each of these loops in sdp_inet.c involving schedule_timeout(), the > > first iteration is correct, but subsequent ones result in busy-wait. Add > > the appropriate set_current_state() call to fix the issue. > > Nish, > > Thank you for the patch. I'm wondering if it would be better to just > move the existing call to set_current_state(TASK_INTERRUPTIBLE) from > outside the loop to inside at the begining, as below? The problem with that change is that since the task-state has not changed, you may miss some signal-related wake-ups before you set the state. Really, you probably could use prepare_to_wait() and finish_wait() for all of this? Thanks, Nish From jlentini at netapp.com Wed Jun 29 14:18:53 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 29 Jun 2005 17:18:53 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [uDAPL] Fix compile of dapl_bpool.c on x86_64 In-Reply-To: <1119972645.4848.2434.camel@hal.voltaire.com> References: <1119972645.4848.2434.camel@hal.voltaire.com> Message-ID: Committed in revision 2751. On Tue, 28 Jun 2005, Hal Rosenstock wrote: halr> I was able to build udapltest for both x86 and x86-64 with the following halr> change: halr> halr> uDAPL: Fix compile of dapl_bpool.c on x86_64 halr> halr> Signed-off-by: Hal Rosenstock halr> halr> --- userspace/test/dapltest/test/dapl_bpool.c.orig 2005-01-18 halr> 13:20:31.000000000 -0500 halr> +++ userspace/test/dapltest/test/dapl_bpool.c 2005-06-24 halr> 23:47:43.000000000 -0400 halr> @@ -356,8 +356,8 @@ halr> "BPOOL alloc_size %x\n", halr> (int) bpool_ptr->alloc_size); halr> DT_Tdep_PT_Printf (phead, halr> - "BPOOL pz_handle %x\n", halr> - (int) bpool_ptr->pz_handle); halr> + "BPOOL pz_handle %p\n", halr> + bpool_ptr->pz_handle); halr> DT_Tdep_PT_Printf (phead, halr> "BPOOL num_segs %x\n", halr> (int) bpool_ptr->num_segs); halr> halr> From nacc at us.ibm.com Wed Jun 29 14:19:05 2005 From: nacc at us.ibm.com (Nishanth Aravamudan) Date: Wed, 29 Jun 2005 14:19:05 -0700 Subject: [openib-general] Re: [PATCH] sdp_inet: fix schedule_timeout() usage In-Reply-To: <20050629182218.GF20102@mellanox.co.il> References: <20050628204855.GA24226@us.ibm.com> <20050629111440.H26240@topspin.com> <20050629182218.GF20102@mellanox.co.il> Message-ID: <20050629211905.GC2615@us.ibm.com> On 29.06.2005 [21:22:18 +0300], Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > Subject: Re: [PATCH] sdp_inet: fix schedule_timeout() usage > > > > On Tue, Jun 28, 2005 at 01:48:55PM -0700, Nishanth Aravamudan wrote: > > > > > > Using schedule_timeout() without setting the state first is broken and > > > causes schedule_timeout() to return immediately (effectively you call > > > schedule() without changing your state and are thus going to run again). > > > In each of these loops in sdp_inet.c involving schedule_timeout(), the > > > first iteration is correct, but subsequent ones result in busy-wait. Add > > > the appropriate set_current_state() call to fix the issue. > > > > Nish, > > > > Thank you for the patch. I'm wondering if it would be better to just > > move the existing call to set_current_state(TASK_INTERRUPTIBLE) from > > outside the loop to inside at the begining, as below? > > > > -Libor > > > > Index: sdp_inet.c > > =================================================================== > > --- sdp_inet.c (revision 2749) > > +++ sdp_inet.c (working copy) > > @@ -317,10 +317,11 @@ > > timeout = sk->sk_lingertime; > > > > add_wait_queue(sk->sk_sleep, &wait); > > - set_current_state(TASK_INTERRUPTIBLE); > > > > while (timeout > 0 && > > !(SDP_ST_MASK_CLOSED & conn->state)) { > > + > > + set_current_state(TASK_INTERRUPTIBLE); > > sdp_conn_unlock(conn); > > timeout = schedule_timeout(timeout); > > sdp_conn_lock(conn); > > @@ -554,10 +555,10 @@ > > > > DECLARE_WAITQUEUE(wait, current); > > add_wait_queue(sk->sk_sleep, &wait); > > - set_current_state(TASK_INTERRUPTIBLE); > > > > while (timeout > 0 && (conn->state & SDP_ST_MASK_CONNECT)) { > > > > + set_current_state(TASK_INTERRUPTIBLE); > > sdp_conn_unlock(conn); > > timeout = schedule_timeout(timeout); > > sdp_conn_lock(conn); > > @@ -710,11 +711,12 @@ > > if (!accept_conn) { > > DECLARE_WAITQUEUE(wait, current); > > add_wait_queue(listen_sk->sk_sleep, &wait); > > - set_current_state(TASK_INTERRUPTIBLE); > > > > while (timeout > 0 && > > listen_conn->state == SDP_CONN_ST_LISTEN && > > !listen_conn->backlog_cnt) { > > + > > + set_current_state(TASK_INTERRUPTIBLE); > > sdp_conn_unlock(listen_conn); > > timeout = schedule_timeout(timeout); > > sdp_conn_lock(listen_conn); > > > > Shouldnt we be using msleep_interruptible rather than playing with > task states explicitly? Not in these cases; msleep_{,interruptible}() cannot be used around wait-queues (they both loop on the requested timeout, and thus ignore all wake-up events (except signals in the latter case)). I am verifying that no other places in the openib code might be able to take advantage of these helpers, though. Thanks, Nish From mst at mellanox.co.il Wed Jun 29 14:24:10 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 30 Jun 2005 00:24:10 +0300 Subject: [openib-general] Re: [PATCH] sdp_inet: fix schedule_timeout() usage In-Reply-To: <20050629211905.GC2615@us.ibm.com> References: <20050628204855.GA24226@us.ibm.com> <20050629111440.H26240@topspin.com> <20050629182218.GF20102@mellanox.co.il> <20050629211905.GC2615@us.ibm.com> Message-ID: <20050629212410.GG20102@mellanox.co.il> Quoting r. Nishanth Aravamudan : > > > @@ -710,11 +711,12 @@ > > > if (!accept_conn) { > > > DECLARE_WAITQUEUE(wait, current); > > > add_wait_queue(listen_sk->sk_sleep, &wait); > > > - set_current_state(TASK_INTERRUPTIBLE); > > > > > > while (timeout > 0 && > > > listen_conn->state == SDP_CONN_ST_LISTEN && > > > !listen_conn->backlog_cnt) { > > > + > > > + set_current_state(TASK_INTERRUPTIBLE); > > > sdp_conn_unlock(listen_conn); > > > timeout = schedule_timeout(timeout); > > > sdp_conn_lock(listen_conn); What is the reason set_current_state is done before sdp_conn_unlock and not just before schedule_timeout? -- MST From nacc at us.ibm.com Wed Jun 29 14:43:29 2005 From: nacc at us.ibm.com (Nishanth Aravamudan) Date: Wed, 29 Jun 2005 14:43:29 -0700 Subject: [openib-general] Re: [PATCH] sdp_inet: fix schedule_timeout() usage In-Reply-To: <20050629212410.GG20102@mellanox.co.il> References: <20050628204855.GA24226@us.ibm.com> <20050629111440.H26240@topspin.com> <20050629182218.GF20102@mellanox.co.il> <20050629211905.GC2615@us.ibm.com> <20050629212410.GG20102@mellanox.co.il> Message-ID: <20050629214329.GD2615@us.ibm.com> On 30.06.2005 [00:24:10 +0300], Michael S. Tsirkin wrote: > Quoting r. Nishanth Aravamudan : > > > > @@ -710,11 +711,12 @@ > > > > if (!accept_conn) { > > > > DECLARE_WAITQUEUE(wait, current); > > > > add_wait_queue(listen_sk->sk_sleep, &wait); > > > > - set_current_state(TASK_INTERRUPTIBLE); > > > > > > > > while (timeout > 0 && > > > > listen_conn->state == SDP_CONN_ST_LISTEN && > > > > !listen_conn->backlog_cnt) { > > > > + > > > > + set_current_state(TASK_INTERRUPTIBLE); > > > > sdp_conn_unlock(listen_conn); > > > > timeout = schedule_timeout(timeout); > > > > sdp_conn_lock(listen_conn); > > What is the reason set_current_state is done before sdp_conn_unlock > and not just before schedule_timeout? That's a good point; doesn't really matter where in the loop the set happens as long as it is before, if Libor's version goes in, or after, if my version does. But, as I said, I think Libor's version might miss wake-up events. Thanks, Nish From rep.nop at aon.at Wed Jun 29 14:50:46 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Wed, 29 Jun 2005 23:50:46 +0200 Subject: [openib-general] Re: [PATCH][dapl] cleanup dapl_cookie In-Reply-To: References: <20050627173951.GB19279@aon.at> Message-ID: <20050629215046.GB17011@aon.at> On Tue, Jun 28, 2005 at 03:39:59PM -0400, James Lentini wrote: > >Hi Bernhard, > >The changes look fine. Why the additional copyright? I need to be able >to explain it to my legal department. My legaleeze states that whatever i do during work-time is contributed to work and whatever is related to work done during leasure time has to be attributed to /me _at_ _least_. As that snippet (which was a test-balloon for that category) clearly was done in my spare time, i'm forced to attribute it accordingly :-/ Does that answer your question satisfactorily? > >james Sorry, -- Bernhard From rep.nop at aon.at Wed Jun 29 15:19:37 2005 From: rep.nop at aon.at (Bernhard Fischer) Date: Thu, 30 Jun 2005 00:19:37 +0200 Subject: [openib-general] Re: [PATCH] [uDAPL] Fix compile of dapl_bpool.c on x86_64 In-Reply-To: References: <1119972645.4848.2434.camel@hal.voltaire.com> Message-ID: <20050629221937.GC17011@aon.at> On Wed, Jun 29, 2005 at 05:18:53PM -0400, James Lentini wrote: > >Committed in revision 2751. > >On Tue, 28 Jun 2005, Hal Rosenstock wrote: > >halr> I was able to build udapltest for both x86 and x86-64 with the following >halr> change: >halr> >halr> uDAPL: Fix compile of dapl_bpool.c on x86_64 >halr> >halr> Signed-off-by: Hal Rosenstock >halr> >halr> --- userspace/test/dapltest/test/dapl_bpool.c.orig 2005-01-18 >halr> 13:20:31.000000000 -0500 >halr> +++ userspace/test/dapltest/test/dapl_bpool.c 2005-06-24 >halr> 23:47:43.000000000 -0400 >halr> @@ -356,8 +356,8 @@ [snip full patch] James, Don't mean to sound rude, but imagine there *still* are modem-users around! That may sound odd to you as you're well connected, still those do exists.. May i kindly ask that you do *not* quote the patch you ACK'ed but just the subj/thread? Folks interrested still can readup in the mail-archives. Look at e.g. Roland, who also does not quote the patch itself but politely follows-up with something along the lines of 'comitted'. Thank you. Yours sincerely, From sean.hefty at intel.com Wed Jun 29 15:23:44 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 29 Jun 2005 15:23:44 -0700 Subject: [openib-general] [PATCH] [CM] fix extracting packet lifetime from local ack timeout Message-ID: The following patch (partially) fixes an issue extracting the packet lifetime from the local ACK timeout when processing a received CM REQ or LAP message. The local ACK timeout carries twice the packet lifetime, plus CA ACK delay - which I'm currently ignoring as being too painful to deal with. Signed-off-by: Sean Hefty Index: core/cm.c =================================================================== --- core/cm.c (revision 2749) +++ core/cm.c (working copy) @@ -1005,6 +1005,7 @@ static inline void cm_format_paths_from_ primary_path->packet_life_time_selector = IB_SA_EQ; primary_path->packet_life_time = cm_req_get_primary_local_ack_timeout(req_msg); + primary_path->packet_life_time -= (primary_path->packet_life_time > 0); if (req_msg->alt_local_lid) { memset(alt_path, 0, sizeof *alt_path); @@ -1025,6 +1026,7 @@ static inline void cm_format_paths_from_ alt_path->packet_life_time_selector = IB_SA_EQ; alt_path->packet_life_time = cm_req_get_alt_local_ack_timeout(req_msg); + alt_path->packet_life_time -= (alt_path->packet_life_time > 0); } } @@ -2306,6 +2308,7 @@ static void cm_format_path_from_lap(stru path->rate = cm_lap_get_packet_rate(lap_msg); path->packet_life_time_selector = IB_SA_EQ; path->packet_life_time = cm_lap_get_local_ack_timeout(lap_msg); + path->packet_life_time -= (path->packet_life_time > 0); } static int cm_lap_handler(struct cm_work *work) From tduffy at sun.com Wed Jun 29 15:53:21 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 29 Jun 2005 15:53:21 -0700 Subject: [openib-general] [PATCH] ucm build fixes for 2.6.13-rc1 Message-ID: <1120085601.31370.45.camel@duffman> This patch fixes building ucm against 2.6.13-rc1. Please apply when 2.6.13 has arrived. Signed-off-by: Tom Duffy Index: linux-2.6.13-rc1/drivers/infiniband/core/ucm.c =================================================================== --- linux-2.6.13-rc1/drivers/infiniband/core/ucm.c (revision 2751) +++ linux-2.6.13-rc1/drivers/infiniband/core/ucm.c (working copy) @@ -1339,7 +1339,7 @@ static struct file_operations ib_ucm_fop }; -static struct class_simple *ib_ucm_class; +static struct class *ib_ucm_class; static struct cdev ib_ucm_cdev; static int __init ib_ucm_init(void) @@ -1360,17 +1360,14 @@ static int __init ib_ucm_init(void) goto err_cdev; } - ib_ucm_class = class_simple_create(THIS_MODULE, "infiniband_cm"); + ib_ucm_class = class_create(THIS_MODULE, "infiniband_cm"); if (IS_ERR(ib_ucm_class)) { result = PTR_ERR(ib_ucm_class); printk(KERN_ERR "UCM: Error <%d> creating class\n", result); goto err_class; } - class_simple_device_add(ib_ucm_class, - IB_UCM_DEV, - NULL, - "ucm"); + class_device_create(ib_ucm_class, IB_UCM_DEV, NULL, "ucm"); idr_init(&ctx_id_table); init_MUTEX(&ctx_id_mutex); @@ -1386,8 +1383,8 @@ err_chr: static void __exit ib_ucm_cleanup(void) { - class_simple_device_remove(IB_UCM_DEV); - class_simple_destroy(ib_ucm_class); + class_device_destroy(ib_ucm_class, IB_UCM_DEV); + class_destroy(ib_ucm_class); cdev_del(&ib_ucm_cdev); unregister_chrdev_region(IB_UCM_DEV, 1); } From tduffy at sun.com Wed Jun 29 15:59:30 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 29 Jun 2005 15:59:30 -0700 Subject: [openib-general] [PATCHv3] SDP: rename sdp_opt to sdp_sock Message-ID: <1120085970.31370.50.camel@duffman> This patch renames the sdp_opt struct sdp_sock as this is what all the other socket classes do. Against r2751. Signed-off-by: Tom Duffy Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_write.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_write.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_write.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_event_write - RDMA write event handler */ -int sdp_event_write(struct sdp_opt *conn, struct ib_wc *comp) +int sdp_event_write(struct sdp_sock *conn, struct ib_wc *comp) { struct sdpc_iocb *iocb; struct sdpc_buff *buff; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -38,7 +38,7 @@ /* * Specific MID handler functions. (RECV) */ -static int sdp_rcvd_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_disconnect(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sock *sk; int result = 0; @@ -102,7 +102,7 @@ error: return result; } -static int sdp_rcvd_abort(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_abort(struct sdp_sock *conn, struct sdpc_buff *buff) { int result = 0; @@ -126,7 +126,7 @@ static int sdp_rcvd_abort(struct sdp_opt return result; } -static int sdp_rcvd_send_sm(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_send_sm(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; @@ -167,7 +167,7 @@ static int sdp_rcvd_send_sm(struct sdp_o return 0; } -static int sdp_rcvd_rdma_wr(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_rdma_wr(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_rwch *rwch; struct sdpc_iocb *iocb; @@ -217,7 +217,7 @@ static int sdp_rcvd_rdma_wr(struct sdp_o return 0; } -static int sdp_rcvd_rdma_rd(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_rdma_rd(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_rrch *rrch; struct sdpc_iocb *iocb; @@ -288,7 +288,7 @@ static int sdp_rcvd_rdma_rd(struct sdp_o return 0; } -static int sdp_rcvd_mode_change(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_mode_change(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_mch *mch; int result; @@ -384,7 +384,7 @@ error: return result; } -static int sdp_rcvd_src_cancel(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_src_cancel(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_advt *advt; int result; @@ -473,7 +473,7 @@ done: return result; } -static int sdp_rcvd_snk_cancel(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_snk_cancel(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_advt *advt; s32 counter; @@ -546,7 +546,7 @@ done: /* * sdp_rcvd_snk_cancel_ack - sink cancel confirmantion */ -static int sdp_rcvd_snk_cancel_ack(struct sdp_opt *conn, +static int sdp_rcvd_snk_cancel_ack(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; @@ -576,7 +576,7 @@ static int sdp_rcvd_snk_cancel_ack(struc /* * sdp_rcvd_resize_buff_ack - buffer size change request */ -static int sdp_rcvd_resize_buff_ack(struct sdp_opt *conn, +static int sdp_rcvd_resize_buff_ack(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_crbh *crbh; @@ -606,7 +606,7 @@ error: return result; } -static int sdp_rcvd_suspend(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_suspend(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_sch *sch; @@ -618,12 +618,12 @@ static int sdp_rcvd_suspend(struct sdp_o return 0; } -static int sdp_rcvd_suspend_ack(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_suspend_ack(struct sdp_sock *conn, struct sdpc_buff *buff) { return 0; } -static int sdp_rcvd_snk_avail(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_snk_avail(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_snkah *snkah; struct sdpc_advt *advt; @@ -748,7 +748,7 @@ consume: return result; } -static int sdp_rcvd_src_avail(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_src_avail(struct sdp_sock *conn, struct sdpc_buff *buff) { struct msg_hdr_srcah *srcah; struct sdpc_advt *advt; @@ -917,7 +917,7 @@ done: /* * sdp_rcvd_data - SDP data message event received */ -static int sdp_rcvd_data(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_data(struct sdp_sock *conn, struct sdpc_buff *buff) { int ret_val; @@ -962,7 +962,7 @@ static int sdp_rcvd_data(struct sdp_opt /* * sdp_rcvd_unsupported - Valid messages we're not expecting */ -static int sdp_rcvd_unsupported(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_rcvd_unsupported(struct sdp_sock *conn, struct sdpc_buff *buff) { /* * Since the gateway only initates RDMA's but is never a target, and @@ -1026,7 +1026,7 @@ static sdp_event_cb_func recv_event_func /* * sdp_event_recv - recv event demultiplexing into sdp messages */ -int sdp_event_recv(struct sdp_opt *conn, struct ib_wc *comp) +int sdp_event_recv(struct sdp_sock *conn, struct ib_wc *comp) { sdp_event_cb_func dispatch_func; struct sdpc_buff *buff; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -130,7 +130,7 @@ MODULE_PARM_DESC(sdp_debug_level, */ void sdp_inet_wake_send(struct sock *sk) { - struct sdp_opt *conn = sdp_sk(sk); + struct sdp_sock *conn = sdp_sk(sk); if (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) && sdp_inet_writable(conn)) { @@ -151,7 +151,7 @@ void sdp_inet_wake_send(struct sock *sk) /* * sdp_inet_disconnect - disconnect a connection */ -static int sdp_inet_disconnect(struct sdp_opt *conn) +static int sdp_inet_disconnect(struct sdp_sock *conn) { struct sock *sk; int result = 0; @@ -235,7 +235,7 @@ error: */ static int sdp_inet_release(struct socket *sock) { - struct sdp_opt *conn; + struct sdp_sock *conn; struct sock *sk = sock->sk; int result; long timeout; @@ -365,7 +365,7 @@ static int sdp_inet_bind(struct socket * { struct sockaddr_in *addr = (struct sockaddr_in *)uaddr; struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; unsigned int addr_result = RTN_UNSPEC; u16 bind_port; int result; @@ -457,7 +457,7 @@ static int sdp_inet_connect(struct socke { struct sockaddr_in *addr = (struct sockaddr_in *)uaddr; struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; long timeout; int result; @@ -620,7 +620,7 @@ done: static int sdp_inet_listen(struct socket *sock, int backlog) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; int result; sk = sock->sk; @@ -679,8 +679,8 @@ static int sdp_inet_accept(struct socket { struct sock *listen_sk; struct sock *accept_sk = NULL; - struct sdp_opt *listen_conn; - struct sdp_opt *accept_conn = NULL; + struct sdp_sock *listen_conn; + struct sdp_sock *accept_conn = NULL; int result; long timeout; @@ -807,7 +807,7 @@ static int sdp_inet_getname(struct socke { struct sockaddr_in *addr = (struct sockaddr_in *)uaddr; struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; sk = sock->sk; conn = sdp_sk(sk); @@ -842,7 +842,7 @@ static unsigned int sdp_inet_poll(struct poll_table *wait) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; unsigned int mask = 0; /* @@ -931,7 +931,7 @@ static int sdp_inet_ioctl(struct socket unsigned long arg) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_buff *buff; int result = 0; int value; @@ -1054,7 +1054,7 @@ static int sdp_inet_setopt(struct socket char __user *optval, int optlen) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; int value; int result = 0; @@ -1119,7 +1119,7 @@ static int sdp_inet_getopt(struct socket char __user *optval, int __user *optlen) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; int value; int len; @@ -1180,7 +1180,7 @@ static int sdp_inet_getopt(struct socket static int sdp_inet_shutdown(struct socket *sock, int flag) { int result = 0; - struct sdp_opt *conn; + struct sdp_sock *conn; conn = sdp_sk(sock->sk); @@ -1267,7 +1267,7 @@ static struct proto_ops lnx_stream_ops = */ static int sdp_inet_create(struct socket *sock, int protocol) { - struct sdp_opt *conn; + struct sdp_sock *conn; sdp_dbg_ctrl(NULL, "SOCKET: type <%d> proto <%d> state <%u:%08lx>", sock->type, protocol, sock->state, sock->flags); @@ -1320,7 +1320,7 @@ static struct net_proto_family sdp_proto struct proto sdp_sk_proto = { .name = "SDP", .owner = THIS_MODULE, - .obj_size = sizeof(struct sdp_opt), + .obj_size = sizeof(struct sdp_sock), }; /* Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_proto.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_proto.h (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_proto.h (working copy) @@ -108,11 +108,11 @@ int sdp_proc_dump_buff_pool(char *buffer /* * Wall between userspace protocol and SDP protocol proper */ -void sdp_conn_abort(struct sdp_opt *conn); +void sdp_conn_abort(struct sdp_sock *conn); -void sdp_conn_inet_error(struct sdp_opt *conn, int error); +void sdp_conn_inet_error(struct sdp_sock *conn, int error); -int sdp_recv_buff(struct sdp_opt *conn, struct sdpc_buff *buff); +int sdp_recv_buff(struct sdp_sock *conn, struct sdpc_buff *buff); /* * Zcopy advertisment managment @@ -166,7 +166,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_ void sdp_iocb_q_remove(struct sdpc_iocb *iocb); -int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_opt *conn); +int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_sock *conn); void sdp_iocb_release(struct sdpc_iocb *iocb); @@ -258,13 +258,13 @@ int sdp_proc_dump_device(char *buffer, off_t start_index, long *end_index); -int sdp_conn_table_remove(struct sdp_opt *conn); +int sdp_conn_table_remove(struct sdp_sock *conn); -struct sdp_opt *sdp_conn_table_lookup(s32 entry); +struct sdp_sock *sdp_conn_table_lookup(s32 entry); -struct sdp_opt *sdp_conn_alloc(int priority); +struct sdp_sock *sdp_conn_alloc(int priority); -int sdp_conn_alloc_ib(struct sdp_opt *conn, +int sdp_conn_alloc_ib(struct sdp_sock *conn, struct ib_device *device, u8 hw_port, u16 pkey); @@ -274,39 +274,39 @@ void sdp_inet_wake_send(struct sock *sk) /* * port/queue managment */ -void sdp_inet_accept_q_put(struct sdp_opt *listen_conn, - struct sdp_opt *accept_conn); +void sdp_inet_accept_q_put(struct sdp_sock *listen_conn, + struct sdp_sock *accept_conn); -struct sdp_opt *sdp_inet_accept_q_get(struct sdp_opt *listen_conn); +struct sdp_sock *sdp_inet_accept_q_get(struct sdp_sock *listen_conn); -int sdp_inet_accept_q_remove(struct sdp_opt *accept_conn); +int sdp_inet_accept_q_remove(struct sdp_sock *accept_conn); -int sdp_inet_listen_start(struct sdp_opt *listen_conn); +int sdp_inet_listen_start(struct sdp_sock *listen_conn); -int sdp_inet_listen_stop(struct sdp_opt *listen_conn); +int sdp_inet_listen_stop(struct sdp_sock *listen_conn); -struct sdp_opt *sdp_inet_listen_lookup(u32 addr, u16 port); +struct sdp_sock *sdp_inet_listen_lookup(u32 addr, u16 port); -int sdp_inet_port_get(struct sdp_opt *conn, u16 port); +int sdp_inet_port_get(struct sdp_sock *conn, u16 port); -int sdp_inet_port_put(struct sdp_opt *conn); +int sdp_inet_port_put(struct sdp_sock *conn); -void sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child); +void sdp_inet_port_inherit(struct sdp_sock *parent, struct sdp_sock *child); /* * active connect functions */ -int sdp_cm_connect(struct sdp_opt *conn); +int sdp_cm_connect(struct sdp_sock *conn); int sdp_cm_rep_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn); + struct sdp_sock *conn); -void sdp_cm_actv_error(struct sdp_opt *conn, int error); +void sdp_cm_actv_error(struct sdp_sock *conn, int error); /* * passive connect functions */ -int sdp_cm_pass_establish(struct sdp_opt *conn); +int sdp_cm_pass_establish(struct sdp_sock *conn); int sdp_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event); @@ -314,36 +314,36 @@ int sdp_cm_req_handler(struct ib_cm_id * /* * post functions */ -int sdp_recv_flush(struct sdp_opt *conn); +int sdp_recv_flush(struct sdp_sock *conn); -int sdp_send_flush(struct sdp_opt *conn); +int sdp_send_flush(struct sdp_sock *conn); -int sdp_send_ctrl_ack(struct sdp_opt *conn); +int sdp_send_ctrl_ack(struct sdp_sock *conn); -int sdp_send_ctrl_disconnect(struct sdp_opt *conn); +int sdp_send_ctrl_disconnect(struct sdp_sock *conn); -int sdp_send_ctrl_abort(struct sdp_opt *conn); +int sdp_send_ctrl_abort(struct sdp_sock *conn); -int sdp_send_ctrl_send_sm(struct sdp_opt *conn); +int sdp_send_ctrl_send_sm(struct sdp_sock *conn); -int sdp_send_ctrl_snk_avail(struct sdp_opt *conn, +int sdp_send_ctrl_snk_avail(struct sdp_sock *conn, u32 size, u32 rkey, u64 addr); -int sdp_send_ctrl_resize_buff_ack(struct sdp_opt *conn, u32 size); +int sdp_send_ctrl_resize_buff_ack(struct sdp_sock *conn, u32 size); -int sdp_send_ctrl_rdma_rd(struct sdp_opt *conn, s32 size); +int sdp_send_ctrl_rdma_rd(struct sdp_sock *conn, s32 size); -int sdp_send_ctrl_rdma_wr(struct sdp_opt *conn, u32 size); +int sdp_send_ctrl_rdma_wr(struct sdp_sock *conn, u32 size); -int sdp_send_ctrl_mode_ch(struct sdp_opt *conn, u8 mode); +int sdp_send_ctrl_mode_ch(struct sdp_sock *conn, u8 mode); -int sdp_send_ctrl_src_cancel(struct sdp_opt *conn); +int sdp_send_ctrl_src_cancel(struct sdp_sock *conn); -int sdp_send_ctrl_snk_cancel(struct sdp_opt *conn); +int sdp_send_ctrl_snk_cancel(struct sdp_sock *conn); -int sdp_send_ctrl_snk_cancel_ack(struct sdp_opt *conn); +int sdp_send_ctrl_snk_cancel_ack(struct sdp_sock *conn); /* * inet functions @@ -352,20 +352,20 @@ int sdp_send_ctrl_snk_cancel_ack(struct /* * event functions */ -int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_opt *conn); +int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_sock *conn); void sdp_cq_event_handler(struct ib_cq *cq, void *arg); int sdp_cm_event_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event); -int sdp_event_recv(struct sdp_opt *conn, struct ib_wc *comp); +int sdp_event_recv(struct sdp_sock *conn, struct ib_wc *comp); -int sdp_event_send(struct sdp_opt *conn, struct ib_wc *comp); +int sdp_event_send(struct sdp_sock *conn, struct ib_wc *comp); -int sdp_event_read(struct sdp_opt *conn, struct ib_wc *comp); +int sdp_event_read(struct sdp_sock *conn, struct ib_wc *comp); -int sdp_event_write(struct sdp_opt *conn, struct ib_wc *comp); +int sdp_event_write(struct sdp_sock *conn, struct ib_wc *comp); /* * DATA transport @@ -381,11 +381,11 @@ int sdp_inet_recv(struct kiocb *iocb, size_t size, int flags); -void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error); +void sdp_iocb_q_cancel_all_read(struct sdp_sock *conn, ssize_t error); -void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error); +void sdp_iocb_q_cancel_all_write(struct sdp_sock *conn, ssize_t error); -void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error); +void sdp_iocb_q_cancel_all(struct sdp_sock *conn, ssize_t error); /* * link address information @@ -415,7 +415,7 @@ void sdp_link_addr_cleanup(void); /* * Event handling function, demultiplexed base on Message ID */ -typedef int (*sdp_event_cb_func)(struct sdp_opt *conn, +typedef int (*sdp_event_cb_func)(struct sdp_sock *conn, struct sdpc_buff *buff); /* @@ -450,7 +450,7 @@ extern int sdp_debug_level; #define sdp_conn_dbg(level, type, conn, format, arg...) \ do { \ - struct sdp_opt *x = (conn); \ + struct sdp_sock *x = (conn); \ if (x) { \ sdp_dbg_out(level, type, \ "<%d> <%04x> " format, \ @@ -524,7 +524,7 @@ do { /* * sdp_inet_write_space - writable space on send side */ -static inline int sdp_inet_write_space(struct sdp_opt *conn, int urg) +static inline int sdp_inet_write_space(struct sdp_sock *conn, int urg) { int size; /* @@ -547,7 +547,7 @@ static inline int sdp_inet_write_space(s /* * sdp_inet_writable - return non-zero if socket is writable */ -static inline int sdp_inet_writable(struct sdp_opt *conn) +static inline int sdp_inet_writable(struct sdp_sock *conn) { if (conn->send_buf > 0) return (sdp_inet_write_space(conn, 0) < @@ -559,7 +559,7 @@ static inline int sdp_inet_writable(stru /* * sdp_conn_stat_dump - dump stats to the log */ -static inline void sdp_conn_stat_dump(struct sdp_opt *conn) +static inline void sdp_conn_stat_dump(struct sdp_sock *conn) { #ifdef _SDP_CONN_STATS_REC int counter; @@ -581,7 +581,7 @@ static inline void sdp_conn_stat_dump(st /* * sdp_conn_state_dump - dump state information to the log */ -static inline void sdp_conn_state_dump(struct sdp_opt *conn) +static inline void sdp_conn_state_dump(struct sdp_sock *conn) { #ifdef _SDP_CONN_STATE_REC int counter; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_read.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_event_read_advt - RDMA read event handler for source advertisments */ -static int sdp_event_read_advt(struct sdp_opt *conn, struct ib_wc *comp) +static int sdp_event_read_advt(struct sdp_sock *conn, struct ib_wc *comp) { struct sdpc_advt *advt; int result; @@ -106,7 +106,7 @@ error: /* * sdp_event_read - RDMA read event handler */ -int sdp_event_read(struct sdp_opt *conn, struct ib_wc *comp) +int sdp_event_read(struct sdp_sock *conn, struct ib_wc *comp) { struct sdpc_iocb *iocb; struct sdpc_buff *buff; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_send.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_send_buff_post - Post a buffer send on a SDP connection */ -static int sdp_send_buff_post(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_send_buff_post(struct sdp_sock *conn, struct sdpc_buff *buff) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -178,7 +178,7 @@ done: /* * sdp_send_data_buff_post - Post data for buffered transmission */ -static int sdp_send_data_buff_post(struct sdp_opt *conn, +static int sdp_send_data_buff_post(struct sdp_sock *conn, struct sdpc_buff *buff) { int result; @@ -282,7 +282,7 @@ static int sdp_send_data_buff_post(struc /* * sdp_send_data_buff_snk - Post data for buffered transmission */ -static int sdp_send_data_buff_snk(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_send_data_buff_snk(struct sdp_sock *conn, struct sdpc_buff *buff) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -407,7 +407,7 @@ error: /* * sdp_send_data_iocb_snk - process a zcopy write advert in the data path */ -static int sdp_send_data_iocb_snk(struct sdp_opt *conn, struct sdpc_iocb *iocb) +static int sdp_send_data_iocb_snk(struct sdp_sock *conn, struct sdpc_iocb *iocb) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -521,7 +521,7 @@ error: /* * sdp_send_data_iocb_src - send a zcopy read advert in the data path */ -static int sdp_send_data_iocb_src(struct sdp_opt *conn, struct sdpc_iocb *iocb) +static int sdp_send_data_iocb_src(struct sdp_sock *conn, struct sdpc_iocb *iocb) { struct msg_hdr_srcah *src_ah; struct sdpc_buff *buff; @@ -737,7 +737,7 @@ static int sdp_send_iocb_buff_write(stru /* * sdp_send_data_iocb_buff - write multiple SDP buffers from an iocb */ -static int sdp_send_data_iocb_buff(struct sdp_opt *conn, struct sdpc_iocb *iocb) +static int sdp_send_data_iocb_buff(struct sdp_sock *conn, struct sdpc_iocb *iocb) { struct sdpc_buff *buff; int result; @@ -805,7 +805,7 @@ error: /* * sdp_send_data_iocb - Post IOCB data for transmission */ -static int sdp_send_data_iocb(struct sdp_opt *conn, struct sdpc_iocb *iocb) +static int sdp_send_data_iocb(struct sdp_sock *conn, struct sdpc_iocb *iocb) { int result = ENOBUFS; @@ -883,7 +883,7 @@ done: /* * sdp_send_data_queue_test - send data buffer if conditions are met */ -static int sdp_send_data_queue_test(struct sdp_opt *conn, +static int sdp_send_data_queue_test(struct sdp_sock *conn, struct sdpc_desc *element) { int result; @@ -914,7 +914,7 @@ static int sdp_send_data_queue_test(stru /* * sdp_send_data_queue_flush - Flush data from send queue, to send post */ -static int sdp_send_data_queue_flush(struct sdp_opt *conn) +static int sdp_send_data_queue_flush(struct sdp_sock *conn) { struct sdpc_desc *element; int result = 0; @@ -941,7 +941,7 @@ static int sdp_send_data_queue_flush(str /* * sdp_send_data_queue - send using the data queue if necessary */ -static int sdp_send_data_queue(struct sdp_opt *conn, struct sdpc_desc *element) +static int sdp_send_data_queue(struct sdp_sock *conn, struct sdpc_desc *element) { int result = 0; @@ -986,7 +986,7 @@ done: /* * sdp_send_data_buff_get - get an appropriate write buffer for send */ -static inline struct sdpc_buff *sdp_send_data_buff_get(struct sdp_opt *conn) +static inline struct sdpc_buff *sdp_send_data_buff_get(struct sdp_sock *conn) { struct sdpc_buff *buff; @@ -1011,7 +1011,7 @@ static inline struct sdpc_buff *sdp_send /* * sdp_send_data_buff_put - place a buffer into the send queue */ -static inline int sdp_send_data_buff_put(struct sdp_opt *conn, +static inline int sdp_send_data_buff_put(struct sdp_sock *conn, struct sdpc_buff *buff, int size, int urg) { @@ -1066,7 +1066,7 @@ static inline int sdp_send_data_buff_put /* * sdp_send_ctrl_buff_test - determine if it's OK to post a control msg */ -static int sdp_send_ctrl_buff_test(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_send_ctrl_buff_test(struct sdp_sock *conn, struct sdpc_buff *buff) { int result = 0; @@ -1092,7 +1092,7 @@ error: /* * sdp_send_ctrl_buff_flush - Flush control buffers, to send post */ -static int sdp_send_ctrl_buff_flush(struct sdp_opt *conn) +static int sdp_send_ctrl_buff_flush(struct sdp_sock *conn) { struct sdpc_desc *element; int result = 0; @@ -1119,7 +1119,7 @@ static int sdp_send_ctrl_buff_flush(stru /* * sdp_send_ctrl_buff_buffered - Send a buffered control message */ -static int sdp_send_ctrl_buff_buffered(struct sdp_opt *conn, +static int sdp_send_ctrl_buff_buffered(struct sdp_sock *conn, struct sdpc_buff *buff) { int result = 0; @@ -1149,7 +1149,7 @@ error: /* * sdp_send_ctrl_buff - Create and Send a buffered control message */ -static int sdp_send_ctrl_buff(struct sdp_opt *conn, u8 mid, int se, int sig) +static int sdp_send_ctrl_buff(struct sdp_sock *conn, u8 mid, int se, int sig) { int result = 0; struct sdpc_buff *buff; @@ -1201,7 +1201,7 @@ static int sdp_send_ctrl_buff(struct sdp /* * do_send_ctrl_disconnect - Send a disconnect request */ -static int do_send_ctrl_disconnect(struct sdp_opt *conn) +static int do_send_ctrl_disconnect(struct sdp_sock *conn) { int result = 0; struct sdpc_buff *buff; @@ -1247,7 +1247,7 @@ error: /* * sdp_send_ctrl_disconnect - potentially send a disconnect request */ -int sdp_send_ctrl_disconnect(struct sdp_opt *conn) +int sdp_send_ctrl_disconnect(struct sdp_sock *conn) { int result = 0; /* @@ -1270,7 +1270,7 @@ int sdp_send_ctrl_disconnect(struct sdp_ /* * sdp_send_ctrl_ack - Send a gratuitous Ack */ -int sdp_send_ctrl_ack(struct sdp_opt *conn) +int sdp_send_ctrl_ack(struct sdp_sock *conn) { /* * The gratuitous ack is not really and ack, but an update of the @@ -1292,7 +1292,7 @@ int sdp_send_ctrl_ack(struct sdp_opt *co /* * sdp_send_ctrl_send_sm - Send a request for buffered mode */ -int sdp_send_ctrl_send_sm(struct sdp_opt *conn) +int sdp_send_ctrl_send_sm(struct sdp_sock *conn) { return sdp_send_ctrl_buff(conn, SDP_MID_SEND_SM, 1, 1); } @@ -1300,7 +1300,7 @@ int sdp_send_ctrl_send_sm(struct sdp_opt /* * sdp_send_ctrl_src_cancel - Send a source cancel */ -int sdp_send_ctrl_src_cancel(struct sdp_opt *conn) +int sdp_send_ctrl_src_cancel(struct sdp_sock *conn) { return sdp_send_ctrl_buff(conn, SDP_MID_SRC_CANCEL, 1, 1); } @@ -1308,7 +1308,7 @@ int sdp_send_ctrl_src_cancel(struct sdp_ /* * sdp_send_ctrl_snk_cancel - Send a sink cancel */ -int sdp_send_ctrl_snk_cancel(struct sdp_opt *conn) +int sdp_send_ctrl_snk_cancel(struct sdp_sock *conn) { return sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL, 1, 1); } @@ -1316,7 +1316,7 @@ int sdp_send_ctrl_snk_cancel(struct sdp_ /* * sdp_send_ctrl_snk_cancel_ack - Send an ack for a sink cancel */ -int sdp_send_ctrl_snk_cancel_ack(struct sdp_opt *conn) +int sdp_send_ctrl_snk_cancel_ack(struct sdp_sock *conn) { return sdp_send_ctrl_buff(conn, SDP_MID_SNK_CANCEL_ACK, 1, 1); } @@ -1324,7 +1324,7 @@ int sdp_send_ctrl_snk_cancel_ack(struct /* * sdp_send_ctrl_abort - Send an abort message */ -int sdp_send_ctrl_abort(struct sdp_opt *conn) +int sdp_send_ctrl_abort(struct sdp_sock *conn) { /* * send @@ -1335,7 +1335,7 @@ int sdp_send_ctrl_abort(struct sdp_opt * /* * sdp_send_ctrl_resize_buff_ack - Send an ack for a buffer size change */ -int sdp_send_ctrl_resize_buff_ack(struct sdp_opt *conn, u32 size) +int sdp_send_ctrl_resize_buff_ack(struct sdp_sock *conn, u32 size) { struct msg_hdr_crbah *crbah; struct sdpc_buff *buff; @@ -1386,7 +1386,7 @@ error: /* * sdp_send_ctrl_rdma_rd - Send an rdma read completion */ -int sdp_send_ctrl_rdma_rd(struct sdp_opt *conn, s32 size) +int sdp_send_ctrl_rdma_rd(struct sdp_sock *conn, s32 size) { struct msg_hdr_rrch *rrch; struct sdpc_buff *buff; @@ -1455,7 +1455,7 @@ error: /* * sdp_send_ctrl_rdma_wr - Send an rdma write completion */ -int sdp_send_ctrl_rdma_wr(struct sdp_opt *conn, u32 size) +int sdp_send_ctrl_rdma_wr(struct sdp_sock *conn, u32 size) { struct msg_hdr_rwch *rwch; struct sdpc_buff *buff; @@ -1512,7 +1512,7 @@ error: /* * sdp_send_ctrl_snk_avail - Send a sink available message */ -int sdp_send_ctrl_snk_avail(struct sdp_opt *conn, u32 size, u32 rkey, u64 addr) +int sdp_send_ctrl_snk_avail(struct sdp_sock *conn, u32 size, u32 rkey, u64 addr) { struct msg_hdr_snkah *snkah; struct sdpc_buff *buff; @@ -1573,7 +1573,7 @@ error: /* * sdp_send_ctrl_mode_ch - Send a mode change command */ -int sdp_send_ctrl_mode_ch(struct sdp_opt *conn, u8 mode) +int sdp_send_ctrl_mode_ch(struct sdp_sock *conn, u8 mode) { struct msg_hdr_mch *mch; struct sdpc_buff *buff; @@ -1679,7 +1679,7 @@ static int sdp_write_src_lookup(struct s static int sdp_inet_write_cancel(struct kiocb *req, struct io_event *ev) { struct sock_iocb *si = kiocb_to_siocb(req); - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_iocb *iocb; int result = 0; @@ -1812,7 +1812,7 @@ done: /* * sdp_send_flush_advt - Flush passive sink advertisments */ -static int sdp_send_flush_advt(struct sdp_opt *conn) +static int sdp_send_flush_advt(struct sdp_sock *conn) { struct sdpc_advt *advt; int result; @@ -1850,7 +1850,7 @@ static int sdp_send_flush_advt(struct sd /* * sdp_send_flush - Flush buffers from send queue, in to send post */ -int sdp_send_flush(struct sdp_opt *conn) +int sdp_send_flush(struct sdp_sock *conn) { struct sock *sk; int result = 0; @@ -1914,7 +1914,7 @@ int sdp_inet_send(struct kiocb *req, str size_t size) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_buff *buff; struct sdpc_iocb *iocb; int result = 0; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -74,7 +74,7 @@ static u32 sdp_psn_generate(void) return psn; } -void sdp_conn_inet_error(struct sdp_opt *conn, int error) +void sdp_conn_inet_error(struct sdp_sock *conn, int error) { struct sock *sk; @@ -100,7 +100,7 @@ void sdp_conn_inet_error(struct sdp_opt sk->sk_error_report(sk); } -void sdp_conn_abort(struct sdp_opt *conn) +void sdp_conn_abort(struct sdp_sock *conn) { int result; int error = -ECONNRESET; @@ -176,10 +176,10 @@ void sdp_conn_abort(struct sdp_opt *conn /* * sdp_inet_accept_q_put - put a conn into a listen conn's accept Q. */ -void sdp_inet_accept_q_put(struct sdp_opt *listen_conn, - struct sdp_opt *accept_conn) +void sdp_inet_accept_q_put(struct sdp_sock *listen_conn, + struct sdp_sock *accept_conn) { - struct sdp_opt *next_conn; + struct sdp_sock *next_conn; BUG_ON(listen_conn->parent); BUG_ON(accept_conn->parent); @@ -203,10 +203,10 @@ void sdp_inet_accept_q_put(struct sdp_op /* * sdp_inet_accept_q_get - get a conn from a listen conn's accept Q. */ -struct sdp_opt *sdp_inet_accept_q_get(struct sdp_opt *listen_conn) +struct sdp_sock *sdp_inet_accept_q_get(struct sdp_sock *listen_conn) { - struct sdp_opt *prev_conn; - struct sdp_opt *accept_conn; + struct sdp_sock *prev_conn; + struct sdp_sock *accept_conn; if (listen_conn->parent || !listen_conn->accept_next || @@ -242,10 +242,10 @@ struct sdp_opt *sdp_inet_accept_q_get(st /* * sdp_inet_accept_q_remove - remove a conn from a conn's accept Q. */ -int sdp_inet_accept_q_remove(struct sdp_opt *accept_conn) +int sdp_inet_accept_q_remove(struct sdp_sock *accept_conn) { - struct sdp_opt *next_conn; - struct sdp_opt *prev_conn; + struct sdp_sock *next_conn; + struct sdp_sock *prev_conn; if (!accept_conn->parent) return -EFAULT; @@ -279,7 +279,7 @@ int sdp_inet_accept_q_remove(struct sdp_ /* * sdp_inet_listen_start - start listening for new connections on a socket */ -int sdp_inet_listen_start(struct sdp_opt *conn) +int sdp_inet_listen_start(struct sdp_sock *conn) { unsigned long flags; @@ -312,9 +312,9 @@ int sdp_inet_listen_start(struct sdp_opt /* * sdp_inet_listen_stop - stop listening for new connections on a socket */ -int sdp_inet_listen_stop(struct sdp_opt *listen_conn) +int sdp_inet_listen_stop(struct sdp_sock *listen_conn) { - struct sdp_opt *accept_conn; + struct sdp_sock *accept_conn; unsigned long flags; if (listen_conn->state != SDP_CONN_ST_LISTEN) { @@ -365,9 +365,9 @@ int sdp_inet_listen_stop(struct sdp_opt /* * sdp_inet_listen_lookup - lookup a connection in the listen list */ -struct sdp_opt *sdp_inet_listen_lookup(u32 addr, u16 port) +struct sdp_sock *sdp_inet_listen_lookup(u32 addr, u16 port) { - struct sdp_opt *conn; + struct sdp_sock *conn; unsigned long flags; /* * table lock @@ -390,11 +390,11 @@ struct sdp_opt *sdp_inet_listen_lookup(u /* * sdp_inet_port_get - bind a socket to a port. */ -int sdp_inet_port_get(struct sdp_opt *conn, u16 port) +int sdp_inet_port_get(struct sdp_sock *conn, u16 port) { struct sock *sk; struct sock *srch; - struct sdp_opt *look; + struct sdp_sock *look; s32 counter; s32 low_port; s32 top_port; @@ -512,7 +512,7 @@ done: /* * sdp_inet_port_put - unbind a socket from a port. */ -int sdp_inet_port_put(struct sdp_opt *conn) +int sdp_inet_port_put(struct sdp_sock *conn) { unsigned long flags; @@ -541,7 +541,7 @@ int sdp_inet_port_put(struct sdp_opt *co /* * sdp_inet_port_inherit - inherit a port from another socket (accept) */ -void sdp_inet_port_inherit(struct sdp_opt *parent, struct sdp_opt *child) +void sdp_inet_port_inherit(struct sdp_sock *parent, struct sdp_sock *child) { unsigned long flags; @@ -568,7 +568,7 @@ void sdp_inet_port_inherit(struct sdp_op /* * sdp_conn_table_insert - insert a connection into the connection table */ -static int sdp_conn_table_insert(struct sdp_opt *conn) +static int sdp_conn_table_insert(struct sdp_sock *conn) { u32 counter; int result = -ENOMEM; @@ -612,7 +612,7 @@ static int sdp_conn_table_insert(struct /* * sdp_conn_table_remove - remove a connection from the connection table */ -int sdp_conn_table_remove(struct sdp_opt *conn) +int sdp_conn_table_remove(struct sdp_sock *conn) { int result = 0; unsigned long flags; @@ -649,9 +649,9 @@ done: /* * sdp_conn_table_lookup - look up connection in the connection table */ -struct sdp_opt *sdp_conn_table_lookup(s32 entry) +struct sdp_sock *sdp_conn_table_lookup(s32 entry) { - struct sdp_opt *conn; + struct sdp_sock *conn; unsigned long flags; /* * lock table @@ -699,7 +699,7 @@ static void sdp_desc_q_cancel_iocb(struc } } -void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error) +void sdp_iocb_q_cancel_all_read(struct sdp_sock *conn, ssize_t error) { sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error); sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error); @@ -707,7 +707,7 @@ void sdp_iocb_q_cancel_all_read(struct s sdp_desc_q_cancel_iocb(&conn->r_src, error); } -void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error) +void sdp_iocb_q_cancel_all_write(struct sdp_sock *conn, ssize_t error) { sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error); @@ -715,7 +715,7 @@ void sdp_iocb_q_cancel_all_write(struct sdp_desc_q_cancel_iocb(&conn->w_snk, error); } -void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error) +void sdp_iocb_q_cancel_all(struct sdp_sock *conn, ssize_t error) { sdp_iocb_q_cancel_all_read(conn, error); sdp_iocb_q_cancel_all_write(conn, error); @@ -728,7 +728,7 @@ void sdp_iocb_q_cancel_all(struct sdp_op /* * sdp_conn_destruct - final destructor for connection. */ -void sdp_conn_destruct(struct sdp_opt *conn) +void sdp_conn_destruct(struct sdp_sock *conn) { int dump = 0; int result; @@ -840,7 +840,7 @@ void sdp_conn_destruct(struct sdp_opt *c /* * sdp_conn_internal_lock - lock the connection (use only from macro) */ -void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags) +void sdp_conn_internal_lock(struct sdp_sock *conn, unsigned long *flags) { DECLARE_WAITQUEUE(wait, current); unsigned long f = *flags; @@ -864,7 +864,7 @@ void sdp_conn_internal_lock(struct sdp_o /* * sdp_conn_relock - test the connection (use only from macro) */ -void sdp_conn_relock(struct sdp_opt *conn) +void sdp_conn_relock(struct sdp_sock *conn) { unsigned long flags; struct ib_wc entry; @@ -928,7 +928,7 @@ void sdp_conn_relock(struct sdp_opt *con /* * sdp_conn_cq_drain - drain one of the the connection's CQs */ -int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn) +int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_sock *conn) { struct ib_wc entry; int result; @@ -980,7 +980,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, /* * sdp_conn_internal_unlock - lock the connection (use only from macro) */ -void sdp_conn_internal_unlock(struct sdp_opt *conn) +void sdp_conn_internal_unlock(struct sdp_sock *conn) { int calls = 0; /* @@ -998,7 +998,7 @@ void sdp_conn_internal_unlock(struct sdp /* * sdp_conn_lock_init - initialize connection lock */ -static void sdp_conn_lock_init(struct sdp_opt *conn) +static void sdp_conn_lock_init(struct sdp_sock *conn) { spin_lock_init(&(conn->lock.slock)); conn->lock.users = 0; @@ -1009,7 +1009,7 @@ static void sdp_conn_lock_init(struct sd /* * sdp_conn_alloc_ib - allocate IB structures for a new connection. */ -int sdp_conn_alloc_ib(struct sdp_opt *conn, struct ib_device *device, +int sdp_conn_alloc_ib(struct sdp_sock *conn, struct ib_device *device, u8 hw_port, u16 pkey) { struct ib_qp_init_attr *init_attr; @@ -1184,9 +1184,9 @@ extern struct proto sdp_sk_proto; /* * sdp_conn_alloc - allocate a new socket, and init. */ -struct sdp_opt *sdp_conn_alloc(int priority) +struct sdp_sock *sdp_conn_alloc(int priority) { - struct sdp_opt *conn; + struct sdp_sock *conn; struct sock *sk; int result; @@ -1377,7 +1377,7 @@ error: int sdp_proc_dump_conn_main(char *buffer, int max_size, off_t start_index, long *end_index) { - struct sdp_opt *conn; + struct sdp_sock *conn; off_t counter = 0; int offset = 0; u64 s_guid; @@ -1482,7 +1482,7 @@ int sdp_proc_dump_conn_data(char *buffer long *end_index) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; off_t counter = 0; int offset = 0; unsigned long flags; @@ -1575,7 +1575,7 @@ done: int sdp_proc_dump_conn_rdma(char *buffer, int max_size, off_t start_index, long *end_index) { - struct sdp_opt *conn; + struct sdp_sock *conn; off_t counter = 0; int offset = 0; unsigned long flags; @@ -1652,7 +1652,7 @@ done: int sdp_proc_dump_conn_sopt(char *buffer, int max_size, off_t start_index, long *end_index) { - struct sdp_opt *conn; + struct sdp_sock *conn; off_t counter = 0; int offset = 0; unsigned long flags; @@ -1970,7 +1970,7 @@ int sdp_conn_table_init(int proto_family goto error_size; } - byte_size = conn_size * sizeof(struct sdp_opt *); + byte_size = conn_size * sizeof(struct sdp_sock *); page_size = (byte_size >> 12) + ((0xfff & byte_size) > 0 ? 1 : 0); for (dev_root_s.sk_ordr = 0; (1 << dev_root_s.sk_ordr) < page_size; dev_root_s.sk_ordr++) ; @@ -2042,7 +2042,7 @@ void sdp_conn_table_clear(void) { sdp_dbg_init("Deleting connection tables."); #if 0 - struct sdp_opt *conn; + struct sdp_sock *conn; /* * drain all the connections */ Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (working copy) @@ -38,7 +38,7 @@ /* * Connection establishment functions */ -void sdp_cm_actv_error(struct sdp_opt *conn, int error) +void sdp_cm_actv_error(struct sdp_sock *conn, int error) { int result; struct sock *sk; @@ -104,7 +104,7 @@ void sdp_cm_actv_error(struct sdp_opt *c /* * sdp_cm_actv_establish - process an accepted connection request. */ -static int sdp_cm_actv_establish(struct sdp_opt *conn) +static int sdp_cm_actv_establish(struct sdp_sock *conn) { struct ib_qp_attr *qp_attr; int attr_mask = 0; @@ -262,7 +262,7 @@ static int sdp_cm_hello_ack_check(struct * sdp_cm_rep_handler - handler for active connection open completion */ int sdp_cm_rep_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { struct sdp_msg_hello_ack *hello_ack; int result = -ECONNRESET; @@ -342,7 +342,7 @@ static void sdp_cm_path_complete(u64 id, { struct ib_cm_req_param param; struct sdp_msg_hello *hello_msg; - struct sdp_opt *conn = (struct sdp_opt *) arg; + struct sdp_sock *conn = (struct sdp_sock *) arg; struct sdpc_buff *buff; int result = 0; /* @@ -512,7 +512,7 @@ done: /* * sdp_cm_connect - initiate a SDP connection with a hello message. */ -int sdp_cm_connect(struct sdp_opt *conn) +int sdp_cm_connect(struct sdp_sock *conn) { int result; /* Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_post_recv_buff - post a single buffers for data recv */ -static int sdp_post_recv_buff(struct sdp_opt *conn) +static int sdp_post_recv_buff(struct sdp_sock *conn) { struct ib_recv_wr receive_param = { NULL }; struct ib_recv_wr *bad_wr; @@ -111,7 +111,7 @@ error: /* * sdp_post_rdma_buff - post a single buffers for rdma read on a conn */ -static int sdp_post_rdma_buff(struct sdp_opt *conn) +static int sdp_post_rdma_buff(struct sdp_sock *conn) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -202,7 +202,7 @@ drop: /* * sdp_post_rdma_iocb_src - post a iocb for rdma read on a conn */ -static int sdp_post_rdma_iocb_src(struct sdp_opt *conn) +static int sdp_post_rdma_iocb_src(struct sdp_sock *conn) { struct ib_send_wr send_param = { NULL }; struct ib_send_wr *bad_wr; @@ -311,7 +311,7 @@ error: /* * sdp_post_rdma_iocb_snk - post a iocb for rdma read on a conn */ -static int sdp_post_rdma_iocb_snk(struct sdp_opt *conn) +static int sdp_post_rdma_iocb_snk(struct sdp_sock *conn) { struct sdpc_iocb *iocb; int result = 0; @@ -379,7 +379,7 @@ error: /* * sdp_post_rdma - post a rdma based requests for a connection */ -static int sdp_post_rdma(struct sdp_opt *conn) +static int sdp_post_rdma(struct sdp_sock *conn) { int result = 0; @@ -458,7 +458,7 @@ done: /* * sdp_recv_flush - post a certain number of buffers on a connection */ -int sdp_recv_flush(struct sdp_opt *conn) +int sdp_recv_flush(struct sdp_sock *conn) { int result = 0; int counter; @@ -625,7 +625,7 @@ static int sdp_read_buff_iocb(struct sdp /* * sdp_recv_buff_iocb_active - Ease AIO read pending pressure */ -static int sdp_recv_buff_iocb_active(struct sdp_opt *conn, +static int sdp_recv_buff_iocb_active(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; @@ -671,7 +671,7 @@ static int sdp_recv_buff_iocb_active(str /* * sdp_recv_buff_iocb_pending - Ease AIO read pending pressure */ -static int sdp_recv_buff_iocb_pending(struct sdp_opt *conn, +static int sdp_recv_buff_iocb_pending(struct sdp_sock *conn, struct sdpc_buff *buff) { struct sdpc_iocb *iocb; @@ -725,7 +725,7 @@ static int sdp_recv_buff_iocb_pending(st /* * sdp_recv_buff - Process a new buffer based on queue type */ -int sdp_recv_buff(struct sdp_opt *conn, struct sdpc_buff *buff) +int sdp_recv_buff(struct sdp_sock *conn, struct sdpc_buff *buff) { int result; int buffered; @@ -835,7 +835,7 @@ static int sdp_read_src_lookup(struct sd static int sdp_inet_read_cancel(struct kiocb *req, struct io_event *ev) { struct sock_iocb *si = kiocb_to_siocb(req); - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_iocb *iocb; int result = 0; @@ -992,7 +992,7 @@ static int sdp_inet_recv_urg_trav(struct static int sdp_inet_recv_urg(struct sock *sk, struct msghdr *msg, int size, int flags) { - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_buff *buff; int result = 0; u8 value; @@ -1061,7 +1061,7 @@ int sdp_inet_recv(struct kiocb *req, st size_t size, int flags) { struct sock *sk; - struct sdp_opt *conn; + struct sdp_sock *conn; struct sdpc_iocb *iocb; struct sdpc_buff *buff; struct sdpc_buff *head = NULL; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -109,12 +109,12 @@ enum sdp_mode { */ #define SDP_MSG_EVENT_TABLE_SIZE 0x20 -static inline struct sdp_opt *sdp_sk(struct sock *sk) +static inline struct sdp_sock *sdp_sk(struct sock *sk) { - return (struct sdp_opt *)sk; + return (struct sdp_sock *)sk; } -static inline struct sock *sk_sdp(struct sdp_opt *conn) +static inline struct sock *sk_sdp(struct sdp_sock *conn) { return (struct sock *)conn; } @@ -176,9 +176,9 @@ struct sdp_conn_lock { /* * SDP Connection structure. */ -struct sdp_opt { +struct sdp_sock { /* - * inet_sock must be first member of sdp_opt + * inet_sock must be first member of sdp_sock * NOTE: this depends on inet_sock having struct sock as its * first member */ @@ -336,17 +336,17 @@ struct sdp_opt { /* * table managment */ - struct sdp_opt *lstn_next; /* next conn in the chain */ - struct sdp_opt **lstn_p_next; /* previous next conn in the chain */ + struct sdp_sock *lstn_next; /* next conn in the chain */ + struct sdp_sock **lstn_p_next; /* previous next conn in the chain */ - struct sdp_opt *bind_next; /* next conn in the chain */ - struct sdp_opt **bind_p_next; /* previous next conn in the chain */ + struct sdp_sock *bind_next; /* next conn in the chain */ + struct sdp_sock **bind_p_next; /* previous next conn in the chain */ /* * listen/accept managment */ - struct sdp_opt *parent; /* listening socket queuing. */ - struct sdp_opt *accept_next; /* sockets waiting for acceptance. */ - struct sdp_opt *accept_prev; /* sockets waiting for acceptance. */ + struct sdp_sock *parent; /* listening socket queuing. */ + struct sdp_sock *accept_next; /* sockets waiting for acceptance. */ + struct sdp_sock *accept_prev; /* sockets waiting for acceptance. */ /* * OS info */ @@ -427,18 +427,18 @@ struct sdp_opt { /* * SDP connection lock */ -extern void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags); -extern void sdp_conn_internal_unlock(struct sdp_opt *conn); -extern void sdp_conn_relock(struct sdp_opt *conn); -extern void sdp_conn_destruct(struct sdp_opt *conn); -extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn); +extern void sdp_conn_internal_lock(struct sdp_sock *conn, unsigned long *flags); +extern void sdp_conn_internal_unlock(struct sdp_sock *conn); +extern void sdp_conn_relock(struct sdp_sock *conn); +extern void sdp_conn_destruct(struct sdp_sock *conn); +extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_sock *conn); #define SDP_CONN_LOCK_IRQ(conn, flags) \ spin_lock_irqsave(&((conn)->lock.slock), flags) #define SDP_CONN_UNLOCK_IRQ(conn, flags) \ spin_unlock_irqrestore(&((conn)->lock.slock), flags) -static inline void sdp_conn_lock(struct sdp_opt *conn) +static inline void sdp_conn_lock(struct sdp_sock *conn) { unsigned long flags; @@ -454,7 +454,7 @@ static inline void sdp_conn_lock(struct spin_unlock_irqrestore(&(conn->lock.slock), flags); } -static inline void sdp_conn_unlock(struct sdp_opt *conn) +static inline void sdp_conn_unlock(struct sdp_sock *conn) { unsigned long flags; @@ -473,12 +473,12 @@ static inline void sdp_conn_unlock(struc /* * connection reference counting. */ -static inline void sdp_conn_hold(struct sdp_opt *conn) +static inline void sdp_conn_hold(struct sdp_sock *conn) { atomic_inc(&conn->refcnt); } -static inline void sdp_conn_put(struct sdp_opt *conn) +static inline void sdp_conn_put(struct sdp_sock *conn) { if (atomic_dec_and_test(&conn->refcnt)) sdp_conn_destruct(conn); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (working copy) @@ -38,7 +38,7 @@ /* * handle incoming passive connection establishment. (RTU) */ -int sdp_cm_pass_establish(struct sdp_opt *conn) +int sdp_cm_pass_establish(struct sdp_sock *conn) { struct ib_qp_attr *qp_attr; int attr_mask = 0; @@ -102,7 +102,7 @@ done: /* * Functions to handle incoming passive connection requests. (REQ) */ -static int sdp_cm_accept(struct sdp_opt *conn) +static int sdp_cm_accept(struct sdp_sock *conn) { struct ib_cm_rep_param param; struct sdp_msg_hello_ack *hello_ack; @@ -231,8 +231,8 @@ error: return result; } -static void sdp_cm_listen_inherit(struct sdp_opt *parent, - struct sdp_opt *child) +static void sdp_cm_listen_inherit(struct sdp_sock *parent, + struct sdp_sock *child) { struct sock *psk; struct sock *csk; @@ -350,8 +350,8 @@ static int sdp_cm_hello_check(struct sdp int sdp_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) { struct sdp_msg_hello *msg_hello = event->private_data; - struct sdp_opt *listen_conn; - struct sdp_opt *conn; + struct sdp_sock *listen_conn; + struct sdp_sock *conn; struct sock *sk; int result; u16 port; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_sent.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_sent.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_sent.c (working copy) @@ -39,7 +39,7 @@ * Specific MID handler functions. (SEND) */ -static int sdp_sent_disconnect(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_sent_disconnect(struct sdp_sock *conn, struct sdpc_buff *buff) { int result; @@ -95,7 +95,7 @@ error: return result; } -static int sdp_sent_abort(struct sdp_opt *conn, struct sdpc_buff *buff) +static int sdp_sent_abort(struct sdp_sock *conn, struct sdpc_buff *buff) { int result; @@ -115,7 +115,7 @@ static int sdp_sent_abort(struct sdp_opt /* * sdp_event_send - send event handler */ -int sdp_event_send(struct sdp_opt *conn, struct ib_wc *comp) +int sdp_event_send(struct sdp_sock *conn, struct ib_wc *comp) { struct sdpc_buff *head = NULL; struct sdpc_buff *buff; Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c (working copy) @@ -413,7 +413,7 @@ static void sdp_mem_lock_cleanup(void) /* * sdp_iocb_register - register an IOCBs memory for advertisment */ -int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_opt *conn) +int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_sock *conn) { int result; +https://openib.org/tiki/tiki-index.php?page=SDPTodoList Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_event.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_event.c (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_event.c (working copy) @@ -42,7 +42,7 @@ /* * sdp_cq_event_locked - main per QP event handler */ -int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_opt *conn) +int sdp_cq_event_locked(struct ib_wc *comp, struct sdp_sock *conn) { int result = 0; @@ -127,7 +127,7 @@ done: void sdp_cq_event_handler(struct ib_cq *cq, void *arg) { s32 hashent = (unsigned long)arg; - struct sdp_opt *conn; + struct sdp_sock *conn; u16 event; unsigned long flags; @@ -183,7 +183,7 @@ void sdp_cq_event_handler(struct ib_cq * sdp_conn_put(conn); } -static void sdp_cm_to_error(struct sdp_opt *conn) +static void sdp_cm_to_error(struct sdp_sock *conn) { sdp_conn_inet_error(conn, -ECONNRESET); conn->cm_id = NULL; @@ -194,7 +194,7 @@ static void sdp_cm_to_error(struct sdp_o */ static int sdp_cm_idle(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { sdp_dbg_ctrl(conn, "CM IDLE. commID <%08x> event <%d> status <%d>", cm_id->local_id, event->event, event->param.send_status); @@ -234,7 +234,7 @@ static int sdp_cm_idle(struct ib_cm_id * static int sdp_cm_established(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { int result = 0; @@ -292,7 +292,7 @@ error: } static int sdp_cm_dreq_rcvd(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { int result = 0; @@ -313,7 +313,7 @@ static int sdp_cm_dreq_rcvd(struct ib_cm * sdp_cm_timewait - handler for connection Time Wait completion */ static int sdp_cm_timewait(struct ib_cm_id *cm_id, struct ib_cm_event *event, - struct sdp_opt *conn) + struct sdp_sock *conn) { int result = 0; @@ -381,7 +381,7 @@ static int sdp_cm_timewait(struct ib_cm_ int sdp_cm_event_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) { s32 hashent = (unsigned long)cm_id->context; - struct sdp_opt *conn = NULL; + struct sdp_sock *conn = NULL; int result = 0; sdp_dbg_ctrl(NULL, "CM state <%d> event <%d> commID <%08x> ID <%d>", Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (revision 2751) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (working copy) @@ -182,12 +182,12 @@ struct sdev_root { u32 sk_ordr; /* order size of region. */ u32 sk_rover; /* next potential available space. */ u32 sk_entry; /* number of socket table entries. */ - struct sdp_opt **sk_array; /* array of sockets. */ + struct sdp_sock **sk_array; /* array of sockets. */ /* * connection managment */ - struct sdp_opt *listen_list; /* list of listening connections */ - struct sdp_opt *bind_list; /* connections bound to a port. */ + struct sdp_sock *listen_list; /* list of listening connections */ + struct sdp_sock *bind_list; /* connections bound to a port. */ /* * list locks */ From tduffy at sun.com Wed Jun 29 16:04:00 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 29 Jun 2005 16:04:00 -0700 Subject: [openib-general] Re: SDP: still getting sk_alloc() panic, any ideas? In-Reply-To: <20050628111913.E26240@topspin.com> References: <9d3b7de7050623130658aa92@mail.gmail.com> <20050627111721.A26240@topspin.com> <1119907675.6148.7.camel@duffman> <20050627160611.C26240@topspin.com> <20050628111913.E26240@topspin.com> Message-ID: <1120086240.31370.52.camel@duffman> On Tue, 2005-06-28 at 11:19 -0700, Libor Michalek wrote: > Here's a patch to reorder listen_lookup() in req_handler() and a few > other fixups, such as making a couple int return functions void. Great, that fixes the panic. Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Wed Jun 29 16:27:22 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 29 Jun 2005 16:27:22 -0700 Subject: [openib-general] [PATCH][RESEND] SDP: update TODO to point to wiki Message-ID: <1120087642.31370.55.camel@duffman> Signed-off-by: Tom Duffy Index: linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/TODO =================================================================== --- linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/TODO (revision 2597) +++ linux-2.6.12-rc6-openib/drivers/infiniband/ulp/sdp/TODO (working copy) @@ -1,58 +1,3 @@ -SDP +Please refer to the TODO at: - - Cosmetic cleanup as needed. - - - Reduce connection state machine complexity. Specifically there exists - redundant tracking of connection state which should be merged for - improved readability and maintainability. - - - Add full connection cleanup once Connection Manager IDLE is in place. - - - Reduce code bloat. - - - Enable FMRs once mthca support is provided. - - - Provide real support for SO_{RCV,SND}BUF - - internal posted buffers - - QP WQ size - - CQ size - - - Create slow start algorithm for posted receive buffers to improve - memory consumption in large connection count applications, especially - for connections that are predominately sending data. - - - sdp_link needs locking protection. - - - sdp_link request cancellation support. - - - SDP RDMA support for blocking socket operations. - - - IPv6 addressing - - - Alternate Path Migration (APM) support. - - - Keepalive (zero byte RDMA write) - - - Socket duplication (i.e. suspend) - -Possible Issues - - - SDP address resolution requires looking at the IPoIB device's private - data structure inorder to determine interface's SGID, Port, and PKey. - - - Memory locking for AIO requires a call to do_mlock() which is not a - kernel exported function, the method for calling the function is not - standard. - -Other - - - Kernel level transparent socket switch for dynamic TCP/SDP selection. - (Possible SDP port mapper protocol from RDMAC) - - - Extended Sockets API (ES-API). Initial Linux AIO based implemetation? - - - New SOCK_DGRAM protocol? - - - NFS over SDP. - - - QP striping? \ No newline at end of file +https://openib.org/tiki/tiki-index.php?page=SDPTodoList From sean.hefty at intel.com Wed Jun 29 16:57:17 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 29 Jun 2005 16:57:17 -0700 Subject: [openib-general] [PATCH] [CM] include packet lifetime in MRA delay Message-ID: The following patch includes the packet lifetime as part of the delay waiting for a request after receiving an MRA. Problem pointed out by Fab Tillier. Signed-off-by: Sean Hefty Index: core/cm.c =================================================================== --- core/cm.c (revision 2752) +++ core/cm.c (working copy) @@ -92,6 +92,7 @@ struct cm_av { union ib_gid dgid; struct ib_ah_attr ah_attr; u16 pkey_index; + u8 packet_life_time; }; struct cm_work { @@ -296,6 +297,7 @@ static int cm_init_av_by_path(struct ib_ av->port = port; cm_set_ah_attr(&av->ah_attr, av->port->port_num, path->dlid, path->sl, path->slid & 0x7F); + av->packet_life_time = path->packet_life_time; return 0; } @@ -2164,7 +2166,8 @@ static int cm_mra_handler(struct cm_work work->cm_event.private_data = &mra_msg->private_data; work->cm_event.param.mra_rcvd.service_timeout = cm_mra_get_service_timeout(mra_msg); - timeout = cm_convert_to_ms(cm_mra_get_service_timeout(mra_msg)); + timeout = cm_convert_to_ms(cm_mra_get_service_timeout(mra_msg)) + + cm_convert_to_ms(cm_id_priv->av.packet_life_time); spin_lock_irqsave(&cm_id_priv->lock, flags); switch (cm_id_priv->id.state) { From libor at topspin.com Wed Jun 29 17:12:21 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 29 Jun 2005 17:12:21 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <42C2E573.5010304@ichips.intel.com>; from mshefty@ichips.intel.com on Wed, Jun 29, 2005 at 11:16:19AM -0700 References: <42C1BA15.7060205@ichips.intel.com> <20050629111038.G26240@topspin.com> <42C2E573.5010304@ichips.intel.com> Message-ID: <20050629171221.I26240@topspin.com> On Wed, Jun 29, 2005 at 11:16:19AM -0700, Sean Hefty wrote: > Libor Michalek wrote: > >>Is it possible for a consumer of uCM to provide a context with the > >>create_id that could be returned with the event? I will have some scale > >>up issues if I have to walk a list looking for a uCM provided connection > >>ID instead of a context that could point directly to the appropriate > >>uDAPL CM object. > > > > It would be easy to add in a context variable. I had left it out on > > purpose, since it's easy to get into a situation where using the context > > as a pointer you can end up referencing deallocated memory. However, I > > suppose it should be there for flexability. > > Can you explain the situation where the application could reference > deallocated memory? I would think that the uCM could take steps that would > make it impossible for a well written app from doing this. Assume that the userspace 'struct ib_cm_event' contains the cm_id as well as a new 'u64 context' which is inherited from the cm_id, and is set at the time of the cm_id creation. This is what I'm assuming that Arlin would like to see. In the case of two threads accessing the CM at once there's a race condition if you are going to use the 'context' variable as a pointer to memory: Thread 1 Thread 2 ------------------------------------- ----------------------------------- cm_object = malloc(sizeof(*cm_object) ib_cm_create_id(&cm_object->cm_id, (u64)cm_object) ib_cm_event_get(&event) ib_cm_destroy_id(cm_object->cm_id) free(cm_object); process_event((void *)event->context); If you're going to insist on using threads, the context variable should provide a reference into some table which contains the cm_object and that table (e.g. lookup, add, remove) needs to be sync'd with a thread lock. -Libor From Arkady.Kanevsky at netapp.com Wed Jun 29 16:58:09 2005 From: Arkady.Kanevsky at netapp.com (Kanevsky, Arkady) Date: Wed, 29 Jun 2005 19:58:09 -0400 Subject: [openib-general] comments on DAT registry in OpenIB Message-ID: Dear DAT and OpenIB members, There is a debate going on on OpenIB and DAT reflectors which is going around about kDAT registry for Linux. I would like to review the requirements we had agreed at DAT collaborative and captured in the kDAT and uDAT specs and review DAT registry in OpenIB from that prospective. I would like to make it clear that I do NOT speak on behalf of DAT Collaborative but as just one of its members. 1. Ability of Consumers to open IA based on its name * OpenIB supports it 2. Support for Consumers to get a list of available IAs to open * OpenIB kDAT and uDAT registry provide this * OpenIB kDAT registry no longer provide Provider attributes as stated above * Preserves dat_registry_list_providers * for kDAPL OpenIB changed dat_provider_info format so binary compatibility not preserved, but source compatibility is preserved. * dat_provider_info differs between uDAPL and kDAPL in OpenIB 3. Ability to enumerate available IAs and their attributes * OpenIB supports that for uDAPL unchanged from DAPL SF RI * for kDAPL openIB supports a single type of thread-safety defined by the Linux kernel and the version of Linux kernel defines the kDAPL APIs that kernel version supports. * for kDAPL query will not return DAT version and thread safety Provider attribute. 4. Map IA_name to Provider library (kDAPL or uDAPL) * OpenIB kDAPL and uDAPL support this 5. Ability for DAT providers to dynamically register and deregister DAPL Provider * OpenIB supports that for kDAPL and uDAPL * All existing registry APIs at DAPL SF RI are preserved 6. Single static DAT registry - platform specific * kDAT and uDAT specs explicitly state that the DAT registry is defined by the platform and DAT collaborative provided an example of Registry for Linux and Windows and agree that the DAT provided registry should be used by all providers. This ensures that DAT Registry will support all Providers and DAT registry from one vendor does not block other providers. * OpenIB kDAT registry is the Linux platform DAT registry which achieves the goal of supporting all kDAPL providers. It also provides additional benefit that it is Linux core which maintain kDAT registry instead of DAT Collaborative * OpenIB uDAT registry remains the DAT collaborative one unchanged. * We can discuss whether or not we want to get uDAT registry closer to the OpenIB kDAT one * The DAT registry for kDAPL and uDAPL are different at DAPL SF RI and OpenIB maintains it. * Some changes may be needed for kDAPL Registry hot plug support for OpenIB. How it may impact uDAPL registry. 7. ia_name is under system admin control * remains the same following a platform convention 8. IA can represent 1. single port 2. several ports 3. several HBAs or RNICs 4. multiple IAs represent the same port * OpenIB kDAPL currently implements #1. Members can submit code patches to support other choices * OpenIB uDAPL remains the same with current implementation providing #1 under Provider control. 9. Support for Consumers to get a list of available IAs to open * OpenIB kDAT and uDAT registry provide this * OpenIB kDAT registry no longer provide Provider attributes as stated above 10. DAT registry supports loading multiple DAPL Providers into the same address space. 1. A Provider library loaded into an address space once 2. A Provider library unloaded only when all open instances of its IAs are closed 3. The same Provider library can be loaded into multiple address spaces * OpenIB uDAPL continues to provide it * OpenIB kDAPL supports it 11. DAT registry shall support polymorphism (Provider independency) 1. Consumer call DAT functions by the DAT handle independently from Provider is used 2. DAT registry provides redirection 3. dat_ia_open is Provider specific and sets up redirection table per address space per Provider * first time open ensures that table redirection for a Provider is set up * OpenIB kDAT and uDAT registry provide that * OpenIB kDAT registry preserves the DAT redirection table as defined by DAT Collaborative * OpenIB kDAT registry preserves DAT_provider structure * need to file errata to DAT to move dat_ia_close after dat_ia_query to match DAPL SF RI and OpenIB one for kDAPL and uDAPL 12. The DAT_handle structure first field provides a pointer for redirection * OpenIB kDAT and uDAT registry support this * DAT registry interpret the DAT_handle as a pointer to redirection table 13. DAT registry supports multiple dat_ia_open for the same library from the same and different address spaces * OpenIB DAT registry preserves that * OpenIB DAT registry supports refcount so the library is not closed until the last Consumer leaves and library is loaded into an address space once. So for uDAT Consumer and uDAT Provider which are compliant with uDAT spec continue to work as before from DAPL SF RI. kDAT Consumer no longer can choose Provider by DAT version # and thread safety. The Linux kernel version defines which version of DAT and its thread safety. Since kernel ULP is specific to the kernel configuration this is OK. If kernel module was choosing DAT Provider based on these attributes it is no longer needed. The kernel module MUST match what kernel configuration, including version number, provides. If kernel ULP behaves differently based on version of DAT API and Provider thread safety this switch will have to be replaced by kernel configuration/version switch. This is "standard" for kernel ULP modules. kDAT Provider will be effected. The dat_info structure is changed. It no longer has fields for DAT major and minor versions and no field for thread safety. DAT Provider must match kernel configuration/version DAT API and thread-safety. Others: kDAT Providers will need to maintain a separate tree for Linux than for generic DAT or DAT for other platforms with appropriate Linux kernel versioning/configuration support. For IB Providers it is not needed since OpenIB kDAT provides that. For iWARP vendors if they do not plug-in into gen2 as HW driver then they need to match the OpenIB/Linux kDAPL version DAT APIs and threading. DAT Provider registration is different for uDAPL and kDAPL. This is true at DAPL SF RI. And it is still the case for OpenIB. The biggest difference from DAT Spec Linux Registry example and OpenIB kDAT registry is a uDAPL/KDAPL registry shared configuration file that contains all available Providers information. But this is just an example. How Registry keeps the Provider info is up to a platform. Consumer can not rely on the back door access to the static DAT registry database. They should use the dat_registry_list_providers to get that info. OpenIB kDAT registry preserves that API with a format change for dat_info structure usable by Consumers. uDAT and kDAT Providers code sharing is slightly diminished. In summary, kDAT registry in OpenIB fulfils all DAT requirements. There is a small change in one of the structure which impacts kDAPL Consumers but is consistent with the way Linux kernel ULP modules operation. The biggest change for DAT Collaborative is that this body no longer "the sole Provider of DAT Registry". I view that as good news since Linux maintainers are much better source of kDAPL and uDAPL registry then DAT Collaborative. Arkady Kanevsky email: arkady at netapp.com Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 Arkady Kanevsky email: arkady at netapp.com Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 -------------- next part -------------- An HTML attachment was scrubbed... URL: From libor at topspin.com Wed Jun 29 18:10:54 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 29 Jun 2005 18:10:54 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <42C2F991.60005@ichips.intel.com>; from ardavis@ichips.intel.com on Wed, Jun 29, 2005 at 12:42:09PM -0700 References: <42C1BA15.7060205@ichips.intel.com> <20050629111038.G26240@topspin.com> <42C2F991.60005@ichips.intel.com> Message-ID: <20050629181054.J26240@topspin.com> On Wed, Jun 29, 2005 at 12:42:09PM -0700, Arlin Davis wrote: > Libor Michalek wrote: > >On Tue, Jun 28, 2005 at 01:59:01PM -0700, Arlin Davis wrote: > >> > >>I have a couple of uCM questions regarding create_id and events... > >> > >>Is it possible for a consumer of uCM to provide a context with the > >>create_id that could be returned with the event? I will have some scale > >>up issues if I have to walk a list looking for a uCM provided connection > >>ID instead of a context that could point directly to the appropriate > >>uDAPL CM object. > >> > > > > It would be easy to add in a context variable. I had left it out on > >purpose, since it's easy to get into a situation where using the context > >as a pointer you can end up referencing deallocated memory. However, I > >suppose it should be there for flexability. > > > Thanks. Also, in the case with listen's (conn_req), the event returns > the new conn_id but not the original conn_id associated with the listen. > I will need the listen conn_id to associate back to the proper listen > service point. The listen id is in the req rcvd event. (event->param.req_rcvd.listen_id) Do you mean that it is not being set correctly? > >>It would also be very helpful if the CM event could be woke from > >>user space. > > > > You mean break CM event out of it's wait? I would instead recommend that > >you call poll on the file descriptor for read readiness, and only call for > >the CM event when there's an event available. Also, if you insist on using > >threads, tou can break out of poll by sending a signal to the thread that > >is waiting on the poll, which will return -1 with an errno of EINTR. > > Fair enough. Have you tried the polling method? I am having problems > waking up on the event even though I see the following kernel message > from ucm. > > "kernel: UCM: Event. CM ID <10> event <1>" Yes, it should be working, for example I just tried this: struct pollfd ufds; int result; ufds.fd = ib_cm_get_fd(); ufds.events = POLLIN; ufds.revents = 0; *event = NULL; result = poll(&ufds, 1, 100000); if (!result) return -ETIMEDOUT; if (result < 0) return result; return ib_cm_event_get(event); -Libor From tomduffy at gmail.com Wed Jun 29 19:34:07 2005 From: tomduffy at gmail.com (Tom Duffy) Date: Wed, 29 Jun 2005 19:34:07 -0700 Subject: [openib-general] Proposal for mapping DAT_RETURN values to ERRNO values In-Reply-To: References: Message-ID: <9d3b7de7050629193430c53865@mail.gmail.com> On 6/28/05, James Lentini wrote: > > The attached PDF, prepared by Itamar Rabenstein of Mellanox, contains > a proposed mapping from DAT_RETURN values to ERRNO values. These all look very good. Thanks Itamar for putting this doc together. -tduffy From rminnich at lanl.gov Wed Jun 29 20:13:38 2005 From: rminnich at lanl.gov (Ronald G. Minnich) Date: Wed, 29 Jun 2005 21:13:38 -0600 (MDT) Subject: [openib-general] Re: [PATCH 05/16] IB uverbs: core implementation In-Reply-To: <20050629002709.GB17805@kroah.com> References: <2005628163.lUk0bfpO8VsSXUh5@cisco.com> <2005628163.jfSiMqRcI78iLMJP@cisco.com> <20050629002709.GB17805@kroah.com> Message-ID: On Tue, 28 Jun 2005, Greg KH wrote: > On Tue, Jun 28, 2005 at 04:03:43PM -0700, Roland Dreier wrote: > > +++ linux/drivers/infiniband/core/uverbs_main.c 2005-06-28 15:20:04.363963991 -0700 > > @@ -0,0 +1,708 @@ > > +/* > > + * Copyright (c) 2005 Topspin Communications. All rights reserved. > > + * Copyright (c) 2005 Cisco Systems. All rights reserved. > > + * > > + * This software is available to you under a choice of one of two > > + * licenses. You may choose to be licensed under the terms of the GNU > > + * General Public License (GPL) Version 2, available from the file > > + * COPYING in the main directory of this source tree, or the > > + * OpenIB.org BSD license below: > > Ok, I've complained about this before, but due to the fact that you are > calling EXPORT_SYMBOL_GPL() only functions in this code, the ability for > it for someone to use the BSD license on it in the future, is pretty > much impossible, right? This does seem odd. If the goal is kernel inclusion, and the kernel is GPL, seems like this license boilerplate should now change. It makes no real sense otherwise, as far as I can tell. ron From caitlinb at siliquent.com Wed Jun 29 20:19:42 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Wed, 29 Jun 2005 20:19:42 -0700 Subject: [openib-general] RE: [dat-discussions] comments on DAT registry in OpenIB Message-ID: <8508251A6FC08A489844A94261D3693A057C5E@fiona.siliquent.com> Source compatability for an existing Provider is *not* maintained by OpenIB kDAPL because there are fields *missing* from the Provider Info. That *will* result in a compilation error. It also precludes experimentation with future releases of DAT through kernel modules. This is the technique that produced the code that is being ported, so this seems a strange policy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Wed Jun 29 20:59:46 2005 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 29 Jun 2005 20:59:46 -0700 Subject: [openib-general] [BUG]driver didn't accept ARP reply packet Message-ID: I did netperf test over Mellanox 23108 4X HCA against r2720, after a while the HCA stopped to accept packets. I am pretty sure it's a bug in driver not IPoIB. rmmod ib_ipoib didn't help, after removing ib_mthca, and restarted, the driver worked again. It's easy to reprocude on my 4-way intel based systems. I did saw the below messages from /var/log/messages, when this happened. Jun 29 12:13:53 elm3a238 kernel: ib_mthca 0000:04:00.0: Port change to down for port 1 Jun 29 12:13:55 elm3a238 kernel: ib_mthca 0000:04:00.0: Port change to active for port 1 And the HCA didn't accept ARP reply packet afterwards. Now I am adding a permanent arp entry to avoid this problem. It seems to work fine. Anybody else saw this problem? Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Wed Jun 29 21:03:48 2005 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 29 Jun 2005 21:03:48 -0700 Subject: [openib-general] [BUG]driver didn't accept ARP reply packet In-Reply-To: Message-ID: >Now I am adding a permanent arp entry to avoid this problem. It seems to work fine. Oops, I was wrong, it stopped to receive any packets. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 Shirley Ma/Beaverton/IBM at IBMUS Sent by: openib-general-bounces at openib.org 06/29/2005 08:59 PM To openib-general at openib.org cc Subject [openib-general] [BUG]driver didn't accept ARP reply packet I did netperf test over Mellanox 23108 4X HCA against r2720, after a while the HCA stopped to accept packets. I am pretty sure it's a bug in driver not IPoIB. rmmod ib_ipoib didn't help, after removing ib_mthca, and restarted, the driver worked again. It's easy to reprocude on my 4-way intel based systems. I did saw the below messages from /var/log/messages, when this happened. Jun 29 12:13:53 elm3a238 kernel: ib_mthca 0000:04:00.0: Port change to down for port 1 Jun 29 12:13:55 elm3a238 kernel: ib_mthca 0000:04:00.0: Port change to active for port 1 And the HCA didn't accept ARP reply packet afterwards. Now I am adding a permanent arp entry to avoid this problem. It seems to work fine. Anybody else saw this problem? Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 _______________________________________________ openib-general mailing list openib-general at openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.j.woodruff at intel.com Wed Jun 29 21:55:01 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Wed, 29 Jun 2005 21:55:01 -0700 Subject: [openib-general] [BUG]driver didn't accept ARP reply packet Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004D1F312@orsmsx408> Shirley wrote, >Now I am adding a permanent arp entry to avoid this problem. It seems to work fine. Oops, I was wrong, it stopped to receive any packets. When this happens, if you reset the switch, does the node resume accepting packets ? I have seen a similar problem (only once or twice) where sometimes a node stops receiving IPoIB packets. I cannot ping other nodes or ping that node from another. When I then reset the switch, it seems that things started to work again. I was not able to reproduce the problem, so I thought it might be a flaky switch, but it sounds somewhat similar to the problem I saw. woody From bill at strahm.net Wed Jun 29 22:39:10 2005 From: bill at strahm.net (Bill Strahm) Date: Wed, 29 Jun 2005 22:39:10 -0700 Subject: [openib-general] Re: IP addressing on InfiniBand networks (Caitlin Bestler) In-Reply-To: <20050629162540.86C21228405@openib.ca.sandia.gov> References: <20050629162540.86C21228405@openib.ca.sandia.gov> Message-ID: <42C3857E.3060508@strahm.net> >------------------------------ > >Message: 2 >Date: Wed, 29 Jun 2005 09:00:37 -0700 >From: Roland Dreier >Subject: Re: [openib-general] IP addressing on InfiniBand networks >To: Caitlin Bestler >Cc: "Lentini, James" , Christoph Hellwig > , openib-general >Message-ID: <5264vxi256.fsf at topspin.com> >Content-Type: text/plain; charset=us-ascii > > Caitlin> An assigned GID meets all of the requirements for an IA > Caitlin> Address. I think taking advantage of that existing > Caitlin> capability is just one of many options that can be done > Caitlin> by the IB CM rather than forcing IB specific changes up > Caitlin> to the application layer. > >Just to be clear, the IBA spec is very clear that a GID _is_ an IPv6 address. > > - R. > > > Just to be REALLY clear - IANA has not allocated IPv6 address space to any Infiniband entities - so they are not Internet IPv6 addresses. GIDs are formatted like IPv6 addresses but in no sense should EVER be used at an IP layer 3 address. From an IPoIB stance - IB is just a very over engineered Layer 2 that has a singularly large MAC address. From an IB ULP point of view - how you get to the layer 2 address that is needed to perform communications that do not include IP, it isn't a problem - but let me tell you.... The leadership of the IETF is scared of IB because of saying things IB GIDs _ARE_ IPv6 addresses. Bill From xma at us.ibm.com Wed Jun 29 22:56:51 2005 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 29 Jun 2005 22:56:51 -0700 Subject: [openib-general] [BUG]driver didn't accept ARP reply packet In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F0004D1F312@orsmsx408> Message-ID: It could be a firmware problem. I used 3.3.2 firmware, the sender side had this problem. The receiver side was OK. Anyway after back to 3.2.0 firmware, it's OK now. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From hch at lst.de Thu Jun 30 03:48:45 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 30 Jun 2005 12:48:45 +0200 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: References: Message-ID: <20050630104845.GA11393@lst.de> Could you please stop that comitte crap? James, what do you think about doing an s/DAT/RDMA/ and s/dat/rdma/ on the code so we can stop this endless mess? In the end it won't look like dat anyway, and the sooner why make that absolutely clear that less idiocy like this is going to happen. From halr at voltaire.com Thu Jun 30 05:48:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 Jun 2005 08:48:02 -0400 Subject: [openib-general] [PATCH] sa_query: Add service record support Message-ID: <1120135681.4848.5971.camel@hal.voltaire.com> sa_query: Add service record support Signed-off-by: Hal Rosenstock Index: include/ib_sa.h =================================================================== --- include/ib_sa.h (revision 2757) +++ include/ib_sa.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -193,6 +194,61 @@ int proxy_join; }; +/* Service Record Component Mask Sec 15.2.5.14 Ver 1.1 */ +#define IB_SR_COMPMASK_SID IB_SA_COMP_MASK(0) +#define IB_SR_COMPMASK_SGID IB_SA_COMP_MASK(1) +#define IB_SR_COMPMASK_SPKEY IB_SA_COMP_MASK(2) +#define IB_SR_COMPMASK_RES1 IB_SA_COMP_MASK(3) +#define IB_SR_COMPMASK_SLEASE IB_SA_COMP_MASK(4) +#define IB_SR_COMPMASK_SKEY IB_SA_COMP_MASK(5) +#define IB_SR_COMPMASK_SNAME IB_SA_COMP_MASK(6) +#define IB_SR_COMPMASK_SDATA8_0 IB_SA_COMP_MASK(7) +#define IB_SR_COMPMASK_SDATA8_1 IB_SA_COMP_MASK(8) +#define IB_SR_COMPMASK_SDATA8_2 IB_SA_COMP_MASK(9) +#define IB_SR_COMPMASK_SDATA8_3 IB_SA_COMP_MASK(10) +#define IB_SR_COMPMASK_SDATA8_4 IB_SA_COMP_MASK(11) +#define IB_SR_COMPMASK_SDATA8_5 IB_SA_COMP_MASK(12) +#define IB_SR_COMPMASK_SDATA8_6 IB_SA_COMP_MASK(13) +#define IB_SR_COMPMASK_SDATA8_7 IB_SA_COMP_MASK(14) +#define IB_SR_COMPMASK_SDATA8_8 IB_SA_COMP_MASK(15) +#define IB_SR_COMPMASK_SDATA8_9 IB_SA_COMP_MASK(16) +#define IB_SR_COMPMASK_SDATA8_10 IB_SA_COMP_MASK(17) +#define IB_SR_COMPMASK_SDATA8_11 IB_SA_COMP_MASK(18) +#define IB_SR_COMPMASK_SDATA8_12 IB_SA_COMP_MASK(19) +#define IB_SR_COMPMASK_SDATA8_13 IB_SA_COMP_MASK(20) +#define IB_SR_COMPMASK_SDATA8_14 IB_SA_COMP_MASK(21) +#define IB_SR_COMPMASK_SDATA8_15 IB_SA_COMP_MASK(22) +#define IB_SR_COMPMASK_SDATA16_0 IB_SA_COMP_MASK(23) +#define IB_SR_COMPMASK_SDATA16_1 IB_SA_COMP_MASK(24) +#define IB_SR_COMPMASK_SDATA16_2 IB_SA_COMP_MASK(25) +#define IB_SR_COMPMASK_SDATA16_3 IB_SA_COMP_MASK(26) +#define IB_SR_COMPMASK_SDATA16_4 IB_SA_COMP_MASK(27) +#define IB_SR_COMPMASK_SDATA16_5 IB_SA_COMP_MASK(28) +#define IB_SR_COMPMASK_SDATA16_6 IB_SA_COMP_MASK(29) +#define IB_SR_COMPMASK_SDATA16_7 IB_SA_COMP_MASK(30) +#define IB_SR_COMPMASK_SDATA32_0 IB_SA_COMP_MASK(31) +#define IB_SR_COMPMASK_SDATA32_1 IB_SA_COMP_MASK(32) +#define IB_SR_COMPMASK_SDATA32_2 IB_SA_COMP_MASK(33) +#define IB_SR_COMPMASK_SDATA32_3 IB_SA_COMP_MASK(34) +#define IB_SR_COMPMASK_SDATA64_0 IB_SA_COMP_MASK(35) +#define IB_SR_COMPMASK_SDATA64_1 IB_SA_COMP_MASK(36) + +#define IB_DEFAULT_SERVICE_LEASE 0xFFFFFFFF + +struct ib_sa_service_rec { + u64 id; + union ib_gid gid; + u16 pkey; + /* u16 resv; */ + u32 lease; + u8 key[16]; + u8 name[64]; + u8 data8[16]; + u16 data16[8]; + u32 data32[4]; + u64 data64[2]; +}; + struct ib_sa_query; void ib_sa_cancel_query(int id, struct ib_sa_query *query); @@ -218,6 +274,17 @@ void *context, struct ib_sa_query **query); +int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, + u8 method, + struct ib_sa_service_rec *rec, + ib_sa_comp_mask comp_mask, + int timeout_ms, int gfp_mask, + void (*callback)(int status, + struct ib_sa_service_rec *resp, + void *context), + void *context, + struct ib_sa_query **sa_query); + /** * ib_sa_mcmember_rec_set - Start an MCMember set query * @device:device to send query on Index: core/sa_query.c =================================================================== --- core/sa_query.c (revision 2757) +++ core/sa_query.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -79,6 +80,12 @@ int id; }; +struct ib_sa_service_query { + void (*callback)(int, struct ib_sa_service_rec *, void *); + void *context; + struct ib_sa_query sa_query; +}; + struct ib_sa_path_query { void (*callback)(int, struct ib_sa_path_rec *, void *); void *context; @@ -320,6 +327,54 @@ .size_bits = 23 }, }; +#define SERVICE_REC_FIELD(field) \ + .struct_offset_bytes = offsetof(struct ib_sa_service_rec, field), \ + .struct_size_bytes = sizeof ((struct ib_sa_service_rec *) 0)->field, \ + .field_name = "sa_service_rec:" #field + +static const struct ib_field service_rec_table[] = { + { SERVICE_REC_FIELD(id), + .offset_words = 0, + .offset_bits = 0, + .size_bits = 64 }, + { SERVICE_REC_FIELD(gid), + .offset_words = 2, + .offset_bits = 0, + .size_bits = 128 }, + { SERVICE_REC_FIELD(pkey), + .offset_words = 6, + .offset_bits = 0, + .size_bits = 16 }, + { SERVICE_REC_FIELD(lease), + .offset_words = 7, + .offset_bits = 0, + .size_bits = 32 }, + { SERVICE_REC_FIELD(key), + .offset_words = 8, + .offset_bits = 0, + .size_bits = 128 }, + { SERVICE_REC_FIELD(name), + .offset_words = 12, + .offset_bits = 0, + .size_bits = 64*8 }, + { SERVICE_REC_FIELD(data8), + .offset_words = 28, + .offset_bits = 0, + .size_bits = 16*8 }, + { SERVICE_REC_FIELD(data16), + .offset_words = 32, + .offset_bits = 0, + .size_bits = 8*16 }, + { SERVICE_REC_FIELD(data32), + .offset_words = 36, + .offset_bits = 0, + .size_bits = 4*32 }, + { SERVICE_REC_FIELD(data64), + .offset_words = 40, + .offset_bits = 0, + .size_bits = 2*64 }, +}; + static void free_sm_ah(struct kref *kref) { struct ib_sa_sm_ah *sm_ah = container_of(kref, struct ib_sa_sm_ah, ref); @@ -443,7 +498,6 @@ .remote_qpn = 1, .remote_qkey = IB_QP1_QKEY, .timeout_ms = timeout_ms, - .retries = 0 } } }; @@ -596,6 +650,114 @@ } EXPORT_SYMBOL(ib_sa_path_rec_get); +static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query, + int status, + struct ib_sa_mad *mad) +{ + struct ib_sa_service_query *query = + container_of(sa_query, struct ib_sa_service_query, sa_query); + + if (mad) { + struct ib_sa_service_rec rec; + + ib_unpack(service_rec_table, ARRAY_SIZE(service_rec_table), + mad->data, &rec); + query->callback(status, &rec, query->context); + } else + query->callback(status, NULL, query->context); +} + +static void ib_sa_service_rec_release(struct ib_sa_query *sa_query) +{ + kfree(sa_query->mad); + kfree(container_of(sa_query, struct ib_sa_service_query, sa_query)); +} + +/** + * ib_sa_service_rec_query - Start Service Record operation + * @device:device to send request on + * @port_num: port number to send request on + * @method:SA method - should be get, set, or delete + * @rec:Service Record to send in request + * @comp_mask:component mask to send in request + * @timeout_ms:time to wait for response + * @gfp_mask:GFP mask to use for internal allocations + * @callback:function called when request completes, times out or is + * canceled + * @context:opaque user context passed to callback + * @sa_query:request context, used to cancel request + * + * Send a Service Record set/get/delete to the SA to register, + * unregister or query a service record. + * The callback function will be called when the request completes (or + * fails); status is 0 for a successful response, -EINTR if the query + * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error + * occurred sending the query. The resp parameter of the callback is + * only valid if status is 0. + * + * If the return value of ib_sa_service_rec_query() is negative, it is an + * error code. Otherwise it is a request ID that can be used to cancel + * the query. + */ +int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method, + struct ib_sa_service_rec *rec, + ib_sa_comp_mask comp_mask, + int timeout_ms, int gfp_mask, + void (*callback)(int status, + struct ib_sa_service_rec *resp, + void *context), + void *context, + struct ib_sa_query **sa_query) +{ + struct ib_sa_service_query *query; + struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client); + struct ib_sa_port *port = &sa_dev->port[port_num - sa_dev->start_port]; + struct ib_mad_agent *agent = port->agent; + int ret; + + if (method != IB_MGMT_METHOD_GET && + method != IB_MGMT_METHOD_SET && + method != IB_SA_METHOD_DELETE) + return -EINVAL; + + query = kmalloc(sizeof *query, gfp_mask); + if (!query) + return -ENOMEM; + query->sa_query.mad = kmalloc(sizeof *query->sa_query.mad, gfp_mask); + if (!query->sa_query.mad) { + kfree(query); + return -ENOMEM; + } + + query->callback = callback; + query->context = context; + + init_mad(query->sa_query.mad, agent); + + query->sa_query.callback = ib_sa_service_rec_callback; + query->sa_query.release = ib_sa_service_rec_release; + query->sa_query.port = port; + query->sa_query.mad->mad_hdr.method = method; + query->sa_query.mad->mad_hdr.attr_id = + cpu_to_be16(IB_SA_ATTR_SERVICE_REC); + query->sa_query.mad->sa_hdr.comp_mask = comp_mask; + + ib_pack(service_rec_table, ARRAY_SIZE(service_rec_table), + rec, query->sa_query.mad->data); + + *sa_query = &query->sa_query; + + ret = send_mad(&query->sa_query, timeout_ms); + if (ret < 0) { + *sa_query = NULL; + kfree(query->sa_query.mad); + kfree(query); + } + + return ret; +} +EXPORT_SYMBOL(ib_sa_service_rec_query); + static void ib_sa_mcmember_rec_callback(struct ib_sa_query *sa_query, int status, struct ib_sa_mad *mad) From arne.redlich at xiranet.com Thu Jun 30 06:51:49 2005 From: arne.redlich at xiranet.com (Arne Redlich) Date: Thu, 30 Jun 2005 15:51:49 +0200 Subject: [openib-general] SDP sk_data_ready() callback In-Reply-To: <20050628153238.F26240@topspin.com> References: <1119429583.8208.28.camel@confield> <20050627113521.B26240@topspin.com> <1119942921.8348.21.camel@confield> <20050628153238.F26240@topspin.com> Message-ID: <1120139509.7867.7.camel@confield> Am Dienstag, den 28.06.2005, 15:32 -0700 schrieb Libor Michalek: > On Tue, Jun 28, 2005 at 09:15:21AM +0200, Arne Redlich wrote: > > Am Montag, den 27.06.2005, 11:35 -0700 schrieb Libor Michalek: > > > On Wed, Jun 22, 2005 at 10:39:43AM +0200, Arne Redlich wrote: > > > > Hi, > > > > > > > > I'm trying to use SDP from within the kernel. My problem is that the > > > > code relies on sk_data_ready() (this callback is modified to wake up a > > > > Rx thread before executing the original function), but sk_data_ready() > > > > apparently never gets called. Is there any way to fix this? > > > > > > You mean that you replace sk->sk_data_ready, with a similar but > > > slightly modified version and so rely on sk->sk_data_ready() being > > > called? > > > > Yes, that's right. Actually the modifications are just minor as I > > already pointed out: the replacement function just wakes the thread and > > then calls the original sk_data_ready(). > > > > > You're right it's currently not being called, instead we're calling > > > the function to which it's pointing directly, and for no real reason. > > > Also, the function that's being called is a duplicate of the function > > > sock_def_readable() in net/sock.c. I'll look at correcting these > > > issues. > > Here's a patch which uses the socket readiness function pointers, > as well as using the default readiness functions. Except for write > readiness, (sk->sk_write_space) which currently uses different info > to determine write space. > > -Libor Thanks a lot, it's working now as expected. Arne -- Arne Redlich Xiranet Communications GmbH From jlentini at netapp.com Thu Jun 30 07:04:36 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 30 Jun 2005 10:04:36 -0400 (EDT) Subject: [openib-general] Re: [PATCH][dapl] cleanup dapl_cookie In-Reply-To: <20050629215046.GB17011@aon.at> References: <20050627173951.GB19279@aon.at> <20050629215046.GB17011@aon.at> Message-ID: On Wed, 29 Jun 2005, Bernhard Fischer wrote: > On Tue, Jun 28, 2005 at 03:39:59PM -0400, James Lentini wrote: >> >> Hi Bernhard, >> >> The changes look fine. Why the additional copyright? I need to be able >> to explain it to my legal department. > > My legaleeze states that whatever i do during work-time is contributed > to work and whatever is related to work done during leasure time has to > be attributed to /me _at_ _least_. As that snippet (which was a test-balloon > for that category) clearly was done in my spare time, i'm forced to > attribute it accordingly :-/ > > Does that answer your question satisfactorily? Thanks Bernhard. That makes sense to me. My legal inquired about the "all rights reserved" qualifier. All the copyrights I found in the OpenIB tree (including NetApp's) use that language. I'll run this by them. From jlentini at netapp.com Thu Jun 30 07:44:38 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 30 Jun 2005 10:44:38 -0400 (EDT) Subject: [openib-general] Proposal for mapping DAT_RETURN values to ERRNO values In-Reply-To: <9d3b7de7050629193430c53865@mail.gmail.com> References: <9d3b7de7050629193430c53865@mail.gmail.com> Message-ID: On Wed, 29 Jun 2005, Tom Duffy wrote: > On 6/28/05, James Lentini wrote: >> >> The attached PDF, prepared by Itamar Rabenstein of Mellanox, contains >> a proposed mapping from DAT_RETURN values to ERRNO values. > > These all look very good. Thanks Itamar for putting this doc together. Thanks for the feedback Tom. I'm going to be on vacation next week. I'll make the conversion when I return on the 11th. If anyone else has comments, please send them by July 10th. From caitlin.bestler at gmail.com Thu Jun 30 08:38:45 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 08:38:45 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <20050630104845.GA11393@lst.de> References: <20050630104845.GA11393@lst.de> Message-ID: <469958e0050630083852187df1@mail.gmail.com> Taking an interface because it has a user base, and then ignoring that user base is just plain idiotic. If you want to design your own RDMA interface do so. But changing DAT to meet your whims rather than actual code requirements makes no sense. If you can't come up with something that remains acceptable to the broader community of DAT users then you should refrain from using the "dat_" symbols and their already established meanings. When there *is* a requirement, the DAT community is willing to work out a solution that will be acceptable to Linux *and* developers for other operating systems. On 6/30/05, Christoph Hellwig wrote: > Could you please stop that comitte crap? > > James, what do you think about doing an s/DAT/RDMA/ and s/dat/rdma/ > on the code so we can stop this endless mess? In the end it won't > look like dat anyway, and the sooner why make that absolutely clear > that less idiocy like this is going to happen. > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From tduffy at sun.com Thu Jun 30 08:40:12 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 30 Jun 2005 08:40:12 -0700 Subject: [openib-general] RE: [dat-discussions] comments on DAT registry in OpenIB In-Reply-To: <8508251A6FC08A489844A94261D3693A057C5E@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A057C5E@fiona.siliquent.com> Message-ID: <1120146012.358.2.camel@duffman> On Wed, 2005-06-29 at 20:19 -0700, Caitlin Bestler wrote: > Source compatability for an existing Provider is *not* maintained by > OpenIB > kDAPL because there are fields *missing* from the Provider Info. That > *will* > result in a compilation error. I have no problem with there being a header file with allows for source compatibility. This header would be maintained by OpenIB, but never included in upstream. I think James has a start of such a header called kdat.h. It needs a bunch more to make it follow the DAT conventions. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From hch at lst.de Thu Jun 30 08:43:57 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 30 Jun 2005 17:43:57 +0200 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <469958e0050630083852187df1@mail.gmail.com> References: <20050630104845.GA11393@lst.de> <469958e0050630083852187df1@mail.gmail.com> Message-ID: <20050630154357.GA16179@lst.de> On Thu, Jun 30, 2005 at 08:38:45AM -0700, Caitlin Bestler wrote: > actual code requirements makes no sense. If you > can't come up with something that remains acceptable > to the broader community of DAT users then you should > refrain from using the "dat_" symbols and their already > established meanings. That's exactly what I proposed. Please read the mail again. From caitlin.bestler at gmail.com Thu Jun 30 08:44:44 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 08:44:44 -0700 Subject: [openib-general] Re: IP addressing on InfiniBand networks (Caitlin Bestler) In-Reply-To: <42C3857E.3060508@strahm.net> References: <20050629162540.86C21228405@openib.ca.sandia.gov> <42C3857E.3060508@strahm.net> Message-ID: <469958e005063008447d4a47f5@mail.gmail.com> That is the primary distinction between an "IA Address" and an "IP Address". An "IA Address" meets all of the *local* characteristics of an IP Address, but does not necessarily meet the *global* characteristics of an IP Address (namely it might not have been assigned by IANA). But there is nothing that says than a network administrator cannot assign a network ID properly granted by IANA to an IP network implemented on InfiniBand. In fact doing so would make it easy to implement an IP/IB bridge/router. In context, I am not advocating that RDMA interfaces require IANA registered addresses. I am merely pointing out that the IPv6 *format* is adequate for all needs, and in fact it is only one of several methods that allows the Connection Management function to properly identify the remote peer's L3 Address in IPv6 format. On 6/29/05, Bill Strahm wrote: > > >------------------------------ > > > >Message: 2 > >Date: Wed, 29 Jun 2005 09:00:37 -0700 > >From: Roland Dreier > >Subject: Re: [openib-general] IP addressing on InfiniBand networks > >To: Caitlin Bestler > >Cc: "Lentini, James" , Christoph Hellwig > > , openib-general > >Message-ID: <5264vxi256.fsf at topspin.com> > >Content-Type: text/plain; charset=us-ascii > > > > Caitlin> An assigned GID meets all of the requirements for an IA > > Caitlin> Address. I think taking advantage of that existing > > Caitlin> capability is just one of many options that can be done > > Caitlin> by the IB CM rather than forcing IB specific changes up > > Caitlin> to the application layer. > > > >Just to be clear, the IBA spec is very clear that a GID _is_ an IPv6 address. > > > > - R. > > > > > > > Just to be REALLY clear - IANA has not allocated IPv6 address space to > any Infiniband entities - so they are not Internet IPv6 addresses. GIDs > are formatted like IPv6 addresses but in no sense should EVER be used at > an IP layer 3 address. > > From an IPoIB stance - IB is just a very over engineered Layer 2 that > has a singularly large MAC address. > > From an IB ULP point of view - how you get to the layer 2 address that > is needed to perform communications that do not include IP, it isn't a > problem - but let me tell you.... The leadership of the IETF is scared > of IB because of saying things IB GIDs _ARE_ IPv6 addresses. > > Bill > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hch at lst.de Thu Jun 30 08:46:51 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 30 Jun 2005 17:46:51 +0200 Subject: [openib-general] RE: [dat-discussions] comments on DAT registry in OpenIB In-Reply-To: <1120146012.358.2.camel@duffman> References: <8508251A6FC08A489844A94261D3693A057C5E@fiona.siliquent.com> <1120146012.358.2.camel@duffman> Message-ID: <20050630154651.GA16223@lst.de> On Thu, Jun 30, 2005 at 08:40:12AM -0700, Tom Duffy wrote: > I have no problem with there being a header file with allows for source > compatibility. This header would be maintained by OpenIB, but never > included in upstream. I think James has a start of such a header called > kdat.h. It needs a bunch more to make it follow the DAT conventions. I don't think it makes a lot of sense to maintain that header. If you want to evolve the dat code into something that's acceptable as an RDMA layer for kernel inclusion it will need to change a lot, and that changes will be rather fundamental to some of the design principles. That doesn't mean you can't add that as a DAT provider again if you care enough, but that'll be much more than a simple header file. From hch at lst.de Thu Jun 30 08:47:53 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 30 Jun 2005 17:47:53 +0200 Subject: [openib-general] RE: [dat-discussions] comments on DAT registry in OpenIB In-Reply-To: <8508251A6FC08A489844A94261D3693A057C5E@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A057C5E@fiona.siliquent.com> Message-ID: <20050630154753.GA16305@lst.de> And please everyone stop Cc'ing the subscribers only yahoogroups thing, the bounce message is totally anoying. From caitlinb at siliquent.com Thu Jun 30 08:47:53 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 08:47:53 -0700 Subject: [openib-general] RE: [dat-discussions] comments on DATregistry in OpenIB Message-ID: <8508251A6FC08A489844A94261D3693A057C70@fiona.siliquent.com> That is a reasonable migration plan, as long as rebuild is the only step required after fetching and placing the kdat.h file. > -----Original Message----- > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Thursday, June 30, 2005 8:40 AM > To: Caitlin Bestler > Cc: dat-discussions at yahoogroups.com; openib-general > Subject: Re: [openib-general] RE: [dat-discussions] comments > on DATregistry in OpenIB > > On Wed, 2005-06-29 at 20:19 -0700, Caitlin Bestler wrote: > > Source compatability for an existing Provider is *not* > maintained by > > OpenIB kDAPL because there are fields *missing* from the Provider > > Info. That > > *will* > > result in a compilation error. > > I have no problem with there being a header file with allows > for source compatibility. This header would be maintained by > OpenIB, but never included in upstream. I think James has a > start of such a header called kdat.h. It needs a bunch more > to make it follow the DAT conventions. > > -tduffy > > From David.Brean at Sun.COM Wed Jun 29 18:37:56 2005 From: David.Brean at Sun.COM (David M. Brean) Date: Wed, 29 Jun 2005 21:37:56 -0400 Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: References: Message-ID: <42C34CF4.70209@sun.com> Hello, Comments inline. James Lentini wrote: > > I'd like to summarize the discussion we've been having around > addressing and start a new email thread with a more appropriate title. > > First off, here is there requirement we are trying to satisfy: > > kDAPL consumers use an Internet Protocol (IP) addresses to > identify remote nodes in an interoperable way. > > On the active side of a connection, an InfiniBand kDAPL provider must > determine the IP address corresponding IB address (see the > dat_ep_connect() function). > > On the passive side of a connection, a InfiniBand kDAPL provider must > determine a source IB address for an InfiniBand connection request. > This information can be obtain by a kDAPL consumer either in the > DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or > dat_cr_query()'s dat_cr_param structure. > > By interoperable, we mean that the solution must not introduce a > non-standard protocol or force ULPs using kDAPL to perform special > operations when using an InfiniBand network. > > The first sentence of this requirement is the most important. This is > to best support the existing practice of ULPs that use IP addresses: > NFS, iSCSI, and and sockets applications. Administrators expect to be > able to configure systems using IP addresses. > > Note that this feature will be used. There has been considerable > discussion on how NFS-RDMA software would make use of this feature. > Other ULPs may use this feature as well. > > Of the two mappings, the second (IB address to IP address) has proven > the most difficult to implement. Here are the different > implementations that have been proposed and a brief critique of each: > > + CM Private Data > > The active side of an IB connection could place its source IP > address in the CM's private data. The passive side would retrieve > the source IP from this location. > > Analysis: This approach introduces a new protocol that is not > hidden from the ULP. The problem becomes where in the software > stack this would be implemented. If it was implemented in the DAT > provider, kDAPL consumers would not be able to communicate fully > with non-kDAPL consumers (the non-kDAPL consumers would not > be expecting the IP address and thus interpret the private data > incorrectly). If this were implemented in a ULP, the ULP would > not be able to use the same code for both an IB interface and an > RNIC interface. > I think the DAT provider should fill in the CM private-data area with the source IP address (and a bit of additional) information. So, ULPs aren't participating in this aspect of IB communication. Also, it should be possible to have a DAT ULP communicate with a peer that isn't implemented using DAT as long as the location and format of the source IP address (and additional) information in the CM private-data area is specified. For example, there is a draft "iSER on IB" document that specifies the format of additional information sent in the CM private-data area. Similar document could be written for NFS using RDMA and other ULPs. [Perhaps there should be an IBA defined structure for this purpose.] For ULPs written for DAT, this DAT provider does this for the ULP. > The security of this is very week. An end node could easily present > a false IP address. > If the DAT provider is responsible for providing this information where DAT is used, the security is no worse than IP. > + Address Translation Service (ATS) > The ATS proposal doesn't describe how to select the P_Key that must be specified when installing the SA record. In cases where IPoIB is running on the IB subnet and the IP address is bound to an IPoIB link interface, it's possible to query the IP stack to determine the P_Key associated with the IP address. In cases where IPoIB isn't used on the IB subnet, it's not clear where the IP address and P_Key come from. [I've been told that one solution is to use the IP address bound to an ethernet port and install an SA entry for every IB partition that the node can access based on the contents of its local P_Key table in the HCA.] If an IB fabric chooses not to run IPoIB, perhaps this is a scenario where the IB GID should be used as the IPv6 address as others have suggested. As a result, ATS is not needed. ATS also requires a mechanism on each IB node that monitors the status of IP addresses and updates the SA state. > Each IB node places a record containing its IP address and IB > address in the SA database. kDAPL consults these records to > map between addresses. > > Analysis: This requires all IB nodes to implement a new > protocol/adhere to a new convention. The advantage is that ULPs do > not need to be aware of it. > > Since a GID can have multiple IP address, there may be multiple > records with the same GID. The passive side would need a convention > for assuming which of these matches to the active side's IP. > > The security of this solution is also fairly weak. The end nodes > must be trusted to place valid records in the SA. Since the > configuration of IP addresses will occur on the end nodes, it would > be difficult for the SA to validate the records. However, the SA is > a central part of the network, so it may be possible to add additional > security features if needed. > It is possible to have the SA authenticate any IB client attempting to install an entry in the SA. The SA record is protected by a Service Key. By default, the Service Key value is probably zero - no protection. However, the IB administrator can assign a value to the Service Key associated with a Service Name. ATS specifies a well-known Service Name of "DAPL Address Translation Service", so the ATS specification could describe steps needed to assign a Service Key value for it. Also, this Service Key would need to be securely distributed to all the IB clients that have authority to enter SA records with the Service Name for ATS. -David > + IPoIB > > IPoIB encapsulates IP packets in InfiniBand messages. There have been > proposals to use the address resolution mechanisms in IPoIB to > implement these features. IPv4 subnets use ARP and IPv6 subnets use > Neighbor Discovery. > > Analysis: > > IPoIB is not free. All nodes would be need to implement it for > this to work. > > The IB address -> IP address mapping on the passive side is > problematic. If a reverse lookup were available, IPoIB would require > both a GID and QP number as input. The passive side would know the GID > but the QP number. > > Further more, reverse lookup is not well supported. On IPv4 subnets, > RARP is quickly becoming (already?) obsolete. Neighbor Discovery > doesn't support reverse lookup at all. [RFC 2461] > > In addition to all this, IPoIB restricts an IP subnet to the same > scope > as an IB subnet. If a kDAPL consumer desired to communicate between > IB subnet's, IPoIB may not be sufficient. > > + GID as an IPv6 Address > > See the attachment to Caitlin Bestler's email: > > http://openib.org/pipermail/openib-general/2005-June/008104.html > > Analysis: > > This has been the least discussed option. One issue is > that GIDs may not be easy to administer. GIDs can be specific > to a particular channel adapter since they can contain EUI-64 > identifiers. Administrators avoid using Ethernet MAC addresses > in configuration files and they should be able to avoid using > adapter specific IB addresses as well. Another issue is how > dynamically assigned SM GIDs would be managed. > > Are there other implementations? Is there a way to more tightly > integrate IP addressing with InfiniBand? From caitlinb at siliquent.com Thu Jun 30 08:52:54 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 08:52:54 -0700 Subject: [openib-general] Making gen2 transport neutral Message-ID: <8508251A6FC08A489844A94261D3693A057C72@fiona.siliquent.com> This is a quick outline of the changes required to make ib_verbs.h transport neutral. Some of the changes were actually required anyway to fully support IB 1.2. Most verbs are nominally unchanged. The differences are in the exact data formats and in some semantic details. These changes will require wide-spread, but minor, changes to a lot of existing code. No structural or design changes are required by these changes. structs: typically "struct ib_xyz" is transformed as follows: struct ib_xyz { /* Only IB specific fields remain */ /* In some cases fields have been split, because * iWARP allows two things to vary that IB had * locked together. SGE limits are the primary * example of this. iWARP can have different * limits on SGE size for each type of message */ }; struct iwarp_xyz { /* equivalent iWARP specific fields */ }; struct rdma_xyz { /* Transport neutral fields. Typically * a subset of what was in struct ib_xyz before */ union { struct ib_xyz ib; struct iwarp_xyz iwarp; } xpt; }; enums: Transport neutral values are first and start RDMA_ That is followed by IB specific values starting with IB_ and finally iWARP specific values starting with IWARP_ Fields are consisered 'transport neutral' only if there is a near total match on their semantics. This results in some cases of seemingly redundant error codes or event statuses, but the intent is to match what is documented in existing transport specific verb specifications, and not create questions about border cases. Creating transport neutral errors and states is to be left to the DAPl/IT-API layer. Some specific issues that need discussion that I have not yet proposed a solution to: a) What is an iWARP "port"? It probably matches an IT-API spigot, which probably matches an Ethernet device. But what is the preferred representation? The primary IP address? the device name? A pointer to a kernel structure? b) Where should page vs. block mode be set? c) Support for shared memory regions. Should we simply match RNIC-PI here and make the consumer and verbs provider responsible for this? Or should there be device independent support? d) Connection Management, and any polymorphism to the interface will be discussed later. e) iWARP does not have address handles, and "UD" service is provided via UDP which does not typically use QPs and in particular does not use the same CQs as RC. This must be resolved. I am assuming that returning a "not supported" error for irrelevant verbs is an acceptable burnden for all providers. iWARP providers do not support "query_gid" for example. Work request formats to support Bind, Local Invalidate and Fast Memory Register have been added. IB vendors would have to reject them if they do not support them as normal work requests. An mr_allocate verb is added to create an empty memory region that can be fast-registered. A new verb is proposed to create a Type (narrow) Memory Window. This could also be done by adding a type to the existing create memory window verb. New verbs are added for physical registration with fixed page sizes to match the RDMAC verbs. Cross support by interpretation of the other format would have to be discussed, it probably should be encouraged except when it requires creation of intermediate data structures. More text is still needed to deal with semantics associated with the interfaces: Different meaning of RDMA Write completion for iWARP. IN/OUT conventions on LKEY/RKEY to deal with "consumer owned portion" Error conditions that may be reported differently, such as when you supply a bad remote address. These are mostly for the applications themselves, they don't really impact the verb and DAPL/IT-API code much. Further postings will deal with remaining structural differences between the gen2 verbs and RNIC-PI and RDMAC verbs. There will also be follow up postings for each resource class (rdma device, PD, MR, MW, QP, CQ, etc.) discussing what was classified as transport neutral versus transport dependent and why. And now the diff file in-line -------------------------- Index: ib_verbs.h =================================================================== --- ib_verbs.h (revision 2744) +++ ib_verbs.h (working copy) @@ -61,24 +61,44 @@ IB_NODE_ROUTER }; -enum ib_device_cap_flags { - IB_DEVICE_RESIZE_MAX_WR = 1, - IB_DEVICE_BAD_PKEY_CNTR = (1<<1), - IB_DEVICE_BAD_QKEY_CNTR = (1<<2), - IB_DEVICE_RAW_MULTI = (1<<3), - IB_DEVICE_AUTO_PATH_MIG = (1<<4), - IB_DEVICE_CHANGE_PHY_PORT = (1<<5), - IB_DEVICE_UD_AV_PORT_ENFORCE = (1<<6), - IB_DEVICE_CURR_QP_STATE_MOD = (1<<7), - IB_DEVICE_SHUTDOWN_PORT = (1<<8), - IB_DEVICE_INIT_TYPE = (1<<9), - IB_DEVICE_PORT_ACTIVE_EVENT = (1<<10), - IB_DEVICE_SYS_IMAGE_GUID = (1<<11), - IB_DEVICE_RC_RNR_NAK_GEN = (1<<12), - IB_DEVICE_SRQ_RESIZE = (1<<13), - IB_DEVICE_N_NOTIFY_CQ = (1<<14), +enum rdma_device_cap_flags { + RDMA_DEVICE_RESIZE_MAX_WR = 1, + RDMA_DEVICE_SRQ_RESIZE = (1<<1), + RDMA_DEVICE_QP_RESIZE = (1<<2), + RDMA_DEVICE_EXT_SRQ = (1<<3), + RDMA_DEVICE_EXT_BMM = (1<<4), + RDMA_DEVICE_EXT_ZBVA = (1<<5), + RDMA_DEVICE_EXT_LIF = (1<<6), + + /* IB specific capabilities */ + IB_DEVICE_BAD_PKEY_CNTR = (1<<6), + IB_DEVICE_BAD_QKEY_CNTR = (1<<7), + IB_DEVICE_RAW_MULTI = (1<<8), + IB_DEVICE_AUTO_PATH_MIG = (1<<9), + IB_DEVICE_CHANGE_PHY_PORT = (1<<10), + IB_DEVICE_UD_AV_PORT_ENFORCE = (1<<11), + IB_DEVICE_CURR_QP_STATE_MOD = (1<<12), + IB_DEVICE_SHUTDOWN_PORT = (1<<13), + IB_DEVICE_INIT_TYPE = (1<<14), + IB_DEVICE_PORT_ACTIVE_EVENT = (1<<15), + IB_DEVICE_SYS_IMAGE_GUID = (1<<16), + IB_DEVICE_RC_RNR_NAK_GEN = (1<<17), + IB_DEVICE_N_NOTIFY_CQ = (1<<18), + + /* iWARP specific capabilities */ + IWARP_DEVICE_CQ_OVERFLOW_DETECTED = (1<<19), + IWARP_DEVICE_MOD_IRD = (1<<20), + IWARP_DEVICE_INC_ORD = (1<<21), + IWARP_DEVICE_SRQ_DEQUEUE_ARRIVAL = (1<<22), + IWARP_DEVICE_TCP_SUPPORTED = (1<<23), + IWARP_DEVICE_SCTP_SUPPORTED = (1<<24), + IWARP_DEVICE_IETF_PERMISSIVE = (1<<25), + IWARP_DEVICE_PREFER_MARKERS = (1<<26), + IWARP_DEVICE_PREFER_CRC = (1<<27), + IWARP_DEVICE_IS_IETF = (1<<28) }; + enum ib_atomic_cap { IB_ATOMIC_NONE, IB_ATOMIC_HCA, @@ -86,23 +106,9 @@ }; struct ib_device_attr { - u64 fw_ver; u64 node_guid; u64 sys_image_guid; - u64 max_mr_size; - u64 page_size_cap; - u32 vendor_id; - u32 vendor_part_id; - u32 hw_ver; - int max_qp; - int max_qp_wr; - int device_cap_flags; - int max_sge; int max_sge_rd; - int max_cq; - int max_cqe; - int max_mr; - int max_pd; int max_qp_rd_atom; int max_ee_rd_atom; int max_res_rd_atom; @@ -120,13 +126,53 @@ int max_ah; int max_fmr; int max_map_per_fmr; - int max_srq; - int max_srq_wr; - int max_srq_sge; u16 max_pkeys; u8 local_ca_ack_delay; }; +struct iwarp_device_attr { + int empty; +}; + +enum rdma_type { + RDMA_IB, + RDMA_IWARP +}; + +struct rdma_device_attr { + enum rdma_type rdma_type; + u64 fw_ver; + u64 max_mr_size; + u64 page_size_cap; + u32 vendor_id; + u32 vendor_part_id; + u32 hw_ver; + unsigned max_qp; + unsigned max_qp_wr; + enum rdma_device_cap_flag device_cap_flags; + unsigned max_sge; + unsigned max_sge_write; + unsigned max_sge_read; + unsigned max_sge_recv; + unsigned max_cq; + unsigned max_cqe; + unsigned max_mr; + unsigned max_pd; + unsigned max_srq; + unsigned max_srq_wr; + unsigned max_srq_sge; + u32 max_phys_buf_entries; + u32 max_ird; + u32 max_ird_per_qp; + u32 max_ord; + u32 max_ord_per_qp; + union { + struct ib_device_attr ib; + struct iwarp_device_attr iwarp; + } xpt; +}; + + enum ib_mtu { IB_MTU_256 = 1, IB_MTU_512 = 2, @@ -221,11 +267,21 @@ u8 phys_state; }; -enum ib_device_modify_flags { +struct iwarp_port_attr { + int tbd; +}; + +union rdma_port_attr { + struct ib_port_attr ib; + struct iwarp_port_attr iwarp; +}; + + +enum rdma_device_modify_flags { IB_DEVICE_MODIFY_SYS_IMAGE_GUID = 1 }; -struct ib_device_modify { +struct rdma_device_modify { u64 sys_image_guid; }; @@ -241,7 +297,14 @@ u8 init_type; }; -enum ib_event_type { +enum rdma_event_type { + RDMA_EVENT_QP_ERR_FROM_SRQ, + RDMA_EVENT_SRQ_LIMIT_REACHED, + RDMA_EVENT_SRQ_CATASTROPHIC, + RDMA_EVENT_QP_RQ_LIMIT_REACHED, + RDMA_EVENT_LAST_WQE_REACHED, + + /* IB Specific */ IB_EVENT_CQ_ERR, IB_EVENT_QP_FATAL, IB_EVENT_QP_REQ_ERR, @@ -255,22 +318,60 @@ IB_EVENT_PORT_ERR, IB_EVENT_LID_CHANGE, IB_EVENT_PKEY_CHANGE, - IB_EVENT_SM_CHANGE + IB_EVENT_SM_CHANGAE, + + /* iWARP Specific */ + IWARP_EVENT_LLP_CLOSE_COMPLETE, + IWARP_TERMINATE_MESSAGE_RECEIVED, + IWARP_LLP_CONNECTION_RESET, + IWARP_LLP,CONNECTION_LOST, + IWARP_LLP_INTEGRITY_ERROR_SIZE, + IWARP_LLP_INTEGRITY_ERROR_CRC, + IWARP_LLP_INTEGRITY_ERROR_BAD_FPDU, + IWARP_EVENT_ROE_DDP_VERSION, + IWARP_EVENT_ROE_RDMA_VERSION, + IWARP_EVENT_ROE_OPCODE, + IWARP_EVENT_ROE_DDP_QN, + IWARP_EVENT_ROE_RDMA_READ_DISABLED, + IWARP_EVENT_ROE_RDMA_WRITE_DISABLED, + IWARP_EVENT_ROE_RDMA_READ_INVALID, + IWARP_EVENT_ROE_NO_L_BIT, + IWARP_EVENT_PE_STAG_QP_MISMATCH, + IWARP_EVENT_PE_BOUNDS_VIOLATION, + IWARP_EVENT_PE_ACCESS_VIOLATION, + IWARP_EVENT_PE_INVALID_PD, + IWARP_EVENT_PE_WRAP_ERROR, + IWARP_EVENT_BAD_CLOSE, + IWARP_EVENT_BAD_LLP_CLOSE, + IWARP_EVENT_RQ_PE_INVALID_MSN, + IWARP_EVENT_RQ_PE_MSN_GAP, + IWARP_EVNET_IRQ_PE_TOOMANY_RDMA_READS, + IWARP_EVENT_IRQ_PE_MSN_GAP, + IWARP_EVENT_IRQ_PE_INVALID_MSN, + IWARP_EVENT_IRQ_PE_INVALID_STAG, + IWARP_EVENT_IRQ_PE_BOUNDS_VIOLATION, + IWARP_EVENT_IRQ_PE_ACCESS_VIOLATION, + IWARP_EVENT_IRQ_PE_INVALID_PD, + IWARP_EVNET_IRQ_PE_WRAP_ERROR, + IWARP_EVENT_CQ_SQ_ERR, + IWARP_EVENT_CQ_RQ_ERR, + IWARP_EVENT_CQ_OVERFLOW, + IWARP_EVENT_CQ_OP_ERR }; -struct ib_event { - struct ib_device *device; +struct rdma_event { + struct rdma_device *device; union { - struct ib_cq *cq; - struct ib_qp *qp; + struct rdma_cq *cq; + struct rdma_qp *qp; u8 port_num; } element; - enum ib_event_type event; + enum rdma_event_type event; }; -struct ib_event_handler { - struct ib_device *device; - void (*handler)(struct ib_event_handler *, struct ib_event *); +struct rdma_event_handler { + struct rdma_device *device; + void (*handler)(struct rdma_event_handler *, struct rdma_event *); struct list_head list; }; @@ -316,13 +417,15 @@ u8 port_num; }; -enum ib_wc_status { - IB_WC_SUCCESS, - IB_WC_LOC_LEN_ERR, - IB_WC_LOC_QP_OP_ERR, +enum rdma_wc_status { + RDMA_WC_SUCCESS, + RDMA_WC_LOC_LEN_ERR, + RDMA_WC_LOC_QP_OP_ERR, + RDMA_WC_LOC_PROT_ERR, + RDMA_WC_WR_FLUSH_ERR, + + /* IB Specific */ IB_WC_LOC_EEC_OP_ERR, - IB_WC_LOC_PROT_ERR, - IB_WC_WR_FLUSH_ERR, IB_WC_MW_BIND_ERR, IB_WC_BAD_RESP_ERR, IB_WC_LOC_ACCESS_ERR, @@ -338,21 +441,49 @@ IB_WC_INV_EEC_STATE_ERR, IB_WC_FATAL_ERR, IB_WC_RESP_TIMEOUT_ERR, - IB_WC_GENERAL_ERR + IB_WC_GENERAL_ERRA, + + /* iWARP Specific */ + IWARP_WC_INVALID_STAG_ERR, + IWARP_WC_WQE_FORMAT_ERR, + IWARP_WC_REMOTE_TERM_ERR, + IWARP_WC_INVALID_PD_ERR, + IWARP_WC_ADDR_WRAP_ERR, + IWARP_WC_ZERO_ORD_ERR, + IWARP_WC_QP_NOT_PRIV_ERR, + IWARP_WC_STAG_NOT_INVALID_ERR, + IWARP_WC_PAGE_SIZE_ERR, + IWARP_WC_BLOCK_SIZE_ERR, + IWARP_WC_PBL_ENTRY_ERR, + IWARP_WC_FBO_ERR, + IWARP_WC_FMR_LEN_ERR, + IWARP_WC_INV_BIND_MR_ERR, + IWARP_WC_INV_BIND_MW_ERR, + IWARP_WC_HUGE_PAYLOAD_ERR }; -enum ib_wc_opcode { - IB_WC_SEND, - IB_WC_RDMA_WRITE, - IB_WC_RDMA_READ, +enum rdma_wc_opcode { + RDMA_WC_SEND, + RDMA_WC_RDMA_WRITE, + RDMA_WC_RDMA_READ, + RDMA_WC_BIND1, + RDMA_WC_BIND2, + RDMA_WC_FMR, + RDMA_WC_LOC_INV, + + /* IB Specific */ IB_WC_COMP_SWAP, IB_WC_FETCH_ADD, - IB_WC_BIND_MW, -/* + + /* iWARP Specific */ + IWARP_WC_RDMA_READ_LOC_INV, +/* Transport neutral again: + * * Set value of IB_WC_RECV so consumers can test if a completion is a * receive by testing (opcode & IB_WC_RECV). */ IB_WC_RECV = 1 << 7, + /* IB Specific */ IB_WC_RECV_RDMA_WITH_IMM }; @@ -361,12 +492,20 @@ IB_WC_WITH_IMM = (1<<1) }; -struct ib_wc { +struct rdma_wc_common { u64 wr_id; - enum ib_wc_status status; - enum ib_wc_opcode opcode; + enum rdma_wc_status status; + enum rdma_wc_opcode opcode; u32 vendor_err; u32 byte_len; +}; + +struct iwarp_wc { + struct rdma_wc_common common; +}; + +struct ib_wc { + struct rdma_wc_common common; __be32 imm_data; u32 qp_num; u32 src_qp; @@ -378,15 +517,24 @@ u8 port_num; /* valid only for DR SMPs on switches */ }; -enum ib_cq_notify { - IB_CQ_SOLICITED, - IB_CQ_NEXT_COMP +union rdma_wc { + struct ib_wc ib; + struct iwarp_wc iwarp; + struct rdma_wc_common common; }; -struct ib_qp_cap { + +enum rdma_cq_notify { + RDMA_CQ_SOLICITED, + RDMA_CQ_NEXT_COMP +}; + +struct rdma_qp_cap { u32 max_send_wr; u32 max_recv_wr; u32 max_send_sge; + u32 max_write_sge; + u32 max_read_sge; u32 max_recv_sge; u32 max_inline_data; }; @@ -396,7 +544,11 @@ IB_SIGNAL_REQ_WR }; -enum ib_qp_type { +enum rdma_qp_type { + /* There is no such thing as a 'transport neutral' QP, + * Any instantiated QP belongs to an actual concrete transport + */ + /* IB Specific */ /* * IB_QPT_SMI and IB_QPT_GSI have to be the first two entries * here (and in that order) since the MAD layer uses them as @@ -409,18 +561,21 @@ IB_QPT_UC, IB_QPT_UD, IB_QPT_RAW_IPV6, - IB_QPT_RAW_ETY + IB_QPT_RAW_ETY, + + /* iWARP Specific */ + IWARP_QPT_RC }; -struct ib_qp_init_attr { - void (*event_handler)(struct ib_event *, void *); +struct rdma_qp_init_attr { + void (*event_handler)(struct rdma_event *, void *); void *qp_context; - struct ib_cq *send_cq; - struct ib_cq *recv_cq; - struct ib_srq *srq; - struct ib_qp_cap cap; - enum ib_sig_type sq_sig_type; - enum ib_qp_type qp_type; + struct rdma_cq *send_cq; + struct rdma_cq *recv_cq; + struct rdma_srq *srq; + struct rdma_qp_cap cap; + enum rdma_sig_type sq_sig_type; + enum rdma_qp_type qp_type; u8 port_num; /* special QP types only */ }; @@ -483,14 +638,20 @@ IB_QP_DEST_QPN = (1<<20) }; -enum ib_qp_state { - IB_QPS_RESET, +enum rdma_qp_state { + RDMA_QPS_RESET, + RDMA_QPS_RTS, + RDMA_QPS_ERR, + + /* IB Specific */ IB_QPS_INIT, IB_QPS_RTR, - IB_QPS_RTS, IB_QPS_SQD, IB_QPS_SQE, - IB_QPS_ERR + + /* iWARP Specific */ + IWARP_QPS_TERMINATE, + IWARP_QPS_CLOSING }; enum ib_mig_state { @@ -500,16 +661,13 @@ }; struct ib_qp_attr { - enum ib_qp_state qp_state; - enum ib_qp_state cur_qp_state; + enum rdma_qp_state cur_qp_state; enum ib_mtu path_mtu; enum ib_mig_state path_mig_state; u32 qkey; u32 rq_psn; u32 sq_psn; u32 dest_qp_num; - int qp_access_flags; - struct ib_qp_cap cap; struct ib_ah_attr ah_attr; struct ib_ah_attr alt_ah_attr; u16 pkey_index; @@ -527,35 +685,63 @@ u8 alt_timeout; }; -enum ib_wr_opcode { - IB_WR_RDMA_WRITE, +struct iwarp_qp_attr { + int empty; +}; + +struct rdma_qp_attr { + enum rdma_qp_state qp_state; + int qp_access_flags; + struct rdma_qp_cap cap; + union { + struct ib_qp_attr ib; + struct iwarp_qp_attr iwarp; + } xpt; +}; + +enum rdma_wr_opcode { + RDMA_WR_RDMA_WRITE, + RDMA_WR_SEND, + RDMA_WR_SEND_WITH_INV, + RDMA_WR_RDMA_READ, + RDMA_WR_BIND1, + RDMA_WR_BIND2, + RDMA_WR_FMR, + RDMA_WR_LOC_INV_MR, + RDMA_WR_LOC_INV_MW, + + /* IB Specific */ + IB_WR_SEND_WITH_IMM, IB_WR_RDMA_WRITE_WITH_IMM, - IB_WR_SEND, - IB_WR_SEND_WITH_IMM, - IB_WR_RDMA_READ, IB_WR_ATOMIC_CMP_AND_SWP, - IB_WR_ATOMIC_FETCH_AND_ADD + IB_WR_ATOMIC_FETCH_AND_ADD, + + /* iWARP specific */ + IWARP_WR_RDMA_READ_INV }; -enum ib_send_flags { - IB_SEND_FENCE = 1, - IB_SEND_SIGNALED = (1<<1), - IB_SEND_SOLICITED = (1<<2), - IB_SEND_INLINE = (1<<3) +enum rdma_send_flags { + RDMA_SEND_FENCE = 1, + RDMA_SEND_SIGNALED = (1<<1), + RDMA_SEND_SOLICITED = (1<<2), + RDMA_SEND_INLINE = (1<<3), + + /* iWARP Only */ + IWARP_SEND_READ_FENCE = (1<<4) }; -struct ib_sge { +struct rdma_sge { u64 addr; u32 length; - u32 lkey; + u32 lkey; /* can be rkey for RDMA Read on iWARP */ }; -struct ib_send_wr { - struct ib_send_wr *next; +struct rdma_send_wr { + struct rdma_send_wr *next; u64 wr_id; - struct ib_sge *sg_list; + struct rdma_sge *sg_list; int num_sge; - enum ib_wr_opcode opcode; + enum rdma_wr_opcode opcode; int send_flags; __be32 imm_data; union { @@ -579,22 +765,53 @@ u16 pkey_index; /* valid for GSI only */ u8 port_num; /* valid for DR SMPs on switch only */ } ud; + struct { + struct rdma_mr *mr; + struct rdma_mw *mw; + u32 rkey; + u64 vaddr; + u32 len; + enum rdma_mem_attr mem_attr; + } bind; + struct { + struct rdma_mr *mr; + u32 lkey; + u32 rkey; + } loc_inv_mr; + struct { + struct rdma_mw *mw; + u32 rkey; + } loc_inv_mw; + struct { + struct rdma_mr *mr; + u32 lkey; + u32 rkey; + u64 *phys_buf_list; + u32 pble_size; + u32 nbufs; + u32 addr_offset; + } fmr; } wr; }; -struct ib_recv_wr { - struct ib_recv_wr *next; +struct rdma_recv_wr { + struct rdma_recv_wr *next; u64 wr_id; - struct ib_sge *sg_list; - int num_sge; + struct rdma_sge *sg_list; + unsigned num_sge; }; -enum ib_access_flags { - IB_ACCESS_LOCAL_WRITE = 1, - IB_ACCESS_REMOTE_WRITE = (1<<1), - IB_ACCESS_REMOTE_READ = (1<<2), - IB_ACCESS_REMOTE_ATOMIC = (1<<3), - IB_ACCESS_MW_BIND = (1<<4) +enum rdma_access_flags { + RDMA_ACCESS_LOCAL_WRITE = 1, + RDMA_ACCESS_REMOTE_WRITE = (1<<1), + RDMA_ACCESS_REMOTE_READ = (1<<2), + RDMA_ACCESS_MW_BIND = (1<<3), + + /* IB specific */ + IB_ACCESS_REMOTE_ATOMIC = (1 << 4), + + /* iWARP Specific */ + IWARP_ACCESS_LOCAL_READ = (1 << 5) }; struct ib_phys_buf { @@ -602,28 +819,28 @@ u64 size; }; -struct ib_mr_attr { - struct ib_pd *pd; +struct rdma_mr_attr { + struct rdma_pd *pd; u64 device_virt_addr; u64 size; - int mr_access_flags; + enum rdma_access_flags mr_access_flags; u32 lkey; u32 rkey; }; -enum ib_mr_rereg_flags { - IB_MR_REREG_TRANS = 1, - IB_MR_REREG_PD = (1<<1), - IB_MR_REREG_ACCESS = (1<<2) +enum rdma_mr_rereg_flags { + RDMA_MR_REREG_TRANS = 1, + RDMA_MR_REREG_PD = (1<<1), + RDMA_MR_REREG_ACCESS = (1<<2) }; -struct ib_mw_bind { - struct ib_mr *mr; +struct rdma_mw_bind { + struct rdma_mr *mr; u64 wr_id; u64 addr; u32 length; - int send_flags; - int mw_access_flags; + enum rdma_send_flags send_flags; + enum rdma_access_flags access_flags; }; struct ib_fmr_attr { @@ -632,8 +849,8 @@ u8 page_size; }; -struct ib_ucontext { - struct ib_device *device; +struct rdma_ucontext { + struct rdma_device *device; struct list_head pd_list; struct list_head mr_list; struct list_head mw_list; @@ -644,14 +861,14 @@ spinlock_t lock; }; -struct ib_uobject { +struct rdma_uobject { u64 user_handle; /* handle given to us by userspace */ - struct ib_ucontext *context; /* associated user context */ + struct rdma_ucontext *context; /* associated user context */ struct list_head list; /* link to context's list */ u32 id; /* index into kernel idr */ }; -struct ib_umem { +struct rdma_umem { unsigned long user_base; unsigned long virt_base; size_t length; @@ -661,14 +878,14 @@ struct list_head chunk_list; }; -struct ib_umem_chunk { +struct rdma_umem_chunk { struct list_head list; int nents; int nmap; struct scatterlist page_list[0]; }; -struct ib_udata { +struct rdma_udata { void __user *inbuf; void __user *outbuf; size_t inlen; @@ -676,79 +893,79 @@ }; #define IB_UMEM_MAX_PAGE_CHUNK \ - ((PAGE_SIZE - offsetof(struct ib_umem_chunk, page_list)) / \ - ((void *) &((struct ib_umem_chunk *) 0)->page_list[1] - \ - (void *) &((struct ib_umem_chunk *) 0)->page_list[0])) + ((PAGE_SIZE - offsetof(struct rdma_umem_chunk, page_list)) / \ + ((void *) &((struct rdma_umem_chunk *) 0)->page_list[1] - \ + (void *) &((struct rdma_umem_chunk *) 0)->page_list[0])) -struct ib_umem_object { - struct ib_uobject uobject; - struct ib_umem umem; +struct rdma_umem_object { + struct rdma_uobject uobject; + struct rdma_umem umem; }; -struct ib_pd { - struct ib_device *device; - struct ib_uobject *uobject; +struct rdma_pd { + struct rdma_device *device; + struct rdma_uobject *uobject; atomic_t usecnt; /* count all resources */ }; struct ib_ah { - struct ib_device *device; - struct ib_pd *pd; - struct ib_uobject *uobject; + struct rdma_device *device; + struct rdma_pd *pd; + struct rdma_uobject *uobject; }; -typedef void (*ib_comp_handler)(struct ib_cq *cq, void *cq_context); +typedef void (*rdma_comp_handler)(struct rdma_cq *cq, void *cq_context); -struct ib_cq { - struct ib_device *device; - struct ib_uobject *uobject; - ib_comp_handler comp_handler; - void (*event_handler)(struct ib_event *, void *); +struct rdma_cq { + struct rdma_device *device; + struct rdma_uobject *uobject; + rdma_comp_hanlder comp_handler; + void (*event_handler)(struct rdma_event *, void *); void * cq_context; int cqe; atomic_t usecnt; /* count number of work queues */ }; -struct ib_srq { - struct ib_device *device; - struct ib_uobject *uobject; - struct ib_pd *pd; +struct rdma_srq { + struct rdma_device *device; + struct rdma_uobject *uobject; + struct rdma_pd *pd; void *srq_context; atomic_t usecnt; }; -struct ib_qp { - struct ib_device *device; - struct ib_pd *pd; - struct ib_cq *send_cq; - struct ib_cq *recv_cq; - struct ib_srq *srq; - struct ib_uobject *uobject; - void (*event_handler)(struct ib_event *, void *); +struct rdma_qp { + struct rdma_device *device; + struct rdma_pd *pd; + struct rdma_cq *send_cq; + struct rdma_cq *recv_cq; + struct rdma_srq *srq; + struct rdma_uobject *uobject; + void (*event_handler)(struct rdma_event *, void *); void *qp_context; - u32 qp_num; - enum ib_qp_type qp_type; + u32 qp_num; + enum rdma_qp_type qp_type; }; -struct ib_mr { - struct ib_device *device; - struct ib_pd *pd; - struct ib_uobject *uobject; +struct rdma_mr { + struct rdma_device *device; + struct rdma_pd *pd; + struct rdma_uobject *uobject; u32 lkey; u32 rkey; atomic_t usecnt; /* count number of MWs */ }; -struct ib_mw { - struct ib_device *device; - struct ib_pd *pd; - struct ib_uobject *uobject; +struct rdma_mw { + struct rdma_device *device; + struct rdma_pd *pd; + struct rdma_uobject *uobject; u32 rkey; }; struct ib_fmr { - struct ib_device *device; - struct ib_pd *pd; + struct rdma_device *device; + struct rdma_pd *pd; struct list_head list; u32 lkey; u32 rkey; @@ -770,19 +987,19 @@ IB_MAD_RESULT_CONSUMED = 1 << 2 /* Packet consumed: stop processing */ }; -#define IB_DEVICE_NAME_MAX 64 +#define RDMA_DEVICE_NAME_MAX 64 struct ib_cache { rwlock_t lock; - struct ib_event_handler event_handler; + struct rdma_event_handler event_handler; struct ib_pkey_cache **pkey_cache; struct ib_gid_cache **gid_cache; }; -struct ib_device { +struct rdma_device { struct device *dma_device; - char name[IB_DEVICE_NAME_MAX]; + char name[RDMA_DEVICE_NAME_MAX]; struct list_head event_handler_list; spinlock_t event_handler_lock; @@ -795,94 +1012,116 @@ u32 flags; - int (*query_device)(struct ib_device *device, - struct ib_device_attr *device_attr); - int (*query_port)(struct ib_device *device, + int (*query_device)(struct rdma_device *device, + struct rdma_device_attr *device_attr); + int (*query_port)(struct rdma_device *device, u8 port_num, - struct ib_port_attr *port_attr); - int (*query_gid)(struct ib_device *device, + union rdma_port_attr *port_attr); + int (*query_gid)(struct rdma_device *device, u8 port_num, int index, union ib_gid *gid); - int (*query_pkey)(struct ib_device *device, + int (*query_pkey)(struct rdma_device *device, u8 port_num, u16 index, u16 *pkey); - int (*modify_device)(struct ib_device *device, + int (*modify_device)(struct rdma_device *device, int device_modify_mask, - struct ib_device_modify *device_modify); - int (*modify_port)(struct ib_device *device, + struct rdma_device_modify *device_modify); + int (*modify_port)(struct rdma_device *device, u8 port_num, int port_modify_mask, struct ib_port_modify *port_modify); - struct ib_ucontext * (*alloc_ucontext)(struct ib_device *device, - struct ib_udata *udata); - int (*dealloc_ucontext)(struct ib_ucontext *context); - int (*mmap)(struct ib_ucontext *context, + struct rdma_ucontext * (*alloc_ucontext)(struct rdma_device *device, + struct rdma_udata *udata); + int (*dealloc_ucontext)(struct rdma_ucontext *context); + int (*mmap)(struct rdma_ucontext *context, struct vm_area_struct *vma); - struct ib_pd * (*alloc_pd)(struct ib_device *device, - struct ib_ucontext *context, - struct ib_udata *udata); - int (*dealloc_pd)(struct ib_pd *pd); - struct ib_ah * (*create_ah)(struct ib_pd *pd, + struct rdma_pd * (*alloc_pd)(struct rdma_device *device, + struct rdma_ucontext *context, + struct rdma_udata *udata); + int (*dealloc_pd)(struct rdma_pd *pd); + struct ib_ah * (*create_ah)(struct rdma_pd *pd, struct ib_ah_attr *ah_attr); int (*modify_ah)(struct ib_ah *ah, struct ib_ah_attr *ah_attr); int (*query_ah)(struct ib_ah *ah, struct ib_ah_attr *ah_attr); int (*destroy_ah)(struct ib_ah *ah); - struct ib_qp * (*create_qp)(struct ib_pd *pd, - struct ib_qp_init_attr *qp_init_attr, - struct ib_udata *udata); - int (*modify_qp)(struct ib_qp *qp, - struct ib_qp_attr *qp_attr, + struct rdma_qp * (*create_qp)(struct rdma_pd *pd, + struct rdma_qp_init_attr *qp_init_attr, + struct rdma_udata *udata); + int (*modify_qp)(struct rdma_qp *qp, + struct rdma_qp_attr *qp_attr, int qp_attr_mask); - int (*query_qp)(struct ib_qp *qp, - struct ib_qp_attr *qp_attr, + int (*query_qp)(struct rdma_qp *qp, + struct rdma_qp_attr *qp_attr, int qp_attr_mask, - struct ib_qp_init_attr *qp_init_attr); - int (*destroy_qp)(struct ib_qp *qp); - int (*post_send)(struct ib_qp *qp, - struct ib_send_wr *send_wr, - struct ib_send_wr **bad_send_wr); - int (*post_recv)(struct ib_qp *qp, - struct ib_recv_wr *recv_wr, - struct ib_recv_wr **bad_recv_wr); - struct ib_cq * (*create_cq)(struct ib_device *device, int cqe, - struct ib_ucontext *context, - struct ib_udata *udata); - int (*destroy_cq)(struct ib_cq *cq); - int (*resize_cq)(struct ib_cq *cq, int *cqe); - int (*poll_cq)(struct ib_cq *cq, int num_entries, - struct ib_wc *wc); - int (*peek_cq)(struct ib_cq *cq, int wc_cnt); - int (*req_notify_cq)(struct ib_cq *cq, + struct rdma_qp_init_attr *qp_init_attr); + int (*destroy_qp)(struct rdma_qp *qp); + int (*post_send)(struct rdma_qp *qp, + struct rdma_send_wr *send_wr, + struct rdma_send_wr **bad_send_wr); + int (*post_recv)(struct rdma_qp *qp, + struct rdma_recv_wr *recv_wr, + struct rdma_recv_wr **bad_recv_wr); + struct rdma_cq * (*create_cq)(struct rdma_device *device, int cqe, + struct rdma_ucontext *context, + struct rdma_udata *udata); + int (*destroy_cq)(struct rdma_cq *cq); + int (*resize_cq)(struct rdma_cq *cq, int *cqe); + int (*poll_cq)(struct rdma_cq *cq, int num_entries, + struct rdma_wc *wc); + int (*peek_cq)(struct rdma_cq *cq, int wc_cnt); + int (*req_notify_cq)(struct rdma_cq *cq, enum ib_cq_notify cq_notify); - int (*req_ncomp_notif)(struct ib_cq *cq, + int (*req_ncomp_notif)(struct rdma_cq *cq, int wc_cnt); - struct ib_mr * (*get_dma_mr)(struct ib_pd *pd, + struct rdma_mr * (*alloc_mr)(struct rdma_pd *pd, + enum rdma_access_flags mr_access_flags, + u32 *addr_list_len); + struct rdma_mr * (*get_dma_mr)(struct rdma_pd *pd, int mr_access_flags); - struct ib_mr * (*reg_phys_mr)(struct ib_pd *pd, + struct rdma_mr * (*reg_phys_mr)(struct rdma_pd *pd, struct ib_phys_buf *phys_buf_array, int num_phys_buf, int mr_access_flags, u64 *iova_start); - struct ib_mr * (*reg_user_mr)(struct ib_pd *pd, - struct ib_umem *region, + struct rdma_mr * (*reg_phys_mr_fixed)(struct rdma_pd *pd, + u64 *phys_buf_array, + unsigned num_phys_buf, + unsigned pble_size, + enum rdma_access_flags mr_access_flags, + u64 *iova_start); + struct rdma_mr * (*reg_user_mr)(struct rdma_pd *pd, + struct rdma_umem *region, int mr_access_flags, - struct ib_udata *udata); - int (*query_mr)(struct ib_mr *mr, - struct ib_mr_attr *mr_attr); - int (*dereg_mr)(struct ib_mr *mr); - int (*rereg_phys_mr)(struct ib_mr *mr, + struct rdma_udata *udata); + struct rdma_mr * (*reg_shared_mr)(struct rdma_pd *pd, + struct rdma_mr *existing_mr, + int mr_access_flags, + u64 *iova_start); + int (*query_mr)(struct rdma_mr *mr, + struct rdma_mr_attr *mr_attr); + int (*dereg_mr)(struct rdma_mr *mr); + int (*rereg_phys_mr)(struct rdma_mr *mr, int mr_rereg_mask, - struct ib_pd *pd, + struct rdma_pd *pd, struct ib_phys_buf *phys_buf_array, int num_phys_buf, int mr_access_flags, u64 *iova_start); - struct ib_mw * (*alloc_mw)(struct ib_pd *pd); - int (*bind_mw)(struct ib_qp *qp, - struct ib_mw *mw, - struct ib_mw_bind *mw_bind); - int (*dealloc_mw)(struct ib_mw *mw); - struct ib_fmr * (*alloc_fmr)(struct ib_pd *pd, + int (*rereg_phys_mr_fixed)(struct rdma_mr *mr, + int mr_rereg_mask, + struct rdma_pd *pd, + u64 *phys_buf_array, + int num_phys_buf, + u32 pble_size, + enum rdma_access_flags mr_access_flags, + u64 *iova_start); + struct rdma_mw * (*alloc_mw)(struct rdma_pd *pd); + struct rdma_mw * (*alloc_mw2)(struct rdma_pd *pd); + int (*bind_mw)(struct rdma_qp *qp, + struct rdma_mw *mw, + struct rdma_mw_bind *mw_bind); + int (*dealloc_mw)(struct rdma_mw *mw); + struct ib_fmr * (*alloc_fmr)(struct rdma_pd *pd, int mr_access_flags, struct ib_fmr_attr *fmr_attr); int (*map_phys_fmr)(struct ib_fmr *fmr, @@ -890,13 +1129,13 @@ u64 iova); int (*unmap_fmr)(struct list_head *fmr_list); int (*dealloc_fmr)(struct ib_fmr *fmr); - int (*attach_mcast)(struct ib_qp *qp, + int (*attach_mcast)(struct rdma_qp *qp, union ib_gid *gid, u16 lid); - int (*detach_mcast)(struct ib_qp *qp, + int (*detach_mcast)(struct rdma_qp *qp, union ib_gid *gid, u16 lid); - int (*process_mad)(struct ib_device *device, + int (*process_mad)(struct rdma_device *device, int process_mad_flags, u8 port_num, struct ib_wc *in_wc, @@ -919,75 +1158,75 @@ u8 phys_port_cnt; }; -struct ib_client { +struct rdma_client { char *name; - void (*add) (struct ib_device *); - void (*remove)(struct ib_device *); + void (*add) (struct rdma_device *); + void (*remove)(struct rdma_device *); struct list_head list; }; -struct ib_device *ib_alloc_device(size_t size); -void ib_dealloc_device(struct ib_device *device); +struct rdma_device *rdma_alloc_device(size_t size); +void rdma_dealloc_device(struct rdma_device *device); -int ib_register_device (struct ib_device *device); -void ib_unregister_device(struct ib_device *device); +int rdma_register_device (struct rdma_device *device); +void rdma_unregister_device(struct rdma_device *device); -int ib_register_client (struct ib_client *client); -void ib_unregister_client(struct ib_client *client); +int rdma_register_client (struct rdma_client *client); +void rdma_unregister_client(struct rdma_client *client); -void *ib_get_client_data(struct ib_device *device, struct ib_client *client); -void ib_set_client_data(struct ib_device *device, struct ib_client *client, +void *rdma_get_client_data(struct rdma_device *device, struct rdma_client *client); +void rdma_set_client_data(struct rdma_device *device, struct rdma_client *client, void *data); -static inline int ib_copy_from_udata(void *dest, struct ib_udata *udata, size_t len) +static inline int rdma_copy_from_udata(void *dest, struct rdma_udata *udata, size_t len) { return copy_from_user(dest, udata->inbuf, len) ? -EFAULT : 0; } -static inline int ib_copy_to_udata(struct ib_udata *udata, void *src, size_t len) +static inline int rdma_copy_to_udata(struct rdma_udata *udata, void *src, size_t len) { return copy_to_user(udata->outbuf, src, len) ? -EFAULT : 0; } -int ib_register_event_handler (struct ib_event_handler *event_handler); -int ib_unregister_event_handler(struct ib_event_handler *event_handler); -void ib_dispatch_event(struct ib_event *event); +int rdma_register_event_handler (struct rdma_event_handler *event_handler); +int rdma_unregister_event_handler(struct rdma_event_handler *event_handler); +void rdma_dispatch_event(struct rdma_event *event); -int ib_query_device(struct ib_device *device, - struct ib_device_attr *device_attr); +int rdma_query_device(struct rdma_device *device, + struct rdma_device_attr *device_attr); -int ib_query_port(struct ib_device *device, - u8 port_num, struct ib_port_attr *port_attr); +int rdma_query_port(struct rdma_device *device, + u8 port_num, union rdma_port_attr *port_attr); -int ib_query_gid(struct ib_device *device, +int rdma_query_gid(struct rdma_device *device, u8 port_num, int index, union ib_gid *gid); -int ib_query_pkey(struct ib_device *device, +int rdma_query_pkey(struct rdma_device *device, u8 port_num, u16 index, u16 *pkey); -int ib_modify_device(struct ib_device *device, +int rdma_modify_device(struct rdma_device *device, int device_modify_mask, - struct ib_device_modify *device_modify); + struct rdma_device_modify *device_modify); -int ib_modify_port(struct ib_device *device, +int rdma_modify_port(struct rdma_device *device, u8 port_num, int port_modify_mask, struct ib_port_modify *port_modify); /** - * ib_alloc_pd - Allocates an unused protection domain. + * rdma_alloc_pd - Allocates an unused protection domain. * @device: The device on which to allocate the protection domain. * * A protection domain object provides an association between QPs, shared * receive queues, address handles, memory regions, and memory windows. */ -struct ib_pd *ib_alloc_pd(struct ib_device *device); +struct rdma_pd *rdma_alloc_pd(struct rdma_device *device); /** - * ib_dealloc_pd - Deallocates a protection domain. + * rdma_dealloc_pd - Deallocates a protection domain. * @pd: The protection domain to deallocate. */ -int ib_dealloc_pd(struct ib_pd *pd); +int rdma_dealloc_pd(struct rdma_pd *pd); /** * ib_create_ah - Creates an address handle for the given address vector. @@ -997,7 +1236,7 @@ * The address handle is used to reference a local or global destination * in all UD QP post sends. */ -struct ib_ah *ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr); +struct ib_ah *ib_create_ah(struct rdma_pd *pd, struct ib_ah_attr *ah_attr); /** * ib_create_ah_from_wc - Creates an address handle associated with the @@ -1011,7 +1250,7 @@ * The address handle is used to reference a local or global destination * in all UD QP post sends. */ -struct ib_ah *ib_create_ah_from_wc(struct ib_pd *pd, struct ib_wc *wc, +struct ib_ah *ib_create_ah_from_wc(struct rdma_pd *pd, struct ib_wc *wc, struct ib_grh *grh, u8 port_num); /** @@ -1039,16 +1278,16 @@ int ib_destroy_ah(struct ib_ah *ah); /** - * ib_create_qp - Creates a QP associated with the specified protection + * rdma_create_qp - Creates a QP associated with the specified protection * domain. * @pd: The protection domain associated with the QP. * @qp_init_attr: A list of initial attributes required to create the QP. */ -struct ib_qp *ib_create_qp(struct ib_pd *pd, - struct ib_qp_init_attr *qp_init_attr); +struct rdma_qp *ib_create_qp(struct rdma_pd *pd, + struct rdma_qp_init_attr *qp_init_attr); /** - * ib_modify_qp - Modifies the attributes for the specified QP and then + * rdma_modify_qp - Modifies the attributes for the specified QP and then * transitions the QP to the given state. * @qp: The QP to modify. * @qp_attr: On input, specifies the QP attributes to modify. On output, @@ -1056,12 +1295,12 @@ * @qp_attr_mask: A bit-mask used to specify which attributes of the QP * are being modified. */ -int ib_modify_qp(struct ib_qp *qp, - struct ib_qp_attr *qp_attr, +int rdma_modify_qp(struct rdma_qp *qp, + struct rdma_qp_attr *qp_attr, int qp_attr_mask); /** - * ib_query_qp - Returns the attribute list and current values for the + * rdma_query_qp - Returns the attribute list and current values for the * specified QP. * @qp: The QP to query. * @qp_attr: The attributes of the specified QP. @@ -1071,16 +1310,16 @@ * The qp_attr_mask may be used to limit the query to gathering only the * selected attributes. */ -int ib_query_qp(struct ib_qp *qp, - struct ib_qp_attr *qp_attr, +int rdma_query_qp(struct rdma_qp *qp, + struct rdma_qp_attr *qp_attr, int qp_attr_mask, - struct ib_qp_init_attr *qp_init_attr); + struct rdma_qp_init_attr *qp_init_attr); /** - * ib_destroy_qp - Destroys the specified QP. + * rdma_destroy_qp - Destroys the specified QP. * @qp: The QP to destroy. */ -int ib_destroy_qp(struct ib_qp *qp); +int rdma_destroy_qp(struct rdma_qp *qp); /** * ib_post_send - Posts a list of work requests to the send queue of @@ -1090,30 +1329,30 @@ * @bad_send_wr: On an immediate failure, this parameter will reference * the work request that failed to be posted on the QP. */ -static inline int ib_post_send(struct ib_qp *qp, - struct ib_send_wr *send_wr, - struct ib_send_wr **bad_send_wr) +static inline int rdma_post_send(struct rdma_qp *qp, + struct rdma_send_wr *send_wr, + struct rdma_send_wr **bad_send_wr) { return qp->device->post_send(qp, send_wr, bad_send_wr); } /** - * ib_post_recv - Posts a list of work requests to the receive queue of + * rdma_post_recv - Posts a list of work requests to the receive queue of * the specified QP. * @qp: The QP to post the work request on. * @recv_wr: A list of work requests to post on the receive queue. * @bad_recv_wr: On an immediate failure, this parameter will reference * the work request that failed to be posted on the QP. */ -static inline int ib_post_recv(struct ib_qp *qp, - struct ib_recv_wr *recv_wr, - struct ib_recv_wr **bad_recv_wr) +static inline int rdma_post_recv(struct rdma_qp *qp, + struct rdma_recv_wr *recv_wr, + struct rdma_recv_wr **bad_recv_wr) { return qp->device->post_recv(qp, recv_wr, bad_recv_wr); } /** - * ib_create_cq - Creates a CQ on the specified device. + * rdma_create_cq - Creates a CQ on the specified device. * @device: The device on which to create the CQ. * @comp_handler: A user-specified callback that is invoked when a * completion event occurs on the CQ. @@ -1125,31 +1364,31 @@ * * Users can examine the cq structure to determine the actual CQ size. */ -struct ib_cq *ib_create_cq(struct ib_device *device, - ib_comp_handler comp_handler, - void (*event_handler)(struct ib_event *, void *), +struct rdma_cq *rdma_create_cq(struct rdma_device *device, + rdma_comp_hanlder comp_handler, + void (*event_handler)(struct rdma_event *, void *), void *cq_context, int cqe); /** - * ib_resize_cq - Modifies the capacity of the CQ. + * rdma_resize_cq - Modifies the capacity of the CQ. * @cq: The CQ to resize. * @cqe: The minimum size of the CQ. * * Users can examine the cq structure to determine the actual CQ size. */ -int ib_resize_cq(struct ib_cq *cq, int cqe); +int rdma_resize_cq(struct rdma_cq *cq, int cqe); /** - * ib_destroy_cq - Destroys the specified CQ. + * rdma_destroy_cq - Destroys the specified CQ. * @cq: The CQ to destroy. */ -int ib_destroy_cq(struct ib_cq *cq); +int rdma_destroy_cq(struct rdma_cq *cq); /** - * ib_poll_cq - poll a CQ for completion(s) + * rdma_poll_cq - poll a CQ for completion(s) * @cq:the CQ being polled * @num_entries:maximum number of completions to return - * @wc:array of at least @num_entries &struct ib_wc where completions + * @wc:array of at least @num_entries &struct rdma_wc where completions * will be returned * * Poll a CQ for (possibly multiple) completions. If the return value @@ -1157,14 +1396,14 @@ * number of completions returned. If the return value is * non-negative and < num_entries, then the CQ was emptied. */ -static inline int ib_poll_cq(struct ib_cq *cq, int num_entries, - struct ib_wc *wc) +static inline int rdma_poll_cq(struct rdma_cq *cq, int num_entries, + struct rdma_wc *wc) { return cq->device->poll_cq(cq, num_entries, wc); } /** - * ib_peek_cq - Returns the number of unreaped completions currently + * rdma_peek_cq - Returns the number of unreaped completions currently * on the specified CQ. * @cq: The CQ to peek. * @wc_cnt: A minimum number of unreaped completions to check for. @@ -1173,29 +1412,29 @@ * this function returns wc_cnt, otherwise, it returns the actual number of * unreaped completions. */ -int ib_peek_cq(struct ib_cq *cq, int wc_cnt); +int rdma_peek_cq(struct rdma_cq *cq, int wc_cnt); /** - * ib_req_notify_cq - Request completion notification on a CQ. + * rdma_req_notify_cq - Request completion notification on a CQ. * @cq: The CQ to generate an event for. * @cq_notify: If set to %IB_CQ_SOLICITED, completion notification will * occur on the next solicited event. If set to %IB_CQ_NEXT_COMP, * notification will occur on the next completion. */ -static inline int ib_req_notify_cq(struct ib_cq *cq, +static inline int rdma_req_notify_cq(struct rdma_cq *cq, enum ib_cq_notify cq_notify) { return cq->device->req_notify_cq(cq, cq_notify); } /** - * ib_req_ncomp_notif - Request completion notification when there are + * rdma_req_ncomp_notif - Request completion notification when there are * at least the specified number of unreaped completions on the CQ. * @cq: The CQ to generate an event for. * @wc_cnt: The number of unreaped completions that should be on the * CQ before an event is generated. */ -static inline int ib_req_ncomp_notif(struct ib_cq *cq, int wc_cnt) +static inline int rdma_req_ncomp_notif(struct rdma_cq *cq, int wc_cnt) { return cq->device->req_ncomp_notif ? cq->device->req_ncomp_notif(cq, wc_cnt) : @@ -1203,15 +1442,15 @@ } /** - * ib_get_dma_mr - Returns a memory region for system memory that is + * rdma_get_dma_mr - Returns a memory region for system memory that is * usable for DMA. * @pd: The protection domain associated with the memory region. * @mr_access_flags: Specifies the memory access rights. */ -struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int mr_access_flags); +struct rdma_mr *rdma_get_dma_mr(struct rdma_pd *pd, int mr_access_flags); /** - * ib_reg_phys_mr - Prepares a virtually addressed memory region for use + * rdma_reg_phys_mr - Prepares a virtually addressed memory region for use * by an HCA. * @pd: The protection domain associated assigned to the registered region. * @phys_buf_array: Specifies a list of physical buffers to use in the @@ -1220,64 +1459,120 @@ * @mr_access_flags: Specifies the memory access rights. * @iova_start: The offset of the region's starting I/O virtual address. */ -struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd, +struct rdma_mr *rdma_reg_phys_mr(struct rdma_pd *pd, struct ib_phys_buf *phys_buf_array, int num_phys_buf, int mr_access_flags, u64 *iova_start); /** - * ib_rereg_phys_mr - Modifies the attributes of an existing memory region. + * rdma_reg_phys_mr_fixed - Prepares a virtually addressed memory region for use + * use by an RNIC with fixed page/block sizes. + * @pd: The protection domain associated assigned to the registered region. + * @phys_buf_array: Specifies a list of physical buffers to use in the + * memory region. + * @num_phys_buf: Specifies the size of the phys_buf_array. + * @pble_size: Size of each page/block in phys_buf_array + * @mr_access_flags: Specifies the memory access rights. + * @iova_start: The offset of the region's starting I/O virtual address. + */ +struct rdma_mr *rdma_reg_phys_mr(struct rdma_pd *pd, + u64 *phys_buf_array, + int num_phys_buf, + u32 pble_size, + int mr_access_flags, + u64 *iova_start); +/** + * rdma_rereg_phys_mr - Modifies the attributes of an existing memory region. * Conceptually, this call performs the functions deregister memory region * followed by register physical memory region. Where possible, * resources are reused instead of deallocated and reallocated. * @mr: The memory region to modify. * @mr_rereg_mask: A bit-mask used to indicate which of the following * properties of the memory region are being modified. - * @pd: If %IB_MR_REREG_PD is set in mr_rereg_mask, this field specifies + * @pd: If %RDMA_MR_REREG_PD is set in mr_rereg_mask, this field specifies * the new protection domain to associated with the memory region, * otherwise, this parameter is ignored. - * @phys_buf_array: If %IB_MR_REREG_TRANS is set in mr_rereg_mask, this + * @phys_buf_array: If %RDMA_MR_REREG_TRANS is set in mr_rereg_mask, this * field specifies a list of physical buffers to use in the new * translation, otherwise, this parameter is ignored. - * @num_phys_buf: If %IB_MR_REREG_TRANS is set in mr_rereg_mask, this + * @num_phys_buf: If %RDMA_MR_REREG_TRANS is set in mr_rereg_mask, this * field specifies the size of the phys_buf_array, otherwise, this * parameter is ignored. - * @mr_access_flags: If %IB_MR_REREG_ACCESS is set in mr_rereg_mask, this + * @mr_access_flags: If %RDMA_MR_REREG_ACCESS is set in mr_rereg_mask, this * field specifies the new memory access rights, otherwise, this * parameter is ignored. * @iova_start: The offset of the region's starting I/O virtual address. */ -int ib_rereg_phys_mr(struct ib_mr *mr, +int rdma_rereg_phys_mr(struct rdma_mr *mr, int mr_rereg_mask, - struct ib_pd *pd, + struct rdma_pd *pd, struct ib_phys_buf *phys_buf_array, int num_phys_buf, int mr_access_flags, u64 *iova_start); /** - * ib_query_mr - Retrieves information about a specific memory region. + * rdma_rereg_phys_mr_fixed - Modifies the attributes of an existing memory region. + * Conceptually, this call performs the functions deregister memory region + * followed by register physical memory region. Where possible, + * resources are reused instead of deallocated and reallocated. + * A fixed size for each page/block is supplied as a distinct parameter + * @mr: The memory region to modify. + * @mr_rereg_mask: A bit-mask used to indicate which of the following + * properties of the memory region are being modified. + * @pd: If %RDMA_MR_REREG_PD is set in mr_rereg_mask, this field specifies + * the new protection domain to associated with the memory region, + * otherwise, this parameter is ignored. + * @phys_buf_array: If %RDMA_MR_REREG_TRANS is set in mr_rereg_mask, this + * field specifies a list of physical buffers to use in the new + * translation, otherwise, this parameter is ignored. + * @num_phys_buf: If %RDMA_MR_REREG_TRANS is set in mr_rereg_mask, this + * field specifies the size of the phys_buf_array, otherwise, this + * parameter is ignored. + * @pble_size: size of each page/block in phys_buf_array. + * @mr_access_flags: If %RDMA_MR_REREG_ACCESS is set in mr_rereg_mask, this + * field specifies the new memory access rights, otherwise, this + * parameter is ignored. + * @iova_start: The offset of the region's starting I/O virtual address. + */ +int rdma_rereg_phys_mr_fixed (struct rdma_mr *mr, + int mr_rereg_mask, + struct rdma_pd *pd, + u64 *phys_buf_array, + int num_phys_buf, + u32 pble_size, + int mr_access_flags, + u64 *iova_start); + +/** + * rdma_query_mr - Retrieves information about a specific memory region. * @mr: The memory region to retrieve information about. * @mr_attr: The attributes of the specified memory region. */ -int ib_query_mr(struct ib_mr *mr, struct ib_mr_attr *mr_attr); +int rdma_query_mr(struct rdma_mr *mr, struct rdma_mr_attr *mr_attr); /** - * ib_dereg_mr - Deregisters a memory region and removes it from the + * rdma_dereg_mr - Deregisters a memory region and removes it from the * HCA translation table. * @mr: The memory region to deregister. */ -int ib_dereg_mr(struct ib_mr *mr); +int rdma_dereg_mr(struct rdma_mr *mr); /** - * ib_alloc_mw - Allocates a memory window. + * rdma_alloc_mw - Allocates a memory window. * @pd: The protection domain associated with the memory window. */ -struct ib_mw *ib_alloc_mw(struct ib_pd *pd); +struct rdma_mw *ib_alloc_mw(struct rdma_pd *pd); /** - * ib_bind_mw - Posts a work request to the send queue of the specified + * rdma_alloc_mw_narrow - Allocates a narrow (type 2) memory window. + * @pd: The protection domain associated with the memory window. + */ +struct rdma_mw *ib_alloc_mw(struct rdma_pd *pd); + +/** + * rdma_bind_mw - Posts a work request to the send queue of the specified * QP, which binds the memory window to the given address range and * remote access attributes. * @qp: QP to post the bind work request on. @@ -1285,9 +1580,9 @@ * @mw_bind: Specifies information about the memory window, including * its address range, remote access rights, and associated memory region. */ -static inline int ib_bind_mw(struct ib_qp *qp, - struct ib_mw *mw, - struct ib_mw_bind *mw_bind) +static inline int rdma_bind_mw(struct rdma_qp *qp, + struct rdma_mw *mw, + struct rdma_mw_bind *mw_bind) { /* XXX reference counting in corresponding MR? */ return mw->device->bind_mw ? @@ -1296,10 +1591,10 @@ } /** - * ib_dealloc_mw - Deallocates a memory window. + * rdma_dealloc_mw - Deallocates a memory window. * @mw: The memory window to deallocate. */ -int ib_dealloc_mw(struct ib_mw *mw); +int rdma_dealloc_mw(struct rdma_mw *mw); /** * ib_alloc_fmr - Allocates a unmapped fast memory region. @@ -1310,7 +1605,7 @@ * A fast memory region must be mapped before it can be used as part of * a work request. */ -struct ib_fmr *ib_alloc_fmr(struct ib_pd *pd, +struct ib_fmr *ib_alloc_fmr(struct rdma_pd *pd, int mr_access_flags, struct ib_fmr_attr *fmr_attr); @@ -1352,7 +1647,7 @@ * the fabric appropriately. The port associated with the specified * QP must also be a member of the multicast group. */ -int ib_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid); +int ib_attach_mcast(struct rdma_qp *qp, union ib_gid *gid, u16 lid); /** * ib_detach_mcast - Detaches the specified QP from a multicast group. @@ -1360,6 +1655,6 @@ * @gid: Multicast group GID. * @lid: Multicast group LID in host byte order. */ -int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid); +int ib_detach_mcast(struct rdma_qp *qp, union ib_gid *gid, u16 lid); #endif /* IB_VERBS_H */ From caitlinb at siliquent.com Thu Jun 30 08:53:23 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 08:53:23 -0700 Subject: [openib-general] gen2/rnic-pi differences Message-ID: <8508251A6FC08A489844A94261D3693A057C74@fiona.siliquent.com> This is a list of structural differences in between OpenIB's gen2 verbs and RNIC-PI that will remain even after gen2 is made "transport neutral". After these distinctions are understood, a decision should be made as to whether the differences represent different objectives and should be merely documented for the benefit of IHVs or whether their should be a migration to end that specific difference. For example, it might be decided that a given RNIC-PI feature was inherently related to other Operating Systems. The response might be to merely not the difference, or to determine when Linux could support such a feature. Through this discussion the term "kVP" is used to reference the model/provider specific code that executes in the kernel, while "uVP" references model/provider specific code that executes in user space. Memory Registration / Lookups gen2 translates virtual memory registrations to physical lists before the kVP is invoked. RNIC-PI expects the virtual memory registration request to be passed to the kVP untranslated. The kVP then makes a callback to obtain address translations and to pin memory. Mapping and pinning may be performed as separate steps, allowing mapping of Consumer pinned memory. gen2 does not currently support registration of shared memory regions. Locking gen2 does not have a clear statement about expection of who is responsible for preventing concurrent data access. Fastpath operations must be callable from within an interrupt or while holding a spinlock. Slowpath operations are allowed to block. It is not clear if gen2 would allow suppression of locking when the caller has taken responsibility for serializing all object access. RNIC-PI leaves division of that responsibility to be worked out between the Consumer and the Access Layer (DAT/IT-API). RNIC-PI allows allocating slowpath operations to block, but does not allow non-allocating slowpath operations or fastpath operations to block or stall indefinitely. For the latter cases it must be legal for the caller to hold a spinlock over the call. By default, RNIC-PI places responsibility for serializing access to an object on the caller. RNIC-PI has tentatively decided to allow a second optional set of verbs where the verb layer will provide serializations. User-Mode Handles gen2 never exports kernel pointers to user-mode, but rather registers all such pointers as handles using standard routines. All handles passed back in from user-mode are validated as a by-product of translation back to kernel mode pointers. RNIC-PI assigns that responsibility to the kAL (Kernel Access Layer) but only requires *validation* of handles. It does not explicitly address *translation* nor placing them in a central registry. os_data / Identification of Consumer Objects gen2 provides minimal support for identification of RDMA resources using consumer supplied handles. A user-supplied context is available in callbacks, but not in work completions. RNIC-PI provides a general "os_data" capability that allows each RNIC-PI object to have a consumer supplied alias that is used for all queries (including on other objects), callbacks and work completions. The RNIC-PI approach can eliminate the need for reverse indexes or per work request tracking data by the verbs consumer (such as the "DTO_COOKIE" in the reference DAPL implementation). This is of greatest concern when reaping a work completion in user mode, as that there is no way to translate a qp_num to a QP object in user mode. It isn't that easy in kernel mode when dealing with multiple vendors, either. Work Request Opaques / Local Solicited / Threshold Solicited RNIC-PI provides "Work Request Opaques" that allow the verbs consumer (especially DAPL/IT-API) to mark certain work requests with pass-through flags rather than using a parallel data structure such as the DTO_COOKIE. One of these flags allows a work request to be marked as "local solicited", which will make it an urgent event (one justifying a completion notification callback) when it completes successfully (essentially setting the solicited bit locally). IHVs can support these bits a) not at all, b) as pass-thru or c) actually implement the Local Solicited semantics in hardware. gen2 also provides an enhanced method for providing an earlier completion callback than provided for in the verbs, but it is more akin to the DAT/IT-API evd threshold feature. RNIC-PI defines no such feature, partially because callbacks always occur in the kernel. kernel callbacks RNIC-PI only provides kernel callbacks. No callbacks are provided in user mode. It is assumed that the callback routine is part of the kAL (Kernel Access Layer) and that it will co-ordinate unblocking of EVD waiters with the uAL (User Access Layer). This allows optimized handling of many callback scenarios where the net effect is to kick a file descriptor, wake another thread or to take no immediate action. It also avoids uVPs having to add callback support, which they were not required to have under the RDMAC verbs. gen2 provides a standardized relay of callback notification events from kernel mode to user mode. ihv_data / model specific data RNIC-PI defines an opaque pointer that can be used to communicate model-specific data between the uVP and kVP. gen2 allows vendors to add extra bytes both IN and OUT to each verb request / response communicated over the user context fd. The same information can be communicated effectively using either approach. user / kernel communications gen2 creates a file descriptor for each open RDMA Device instance (i.e, per device per client). RNIC-PI does not define how the sysCall is implemented, but implies that there is one per client (no matter how many devices). Additional error information gen2 provides for vendor specific error information in work completions. RNIC-PI provides for additional OS-specific error reporting through the 'err_data' opaque in a variety of contexts. But it is OS-specific. STag0 (or equivalent) RNIC-PI has each rnic define a pre-existing "all physical memory" memory region (STag0 for iWARP). gen2 provides a verb for the kVP to create such a memory region. Equivalent results can be achieved with either interface, and equivalent support from the kVP is required in either case. Doorbells gen2 defines a standard method for mapping the doorbell. RNIC-PI presumes that this will be solved between the uVP and kVP and that no standard interface is required. peek_cq gen2 defines a method to peek at a cq (see if there are more entries there without attempting to reap a completion). RNIC-PI does not define this, but it should be feasible for almost all implementations. From hch at lst.de Thu Jun 30 09:03:02 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 30 Jun 2005 18:03:02 +0200 Subject: [openib-general] Making gen2 transport neutral In-Reply-To: <8508251A6FC08A489844A94261D3693A057C72@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A057C72@fiona.siliquent.com> Message-ID: <20050630160302.GB16573@lst.de> On Thu, Jun 30, 2005 at 08:52:54AM -0700, Caitlin Bestler wrote: > structs: > typically "struct ib_xyz" is transformed as follows: > > struct ib_xyz { > /* Only IB specific fields remain */ > /* In some cases fields have been split, because > * iWARP allows two things to vary that IB had > * locked together. SGE limits are the primary > * example of this. iWARP can have different > * limits on SGE size for each type of message > */ > }; > > struct iwarp_xyz { > /* equivalent iWARP specific fields */ > }; > > struct rdma_xyz { > /* Transport neutral fields. Typically > * a subset of what was in struct ib_xyz before > */ > union { > struct ib_xyz ib; > struct iwarp_xyz iwarp; > } xpt; > }; wrong way around, but we had that before. It should be struct ib_foo { struct rdma_foo common; ... } struct iwarp_foo { struct rdma_foo common; ... } see filesystem and network protocol private data for example where why historically did it that union way and it didn't work out at all long-term. > I am assuming that returning a "not supported" error for irrelevant > verbs is an acceptable burnden for all providers. Even better make the methods implementing them optional and let the upper layer return EOPNOTSUPP when it's not implemeneted. From rolandd at cisco.com Thu Jun 30 09:03:26 2005 From: rolandd at cisco.com (Roland Dreier) Date: Thu, 30 Jun 2005 09:03:26 -0700 Subject: [openib-general] [BUG]driver didn't accept ARP reply packet In-Reply-To: (Shirley Ma's message of "Wed, 29 Jun 2005 20:59:46 -0700") References: Message-ID: <52hdffbzn5.fsf@topspin.com> Shirley> I did netperf test over Mellanox 23108 4X HCA against Shirley> r2720, after a while the HCA stopped to accept packets. Shirley> I am pretty sure it's a bug in driver not IPoIB. rmmod Shirley> ib_ipoib didn't help, after removing ib_mthca, and Shirley> restarted, the driver worked again. It's easy to Shirley> reprocude on my 4-way intel based systems. What is your recipe for reproducing? Shirley> I did saw the below messages from /var/log/messages, when Shirley> this happened. Shirley> ib_mthca 0000:04:00.0: Port change to down for port 1 Shirley> ib_mthca 0000:04:00.0: Port change to active for port 1 This means that your link is going down and coming back up. What do the error counters in /sys/class/infiniband/mthca0/ports/1/counters show before and after this happens? Shirley> And the HCA didn't accept ARP reply packet afterwards. Shirley> Now I am adding a permanent arp entry to avoid this Shirley> problem. It seems to work fine. Anybody else saw this Shirley> problem? No, I haven't seen this happen. - R. From rolandd at cisco.com Thu Jun 30 09:05:20 2005 From: rolandd at cisco.com (Roland Dreier) Date: Thu, 30 Jun 2005 09:05:20 -0700 Subject: [openib-general] [BUG]driver didn't accept ARP reply packet In-Reply-To: (Shirley Ma's message of "Wed, 29 Jun 2005 22:56:51 -0700") References: Message-ID: <52d5q3bzjz.fsf@topspin.com> Shirley> It could be a firmware problem. I used 3.3.2 firmware, Shirley> the sender side had this problem. The receiver side was Shirley> OK. Anyway after back to 3.2.0 firmware, it's OK now. Hmm, if firmware makes a difference then it is either a firmware bug or a very subtle timing problem in the driver. Can you give details on how you reproduce this problem? Thanks, Roland From robert.j.woodruff at intel.com Thu Jun 30 09:06:20 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Thu, 30 Jun 2005 09:06:20 -0700 Subject: [openib-general] comments on DAT registry in OpenIB Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408> Christoph wrote, >On Thu, Jun 30, 2005 at 08:38:45AM -0700, Caitlin Bestler wrote: >> actual code requirements makes no sense. If you >> can't come up with something that remains acceptable >> to the broader community of DAT users then you should >> refrain from using the "dat_" symbols and their already >> established meanings. >That's exactly what I proposed. Please read the mail again. I think that your suggestion to s/DAT/RDMA makes sense, since this code is quickly becoming "the" RDMA transport independent interface for Linux, rather than trying to RNIC-PI unionize the IB core layer to make it support both IB and iWarp. From caitlinb at siliquent.com Thu Jun 30 09:09:47 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 09:09:47 -0700 Subject: [openib-general] Making gen2 transport neutral Message-ID: <8508251A6FC08A489844A94261D3693A057C77@fiona.siliquent.com> > -----Original Message----- > From: Christoph Hellwig [mailto:hch at lst.de] > Sent: Thursday, June 30, 2005 9:03 AM > To: Caitlin Bestler > Cc: openib-general; rdma-developers at lists.sourceforge.net > Subject: Re: [openib-general] Making gen2 transport neutral > > On Thu, Jun 30, 2005 at 08:52:54AM -0700, Caitlin Bestler wrote: > > structs: > > typically "struct ib_xyz" is transformed as follows: > > > > struct ib_xyz { > > /* Only IB specific fields remain */ > > /* In some cases fields have been split, because > > * iWARP allows two things to vary that IB had > > * locked together. SGE limits are the primary > > * example of this. iWARP can have different > > * limits on SGE size for each type of message > > */ > > }; > > > > struct iwarp_xyz { > > /* equivalent iWARP specific fields */ > > }; > > > > struct rdma_xyz { > > /* Transport neutral fields. Typically > > * a subset of what was in struct ib_xyz before > > */ > > union { > > struct ib_xyz ib; > > struct iwarp_xyz iwarp; > > } xpt; > > }; > > wrong way around, but we had that before. It should be > > struct ib_foo { > struct rdma_foo common; > ... > } > > struct iwarp_foo { > struct rdma_foo common; > ... > } > That's a style question that I've heard both opinions on. I'm comfortable with either approach, and actually prefer the one you are suggesting. But I've ususally been on the losing side of those discussions (not that I care strongly either way). If there's a citation in coding standards somewhere that makes this a closed issue please cite it. That would close the discussion quickly. If there isn't a solid rule it really comes down to a question of how likely a new RDMA transport is that is not based upon either IB or iWARP. I doubt it. iWARP differs from IB because it *had* to in order to ride over TCP/IP and SCTP. IB has assumptions that are only possible when the tranport and RDMA layers are intergrated and share acknowledgement messages. For iWARP to emulate that would have required redundant ack messages that would have cluttered the network. From rolandd at cisco.com Thu Jun 30 09:10:09 2005 From: rolandd at cisco.com (Roland Dreier) Date: Thu, 30 Jun 2005 09:10:09 -0700 Subject: [openib-general] Making gen2 transport neutral In-Reply-To: <20050630160302.GB16573@lst.de> (Christoph Hellwig's message of "Thu, 30 Jun 2005 18:03:02 +0200") References: <8508251A6FC08A489844A94261D3693A057C72@fiona.siliquent.com> <20050630160302.GB16573@lst.de> Message-ID: <523bqzbzby.fsf@topspin.com> Christoph> Even better make the methods implementing them optional Christoph> and let the upper layer return EOPNOTSUPP when it's not Christoph> implemeneted. That's what the current drivers/infiniband code does, except we use ENOSYS. However I have no objection to changing to EOPNOTSUPP if that seems like a more sensible value. - R. From mshefty at ichips.intel.com Thu Jun 30 09:13:28 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 09:13:28 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <20050629171221.I26240@topspin.com> References: <42C1BA15.7060205@ichips.intel.com> <20050629111038.G26240@topspin.com> <42C2E573.5010304@ichips.intel.com> <20050629171221.I26240@topspin.com> Message-ID: <42C41A28.4000809@ichips.intel.com> Libor Michalek wrote: > Assume that the userspace 'struct ib_cm_event' contains the cm_id as > well as a new 'u64 context' which is inherited from the cm_id, and is > set at the time of the cm_id creation. This is what I'm assuming that > Arlin would like to see. > > In the case of two threads accessing the CM at once there's a race > condition if you are going to use the 'context' variable as a pointer > to memory: > > Thread 1 Thread 2 > ------------------------------------- ----------------------------------- > cm_object = malloc(sizeof(*cm_object) > ib_cm_create_id(&cm_object->cm_id, > (u64)cm_object) > > ib_cm_event_get(&event) > ib_cm_destroy_id(cm_object->cm_id) > free(cm_object); > process_event((void *)event->context); I see. This appears to come from a difference between the event reporting model used by the kernel CM versus the usermode CM (callback versus calldown). Maybe there's a way to assist the user here. Can we report a destruction event, or require a second call to indicate that an event has been processed? In the latter case, destruction could block while the event is being processed. Not sure if either of these would help if the user processed events using multiple threads, but I think with additional serialization in the CM it might be able to work. - Sean From hch at lst.de Thu Jun 30 09:14:03 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 30 Jun 2005 18:14:03 +0200 Subject: [openib-general] Making gen2 transport neutral In-Reply-To: <523bqzbzby.fsf@topspin.com> References: <8508251A6FC08A489844A94261D3693A057C72@fiona.siliquent.com> <20050630160302.GB16573@lst.de> <523bqzbzby.fsf@topspin.com> Message-ID: <20050630161403.GA16871@lst.de> On Thu, Jun 30, 2005 at 09:10:09AM -0700, Roland Dreier wrote: > Christoph> Even better make the methods implementing them optional > Christoph> and let the upper layer return EOPNOTSUPP when it's not > Christoph> implemeneted. > > That's what the current drivers/infiniband code does, except we use > ENOSYS. However I have no objection to changing to EOPNOTSUPP if that > seems like a more sensible value. The actual value doesn't really matter. From hch at lst.de Thu Jun 30 09:14:44 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 30 Jun 2005 18:14:44 +0200 Subject: [openib-general] Making gen2 transport neutral In-Reply-To: <8508251A6FC08A489844A94261D3693A057C77@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A057C77@fiona.siliquent.com> Message-ID: <20050630161444.GB16871@lst.de> On Thu, Jun 30, 2005 at 09:09:47AM -0700, Caitlin Bestler wrote: > If there's a citation in coding standards somewhere that > makes this a closed issue please cite it. That would close > the discussion quickly. It's far too highlevel for the coding standards documents we have. Maybe I'll find time to write up a more highlevel one. From halr at voltaire.com Thu Jun 30 09:13:07 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 Jun 2005 12:13:07 -0400 Subject: [openib-general] [PATCH] mad_rmpp: Fix non first segment copying in ib_coalesce_recv_mad Message-ID: <1120147987.4371.52.camel@hal.voltaire.com> mad_rmpp: Fix non first segment copying in ib_coalesce_recv_mad Signed-off-by: Hal Rosenstock Index: mad_rmpp.c =================================================================== --- mad_rmpp.c (revision 2757) +++ mad_rmpp.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Intel Inc. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -443,7 +444,7 @@ data = rmpp_mad; size = sizeof(*rmpp_mad); } else { - data = rmpp_mad + offset; + data = (void *)rmpp_mad + offset; if (flags & IB_MGMT_RMPP_FLAG_LAST) size = len; else From rolandd at cisco.com Thu Jun 30 09:18:20 2005 From: rolandd at cisco.com (Roland Dreier) Date: Thu, 30 Jun 2005 09:18:20 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408> (Robert J. Woodruff's message of "Thu, 30 Jun 2005 09:06:20 -0700") References: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408> Message-ID: <52slyzakdv.fsf@topspin.com> Robert> I think that your suggestion to s/DAT/RDMA makes sense, Robert> since this code is quickly becoming "the" RDMA transport Robert> independent interface for Linux, rather than trying to Robert> RNIC-PI unionize the IB core layer to make it support both Robert> IB and iWarp. I disagree. It doesn't make sense to me for us to add an abstraction layer on top of another abstraction layer -- let's just fix the first abstraction layer. If we follow the approach of changing the name of DAT to RDMA and then putting it in the kernel, we end up with a stack that looks like: upper layer protocol <-> RDMA midlayer <-> IB RDMA provider <-> IB midlayer <-> IB low-level driver Let's just evolve the IB midlayer so the picture can be more sensible: upper layer protocol <-> RDMA midlayer <-> IB low-level driver - R. From hch at lst.de Thu Jun 30 09:22:10 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 30 Jun 2005 18:22:10 +0200 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <52slyzakdv.fsf@topspin.com> References: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408> <52slyzakdv.fsf@topspin.com> Message-ID: <20050630162210.GA17105@lst.de> On Thu, Jun 30, 2005 at 09:18:20AM -0700, Roland Dreier wrote: > Robert> I think that your suggestion to s/DAT/RDMA makes sense, > Robert> since this code is quickly becoming "the" RDMA transport > Robert> independent interface for Linux, rather than trying to > Robert> RNIC-PI unionize the IB core layer to make it support both > Robert> IB and iWarp. > > I disagree. It doesn't make sense to me for us to add an abstraction > layer on top of another abstraction layer -- let's just fix the first > abstraction layer. Well, the plan was to take the parts of the DAT API that make sense and put them into that generic RDMA layer. DAT advocates claimed there were such useful higherlevel abstractions, but the more I look at the dat/dat-provider codebase I doubt there's a lot of them. From mshefty at ichips.intel.com Thu Jun 30 09:28:33 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 09:28:33 -0700 Subject: [openib-general] Re: [PATCH] mad_rmpp: Fix non first segment copying in ib_coalesce_recv_mad In-Reply-To: <1120147987.4371.52.camel@hal.voltaire.com> References: <1120147987.4371.52.camel@hal.voltaire.com> Message-ID: <42C41DB1.3010903@ichips.intel.com> Hal Rosenstock wrote: > mad_rmpp: Fix non first segment copying in ib_coalesce_recv_mad > > Signed-off-by: Hal Rosenstock Thanks! Committed. - Sean From rolandd at cisco.com Thu Jun 30 09:29:16 2005 From: rolandd at cisco.com (Roland Dreier) Date: Thu, 30 Jun 2005 09:29:16 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <20050630162210.GA17105@lst.de> (Christoph Hellwig's message of "Thu, 30 Jun 2005 18:22:10 +0200") References: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408> <52slyzakdv.fsf@topspin.com> <20050630162210.GA17105@lst.de> Message-ID: <52hdffajvn.fsf@topspin.com> Christoph> Well, the plan was to take the parts of the DAT API Christoph> that make sense and put them into that generic RDMA Christoph> layer. DAT advocates claimed there were such useful Christoph> higherlevel abstractions, but the more I look at the Christoph> dat/dat-provider codebase I doubt there's a lot of Christoph> them. That seems fine. I agree that there are probably some useful abstractions that can be put in a library. An example I can think of is a facility to "connect queue pair X to remote IP address Y" -- that would be useful to SDP, iSER and NFS/RDMA. However, having a different API for posting work requests or receiving events doesn't seem useful. - R. From halr at voltaire.com Thu Jun 30 09:26:20 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 Jun 2005 12:26:20 -0400 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support Message-ID: <1120148780.4371.68.camel@hal.voltaire.com> user_mad: Add receive side RMPP support Signed-off-by: Hal Rosenstock Index: user_mad.c =================================================================== --- user_mad.c (revision 2757) +++ user_mad.c (working copy) @@ -176,7 +176,7 @@ if (mad_recv_wc->wc->status != IB_WC_SUCCESS) goto out; - length = 256; /* until RMPP is supported */ + length = mad_recv_wc->mad_len; packet = kmalloc(sizeof *packet + length, GFP_KERNEL); if (!packet) goto out; @@ -184,7 +184,8 @@ memset(packet, 0, sizeof *packet + length); packet->length = length; - memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, length); + ib_coalesce_recv_mad(mad_recv_wc, packet->mad.data); + packet->mad.hdr.status = 0; packet->mad.hdr.qpn = cpu_to_be32(mad_recv_wc->wc->src_qp); packet->mad.hdr.lid = cpu_to_be16(mad_recv_wc->wc->slid); @@ -214,7 +215,7 @@ struct ib_umad_packet *packet; ssize_t ret; - if (count < sizeof (struct ib_user_mad) + 256) /* until RMPP supported */ + if (count < sizeof (struct ib_user_mad) + 256) return -EINVAL; spin_lock_irq(&file->recv_lock); @@ -238,13 +239,14 @@ spin_unlock_irq(&file->recv_lock); if (count < packet->length + sizeof (struct ib_user_mad)) - ret = -EINVAL; + /* return length needed if too small */ + ret = packet->length + sizeof (struct ib_user_mad); else if (copy_to_user(buf, &packet->mad, packet->length + sizeof (struct ib_user_mad))) ret = -EFAULT; else ret = packet->length + sizeof (struct ib_user_mad); - if (ret < 0) { + if (ret < 0 || count < packet->length + sizeof (struct ib_user_mad)) { /* Requeue packet */ spin_lock_irq(&file->recv_lock); list_add(&packet->list, &file->recv_list); From caitlinb at siliquent.com Thu Jun 30 09:33:26 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 09:33:26 -0700 Subject: [openib-general] RE: Making gen2 transport neutral Message-ID: <8508251A6FC08A489844A94261D3693A057C7C@fiona.siliquent.com> > -----Original Message----- > > >This is a quick outline of the changes required to make ib_verbs.h > >transport neutral. Some of the changes were actually > required anyway to > >fully support IB 1.2. > > Same comment on the following structure, it is not an RDMA > transport independent structure, it is an IB/iWarp only > structure that will require changes to support a new RDMA > capable device. > The goal of having "transport neutral verbs" has to be understood. My assumption is that it is not a "verb layer" interface if it abstracts or limits the ability of system software to control and understand what is happening. DAT/IT-API already provide abstractions to allow *applications* to be unaware of these details, the goal of a lower layer 'transport neutral' interface is to preserve as much transport independence as possible *without* the loss of information or control. But of course to be useful it must be possible to do a substantial number of operations in a transport neutral way. There is no point to even having the interface if it is nothing more than endless unions. I believe this is achievable, and that is what is outlined. The most commonly used operations can be invoked in a transport neutral fashion. You can create PDs, virtual memory MRs, MWs, QPs, SRQs, CQ and post to connected QPs without needing a single 'if(ib)' statement. You still must deal with connection management and many error conditions in a transport neutral fashion. I also believe that any third transport would proably be able to define itself without the addition of any new connection establishment controls, error returns or error completions or asynchronous events. Any additions would be minor. The large number of distinctions between IB and iWARP are a direct result of two different models of how L4.5 (RDMA) interfaces with L2-4. I doubt that a third transport would come up with a third model. It will resemble iWARP or it will resemble IB. From mshefty at ichips.intel.com Thu Jun 30 09:34:59 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 09:34:59 -0700 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <1120148780.4371.68.camel@hal.voltaire.com> References: <1120148780.4371.68.camel@hal.voltaire.com> Message-ID: <42C41F33.4010605@ichips.intel.com> Hal Rosenstock wrote: > - if (count < sizeof (struct ib_user_mad) + 256) /* until RMPP supported */ > + if (count < sizeof (struct ib_user_mad) + 256) > return -EINVAL; Does it make more sense to use sizeof(struct ib_mad) rather than 256? - Sean From caitlinb at siliquent.com Thu Jun 30 09:38:25 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 09:38:25 -0700 Subject: [openib-general] RE: [Rdma-developers] RE: Making gen2 transport neutral Message-ID: <8508251A6FC08A489844A94261D3693A057C7D@fiona.siliquent.com> > > You still must deal with connection management and many error > conditions in a transport neutral fashion. > Typo: transport *dependent* fashion. From robert.j.woodruff at intel.com Thu Jun 30 09:38:57 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 30 Jun 2005 09:38:57 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <52slyzakdv.fsf@topspin.com> Message-ID: Roland wrote, >I disagree. It doesn't make sense to me for us to add an abstraction >layer on top of another abstraction layer -- let's just fix the first >abstraction layer. >If we follow the approach of changing the name of DAT to RDMA and then >putting it in the kernel, we end up with a stack that looks like: > upper layer protocol <-> RDMA midlayer <-> IB RDMA provider <-> IB midlayer <-> >IB low-level driver >Let's just evolve the IB midlayer so the picture can be more sensible: > upper layer protocol <-> RDMA midlayer <-> IB low-level driver > - R. At some layer the interface is transport dependent. I think that this is at the IB core mid-layer. For example, the IB CM and iWarp CM are totally different. The IB layer has SA queries that are not needed by iWarp. I think it could look like, ULP <-> RDMA-midlayer <-> ib-midlayer <-> H/W driver <-> iwarp-midlayer <-> H/W driver where ib midlayer and iWarp midlayer can share common code for functions that both need. For example, memory pinning and such might be common so you could have ib_reg_mem() { common_memory_pin() register_mem_with_IB_HCA() } iWarp_reg_mem() common_memory_pin() register_mem_with_iWarp_Nic() } Also starting with separate mid-layers for ib and iwarp allows them to initially be developed independently and in parallel, with the factoring out of common code later and not totally disrupting the ib tree. my 2 cents. woody From halr at voltaire.com Thu Jun 30 09:40:27 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 Jun 2005 12:40:27 -0400 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <42C41F33.4010605@ichips.intel.com> References: <1120148780.4371.68.camel@hal.voltaire.com> <42C41F33.4010605@ichips.intel.com> Message-ID: <1120149626.4371.75.camel@hal.voltaire.com> On Thu, 2005-06-30 at 12:34, Sean Hefty wrote: > Hal Rosenstock wrote: > > - if (count < sizeof (struct ib_user_mad) + 256) /* until RMPP supported */ > > + if (count < sizeof (struct ib_user_mad) + 256) > > return -EINVAL; > > Does it make more sense to use sizeof(struct ib_mad) rather than 256? But of course. I'll update and reissue the patch. Thanks. -- Hal From krause at cup.hp.com Thu Jun 30 09:44:32 2005 From: krause at cup.hp.com (Michael Krause) Date: Thu, 30 Jun 2005 09:44:32 -0700 Subject: [openib-general] Re: IP addressing on InfiniBand networks (Caitlin Bestler) Message-ID: <6.2.0.14.2.20050630094418.029b9008@esmail.cup.hp.com> At 10:39 PM 6/29/2005, Bill Strahm wrote: >>------------------------------ >> >>Message: 2 >>Date: Wed, 29 Jun 2005 09:00:37 -0700 >>From: Roland Dreier >>Subject: Re: [openib-general] IP addressing on InfiniBand networks >>To: Caitlin Bestler >>Cc: "Lentini, James" , Christoph Hellwig >> , openib-general >>Message-ID: <5264vxi256.fsf at topspin.com> >>Content-Type: text/plain; charset=us-ascii >> >> Caitlin> An assigned GID meets all of the requirements for an IA >> Caitlin> Address. I think taking advantage of that existing >> Caitlin> capability is just one of many options that can be done >> Caitlin> by the IB CM rather than forcing IB specific changes up >> Caitlin> to the application layer. >> >>Just to be clear, the IBA spec is very clear that a GID _is_ an IPv6 address. >> >>- R. >> >> >Just to be REALLY clear - IANA has not allocated IPv6 address space to any >Infiniband entities - so they are not Internet IPv6 addresses. GIDs are >formatted like IPv6 addresses but in no sense should EVER be used at an IP >layer 3 address. > > From an IPoIB stance - IB is just a very over engineered Layer 2 that has > a singularly large MAC address. > > From an IB ULP point of view - how you get to the layer 2 address that is > needed to perform communications that do not include IP, it isn't a > problem - but let me tell you.... The leadership of the IETF is scared of > IB because of saying things IB GIDs _ARE_ IPv6 addresses. Being the person who led the addressing definition for IB, I can state quite clearly that GID are NOT IPv6 addresses. They were intentionally defined to have a similar look-n-feel since they were derived in large part from Future I/O which had them as real IPv6 addresses. But again, they are NOT IPv6 addresses. For IP over IB, it is unfortunate that we could not have simply used a raw datagram service as that would have made life very simple but we are in the state we are so that means there is a UD transport providing a layer 2 Ethernet equivalent of functionality. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.j.woodruff at intel.com Thu Jun 30 09:46:54 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Thu, 30 Jun 2005 09:46:54 -0700 Subject: [openib-general] comments on DAT registry in OpenIB Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004D1F71F@orsmsx408> Christoph wrote, >Well, the plan was to take the parts of the DAT API that make sense >and put them into that generic RDMA layer. DAT advocates claimed there >were such useful higherlevel abstractions, but the more I look at >the dat/dat-provider codebase I doubt there's a lot of them. One example of a useful abstraction in the DAT layer is connection establishment. The ULP can call the DAT connection establishment routines and it deals with the transport specific connection mechanism, which are very different in IB and iWarp for example. woody From halr at voltaire.com Thu Jun 30 09:46:08 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 Jun 2005 12:46:08 -0400 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support Message-ID: <1120149968.4371.81.camel@hal.voltaire.com> user_mad: Add receive side RMPP support Signed-off-by: Hal Rosenstock Index: user_mad.c =================================================================== --- user_mad.c (revision 2760) +++ user_mad.c (working copy) @@ -176,7 +176,7 @@ if (mad_recv_wc->wc->status != IB_WC_SUCCESS) goto out; - length = 256; /* until RMPP is supported */ + length = mad_recv_wc->mad_len; packet = kmalloc(sizeof *packet + length, GFP_KERNEL); if (!packet) goto out; @@ -184,7 +184,8 @@ memset(packet, 0, sizeof *packet + length); packet->length = length; - memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, length); + ib_coalesce_recv_mad(mad_recv_wc, packet->mad.data); + packet->mad.hdr.status = 0; packet->mad.hdr.qpn = cpu_to_be32(mad_recv_wc->wc->src_qp); packet->mad.hdr.lid = cpu_to_be16(mad_recv_wc->wc->slid); @@ -214,7 +215,7 @@ struct ib_umad_packet *packet; ssize_t ret; - if (count < sizeof (struct ib_user_mad) + 256) /* until RMPP supported */ + if (count < sizeof (struct ib_user_mad) + sizeof (struct ib_mad)) return -EINVAL; spin_lock_irq(&file->recv_lock); @@ -238,13 +239,14 @@ spin_unlock_irq(&file->recv_lock); if (count < packet->length + sizeof (struct ib_user_mad)) - ret = -EINVAL; + /* return length needed if too small */ + ret = packet->length + sizeof (struct ib_user_mad); else if (copy_to_user(buf, &packet->mad, packet->length + sizeof (struct ib_user_mad))) ret = -EFAULT; else ret = packet->length + sizeof (struct ib_user_mad); - if (ret < 0) { + if (ret < 0 || count < packet->length + sizeof (struct ib_user_mad)) { /* Requeue packet */ spin_lock_irq(&file->recv_lock); list_add(&packet->list, &file->recv_list); From caitlin.bestler at gmail.com Thu Jun 30 09:53:18 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 09:53:18 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <52slyzakdv.fsf@topspin.com> References: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408> <52slyzakdv.fsf@topspin.com> Message-ID: <469958e005063009536dc832fc@mail.gmail.com> We have to keep in mind that DAT was designed for a broader purpose than what is proposed for "RDMA verbs". It was designed to support transport *and* OS neutral applications that could be migrated easily in a number of ways: transport, form OS to OS, from user to kernel, etc. So in some ways DAT is like RPC and other middleware. It simplifies life for many applications, but it should not be the only options available. If the RDMA verb layer is well designed it will allow DAT (or IT-API) to be implemented simply and at low cost. Developers can then decided whether they prefer a higher layer API with more portability or the 'raw' interface that is closer to the wire. With the latter they get more control, but will have to respond to more explicit error condiditons, etc. That would end up looking something like: application ---------> DAT Layer -----------> RDMA Verbs --------> model-specific code \ ^ \ / \------------------------------------/ I don't think that trying to shoehorm the DAT Layer into *being* the RDMA verbs definition makes sense, especially if doing so involves sacrificing it's other objectives. It is, however, an excellent interim solution that allows time to properly specify and develop a usable verb layer RDMA interface. And DAT will remain a valuable option for application developers that prefer wider portability at least one layer of abstracting of the raw results into terms that are relevant to the application. From mshefty at ichips.intel.com Thu Jun 30 10:00:08 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 10:00:08 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <469958e005063009536dc832fc@mail.gmail.com> References: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408> <52slyzakdv.fsf@topspin.com> <469958e005063009536dc832fc@mail.gmail.com> Message-ID: <42C42518.9060709@ichips.intel.com> Caitlin Bestler wrote: > We have to keep in mind that DAT was designed for a broader > purpose than what is proposed for "RDMA verbs". It was designed > to support transport *and* OS neutral applications that could > be migrated easily in a number of ways: transport, form OS to > OS, from user to kernel, etc. Cod that's OS neutral is nice. It just doesn't belong in the Linux kernel. - Sean From robert.j.woodruff at intel.com Thu Jun 30 10:27:02 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 30 Jun 2005 10:27:02 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <469958e005063009536dc832fc@mail.gmail.com> Message-ID: Catlin wrote, >application ---------> DAT Layer -----------> RDMA Verbs --------> >model-specific code > \ ^ > \ / > \------------------------------------/ Tell me how you are going to make the CM transport independent. You cannot, they are simply different for IB and iWarp. We would have to do something like, rdma_verbs_connect(xport_type, ....) { if (xport_type == ib) { do_sa_queries() ib_connect() } else if (xport_type == iwarp) { do_iWarp_connect() } } Thus all you have done is added another layer, >application ---------> DAT Layer --> RDMA Verbs -> ib_specific_routines -> drv >model-specific code -> iwarp_specific_routines -> drv > \ ^ -> common routines -> drv > \ / > \------------------------/ Another approach would be just to: s/DAT/RDMA and make it as thin as possible >application ---------> RDMA Verbs -> ib_specific_routines -> drv >model-specific code -> iwarp_specific_routines -> drv > -> common routines -> drv > And we have achieved the same thing. From mshefty at ichips.intel.com Thu Jun 30 10:35:09 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 10:35:09 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <42C42518.9060709@ichips.intel.com> References: <1AC79F16F5C5284499BB9591B33D6F0004D1F645@orsmsx408> <52slyzakdv.fsf@topspin.com> <469958e005063009536dc832fc@mail.gmail.com> <42C42518.9060709@ichips.intel.com> Message-ID: <42C42D4D.3090106@ichips.intel.com> Sean Hefty wrote: > Caitlin Bestler wrote: > >> We have to keep in mind that DAT was designed for a broader >> purpose than what is proposed for "RDMA verbs". It was designed >> to support transport *and* OS neutral applications that could >> be migrated easily in a number of ways: transport, form OS to >> OS, from user to kernel, etc. > > > Cod that's OS neutral is nice. It just doesn't belong in the Linux kernel. Before I get another fish comment... "Code" even... :) From caitlin.bestler at gmail.com Thu Jun 30 10:37:06 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 30 Jun 2005 10:37:06 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: References: <469958e005063009536dc832fc@mail.gmail.com> Message-ID: <469958e005063010377ac56d66@mail.gmail.com> On 6/30/05, Bob Woodruff wrote: > Catlin wrote, > >application ---------> DAT Layer -----------> RDMA Verbs --------> > >model-specific code > > \ ^ > > \ / > > \------------------------------------/ > > > Tell me how you are going to make the CM transport independent. > You cannot, they are simply different for IB and iWarp. > > We would have to do something like, > > rdma_verbs_connect(xport_type, ....) > { > if (xport_type == ib) { > do_sa_queries() > ib_connect() > } > else if (xport_type == iwarp) { > do_iWarp_connect() > } > } > > Thus all you have done is added another layer, > True. My assumption is that each transport would have its own CM module with similar, but not identical APIs. An application wants to go direct it could do the if/else ifs on its own. We could define polymorphic interface which all CMs MUST support, but each would have its own transport specific extensions. From rolandd at cisco.com Thu Jun 30 10:49:41 2005 From: rolandd at cisco.com (Roland Dreier) Date: Thu, 30 Jun 2005 10:49:41 -0700 Subject: [openib-general] Re: IP addressing on InfiniBand networks (Caitlin Bestler) In-Reply-To: <6.2.0.14.2.20050630094418.029b9008@esmail.cup.hp.com> (Michael Krause's message of "Thu, 30 Jun 2005 09:44:32 -0700") References: <6.2.0.14.2.20050630094418.029b9008@esmail.cup.hp.com> Message-ID: <52r7ej68ga.fsf@topspin.com> Michael> Being the person who led the addressing definition for Michael> IB, I can state quite clearly that GID are NOT IPv6 Michael> addresses. They were intentionally defined to have a Michael> similar look-n-feel since they were derived in large part Michael> from Future I/O which had them as real IPv6 addresses. Michael> But again, they are NOT IPv6 addresses. The IBA spec seems to have a different idea. In fact chapter 4 says: "A GID is a valid 128-bit IPv6 address (per RFC 2373)...." - R. From robert.j.woodruff at intel.com Thu Jun 30 11:16:05 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 30 Jun 2005 11:16:05 -0700 Subject: [openib-general] comments on DAT registry in OpenIB In-Reply-To: <469958e005063010377ac56d66@mail.gmail.com> Message-ID: Carlin wrote, >True. >My assumption is that each transport would have its >own CM module with similar, but not identical APIs. >An application wants to go direct it could do the >if/else ifs on its own. If the application will have to have if/else for some functions, then it already has to be aware of the specific transport, so one option is to have if/else for all calls. There are however some functions, like posting work requests and creating some resources that are probably similar enough to have a common routine, but it may require the routine to make a check and vector to the appropriate transport, which will add latency. A scenerio something like, if(IB) { ib_sa_query() ib_connect() } elseif (iWarp) { iWarp_connect() } rdma_common_verb_create_pd() rdma_common_verb_create_wq() . . rdma_common_verb_post_wqe() i.e., in some cases you might end up with, rdma_common_verb_create_pd(xport_type) { switch (xport_type) case IB: ib_create_pd() case IWARP: iwarp_create_pd() . . . } From rolandd at cisco.com Thu Jun 30 11:34:02 2005 From: rolandd at cisco.com (Roland Dreier) Date: Thu, 30 Jun 2005 11:34:02 -0700 Subject: [openib-general] [PATCH] IPoIB: handle TX ring index wrap Message-ID: <52k6kb66ed.fsf@topspin.com> It was pointed out to me that comparing tx_tail and tx_head directly is not safe, since they're unsigned ints that will eventually wrap. Therefore we need to take their difference as signed ints and check that. Does this look right? Did I miss any other spots? - R. Index: infiniband/ulp/ipoib/ipoib_ib.c =================================================================== --- infiniband/ulp/ipoib/ipoib_ib.c (revision 2710) +++ infiniband/ulp/ipoib/ipoib_ib.c (working copy) @@ -82,7 +82,7 @@ void ipoib_free_ah(struct kref *kref) unsigned long flags; - if (ah->last_send <= priv->tx_tail) { + if ((int) priv->tx_tail - (int) ah->last_send >= 0) { ipoib_dbg(priv, "Freeing ah %p\n", ah->ah); ib_destroy_ah(ah->ah); kfree(ah); @@ -356,7 +356,7 @@ static void __ipoib_reap_ah(struct net_d spin_lock_irq(&priv->lock); list_for_each_entry_safe(ah, tah, &priv->dead_ahs, list) - if (ah->last_send <= priv->tx_tail) { + if ((int) priv->tx_tail - (int) ah->last_send >= 0) { list_del(&ah->list); list_add_tail(&ah->list, &remove_list); } @@ -487,7 +487,7 @@ int ipoib_ib_dev_stop(struct net_device * assume the HW is wedged and just free up * all our pending work requests. */ - while (priv->tx_tail < priv->tx_head) { + while ((int) priv->tx_tail - (int) priv->tx_head < 0) { tx_req = &priv->tx_ring[priv->tx_tail & (IPOIB_TX_RING_SIZE - 1)]; dma_unmap_single(priv->ca->dma_device, From jlentini at netapp.com Thu Jun 30 11:38:43 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 30 Jun 2005 14:38:43 -0400 (EDT) Subject: [openib-general] IP addressing on InfiniBand networks In-Reply-To: <521x6lf6vf.fsf@topspin.com> References: <52br5qtgk9.fsf@topspin.com> <521x6lf6vf.fsf@topspin.com> Message-ID: On Wed, 29 Jun 2005, Roland Dreier wrote: > James> - IB services (both kernel and user space) will be configured using > James> IP addresses. By IB services, I'm referring to protocols that are > James> layered directly on top of the InfiniBand protocols (e.g. SDP, > James> NFS-RDMA, iSER, etc.). > James> - IB services will resolve IP addresses to IB addresses using IPoIB > James> ARP (all IB nodes would be required to support IPoIB). > James> - each IB service would place appropriate IP address information in > James> its protocol messages > > That's mostly right, except that I would leave it up to the individual > protocols if and how they want to use IP addressing. For example, the > SCSI RDMA Protocol (SRP) published by INCITS T10 does not use IP > addresses in any way, and it seems that should be allowed and supported. Thanks for the clarification. I understand your position now. I think it is worth reiterating the two major dependencies imposed by this solution: - IB nodes must run IPoIB - ULPs must add IP addressing information on IB networks Of those two, I see the second as the most difficult. The standardization of additional protocol messages may take a significant amount of time. From rolandd at cisco.com Thu Jun 30 11:45:51 2005 From: rolandd at cisco.com (Roland Dreier) Date: Thu, 30 Jun 2005 11:45:51 -0700 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <1120149968.4371.81.camel@hal.voltaire.com> (Hal Rosenstock's message of "30 Jun 2005 12:46:08 -0400") References: <1120149968.4371.81.camel@hal.voltaire.com> Message-ID: <52d5q365uo.fsf@topspin.com> Looking at the code, I have to say that I don't like the strategy of returning the actual length of the MAD but not copying anything if the MAD is too big. It seems error prone to return a length bigger than the user passed in to read(), and it doesn't feel right either. I understand and agree with the sentiment of not wanting to add another ioctl() to get the length. Instead, how about returning a ib_user_mad_hdr with a status of ENOSPC and putting the actual length somewhere. I'm not sure if it's better to change the ABI and add a length field to ib_user_mad_hdr, or if we want to return the first 36 bytes of an RMPP MAD so the user can figure out the correct length. Also, since we check if the MAD at the head of the receive queue won't fit in the user's buffer when we're about to pop it off the queue, instead of dequeuing and then requeuing it, we can just leave it at the head of the queue. (I'm not sure if this buys us anything but it seems there might be a race MADs get out of order) Sorry for not bringing this up when you asked about how to handle too-big MADs a few days ago -- I didn't understand your question really. - R. From jlentini at netapp.com Thu Jun 30 12:24:07 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 30 Jun 2005 15:24:07 -0400 (EDT) Subject: [openib-general] RE: [dat-discussions] comments on DAT registry in OpenIB In-Reply-To: <1120146012.358.2.camel@duffman> References: <8508251A6FC08A489844A94261D3693A057C5E@fiona.siliquent.com> <1120146012.358.2.camel@duffman> Message-ID: On Thu, 30 Jun 2005, Tom Duffy wrote: > On Wed, 2005-06-29 at 20:19 -0700, Caitlin Bestler wrote: >> Source compatability for an existing Provider is *not* maintained >> by OpenIB kDAPL because there are fields *missing* from the >> Provider Info. That *will* result in a compilation error. > > I have no problem with there being a header file with allows for source > compatibility. This header would be maintained by OpenIB, but never > included in upstream. I think James has a start of such a header called > kdat.h. It needs a bunch more to make it follow the DAT conventions. Tom is correct. The kdat.h file is intended as a way to hide as many (ideally all) differences. From libor at topspin.com Thu Jun 30 12:42:12 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 30 Jun 2005 12:42:12 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <42C41A28.4000809@ichips.intel.com>; from mshefty@ichips.intel.com on Thu, Jun 30, 2005 at 09:13:28AM -0700 References: <42C1BA15.7060205@ichips.intel.com> <20050629111038.G26240@topspin.com> <42C2E573.5010304@ichips.intel.com> <20050629171221.I26240@topspin.com> <42C41A28.4000809@ichips.intel.com> Message-ID: <20050630124212.A29020@topspin.com> On Thu, Jun 30, 2005 at 09:13:28AM -0700, Sean Hefty wrote: > Libor Michalek wrote: > > Assume that the userspace 'struct ib_cm_event' contains the cm_id as > > well as a new 'u64 context' which is inherited from the cm_id, and is > > set at the time of the cm_id creation. This is what I'm assuming that > > Arlin would like to see. > > > > In the case of two threads accessing the CM at once there's a race > > condition if you are going to use the 'context' variable as a pointer > > to memory: > > > > Thread 1 Thread 2 > > ------------------------------------- ----------------------------------- > > cm_object = malloc(sizeof(*cm_object) > > ib_cm_create_id(&cm_object->cm_id, > > (u64)cm_object) > > > > ib_cm_event_get(&event) > > ib_cm_destroy_id(cm_object->cm_id) > > free(cm_object); > > process_event((void *)event->context); > > I see. This appears to come from a difference between the event reporting > model used by the kernel CM versus the usermode CM (callback versus > calldown). Do you block the destroy on a lock while a callback for that cm_id is active? I wouldn't say that the difference is attributed to callback vs. calldown, in both cases it's a matter of serializing the destroy with the event. > Maybe there's a way to assist the user here. Can we report a > destruction event, or require a second call to indicate that an event has > been processed? A destruction event could work, but with some limits which might make it impracticle. The user would have to be really carefull not to do _anything_ with the object after calling destroy, and only cleanup in the same thread that is used to get the destroy completion event. The destroy completion event could be retreived and processed before the original destroy call returns. Also, the user would need to make sure that they are getting events in a _single_ thread, since multiple event get threads could pose the same problem as before. Blocking on the destroy seems like it could be error prone, that you could easily deadlock the user, who probably has a lock around the object which contains the cm_id... We could build the serialization table for the API consumer, have all cm_id calls and events go through a level of indirection in a table locked against multiple threads. This was the way we ended up doing it in our old code for the userCM that we used for uDAPL. I had left this out since it seems reasonable that not all apps would want/need this guarantee from the API, and that they could implement it themselves if they did want it... I could be wrong. -Libor From mshefty at ichips.intel.com Thu Jun 30 12:55:23 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 12:55:23 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <20050630124212.A29020@topspin.com> References: <42C1BA15.7060205@ichips.intel.com> <20050629111038.G26240@topspin.com> <42C2E573.5010304@ichips.intel.com> <20050629171221.I26240@topspin.com> <42C41A28.4000809@ichips.intel.com> <20050630124212.A29020@topspin.com> Message-ID: <42C44E2B.2000703@ichips.intel.com> Libor Michalek wrote: >>I see. This appears to come from a difference between the event reporting >>model used by the kernel CM versus the usermode CM (callback versus >>calldown). > > Do you block the destroy on a lock while a callback for that cm_id > is active? I wouldn't say that the difference is attributed to callback > vs. calldown, in both cases it's a matter of serializing the destroy > with the event. Yes - the destroy call in the kernel blocks while there's a callback in progress. After destroy returns, the CM guarantees that no additional callbacks will be received by the user. The blocking destroy call was the reason for letting the user destroy the cm_id by returning a non-zero value from the callback. >> Maybe there's a way to assist the user here. Can we report a >>destruction event, or require a second call to indicate that an event has >>been processed? > > A destruction event could work, but with some limits which might make > it impracticle. The user would have to be really carefull not to do > _anything_ with the object after calling destroy, and only cleanup in > the same thread that is used to get the destroy completion event. The > destroy completion event could be retreived and processed before the > original destroy call returns. Also, the user would need to make sure > that they are getting events in a _single_ thread, since multiple event > get threads could pose the same problem as before. I agree that the destroy event could occur before destroy returns, so the user would need to be careful there. Arlin mentioned that there's a put event call that needs to be invoked after getting an event. If so, then the CM can track the number of outstanding events that are in process. It could then either delay the destroy call while an event is outstanding, or delay reporting the destroy event until all events have been processed. This should handle the multi-thread issues. It may also make sense to have the uCM serialize all events to a single cm_id anyway. I ended up doing this in the kernel, which simplified the application's event handling. Otherwise, the events can end up being processed out of order. E.g. a REJ is reported, followed by a MRA to a REQ. > We could build the serialization table for the API consumer, have > all cm_id calls and events go through a level of indirection in a > table locked against multiple threads. This was the way we ended up > doing it in our old code for the userCM that we used for uDAPL. I > had left this out since it seems reasonable that not all apps would > want/need this guarantee from the API, and that they could implement > it themselves if they did want it... I could be wrong. I need to spend more time looking at the uCM API/implementation to see if there's a way to help protect against reporting/processing events. - Sean From tduffy at sun.com Thu Jun 30 13:58:04 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 30 Jun 2005 13:58:04 -0700 Subject: [openib-general] [PATCH] SDP: use linux/list.h for listen conns Message-ID: <1120165084.29522.7.camel@duffman> This patch changes sdp to use Linux native lists for the listen conn list. Signed-off-by: Tom Duffy Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2763) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -291,21 +291,11 @@ int sdp_inet_listen_start(struct sdp_soc conn->state = SDP_CONN_ST_LISTEN; conn->accept_next = conn; conn->accept_prev = conn; - /* - * table lock - */ - spin_lock_irqsave(&dev_root_s.listen_lock, flags); - /* - * insert into listening list. - */ - conn->lstn_next = dev_root_s.listen_list; - dev_root_s.listen_list = conn; - conn->lstn_p_next = &dev_root_s.listen_list; - - if (conn->lstn_next) - conn->lstn_next->lstn_p_next = &conn->lstn_next; + spin_lock_irqsave(&dev_root_s.listen_lock, flags); + list_add(&conn->lstn_next, &dev_root_s.listen_list); spin_unlock_irqrestore(&dev_root_s.listen_lock, flags); + return 0; } @@ -323,22 +313,11 @@ int sdp_inet_listen_stop(struct sdp_sock } listen_conn->state = SDP_CONN_ST_CLOSED; - /* - * table lock - */ - spin_lock_irqsave(&dev_root_s.listen_lock, flags); - /* - * remove from listening list. - */ - if (listen_conn->lstn_next) - listen_conn->lstn_next->lstn_p_next = listen_conn->lstn_p_next; - - *(listen_conn->lstn_p_next) = listen_conn->lstn_next; - - listen_conn->lstn_p_next = NULL; - listen_conn->lstn_next = NULL; + spin_lock_irqsave(&dev_root_s.listen_lock, flags); + list_del(&listen_conn->lstn_next); spin_unlock_irqrestore(&dev_root_s.listen_lock, flags); + /* * reject and delete all pending connections */ @@ -376,7 +355,7 @@ struct sdp_sock *sdp_inet_listen_lookup( /* * first find a listening connection */ - for (conn = dev_root_s.listen_list; conn; conn = conn->lstn_next) + list_for_each_entry(conn, &dev_root_s.listen_list, lstn_next) if (port == conn->src_port && (INADDR_ANY == conn->src_addr || addr == conn->src_addr)) { sdp_conn_hold(conn); @@ -1936,7 +1915,7 @@ int sdp_conn_table_init(int proto_family /* * list */ - dev_root_s.listen_list = NULL; + INIT_LIST_HEAD(&dev_root_s.listen_list); dev_root_s.bind_list = NULL; spin_lock_init(&dev_root_s.sock_lock); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (revision 2763) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -36,6 +36,8 @@ #ifndef _SDP_CONN_H #define _SDP_CONN_H +#include + #include "sdp_advt.h" #include "sdp_iocb.h" #include "sdp_dev.h" @@ -336,8 +338,7 @@ struct sdp_sock { /* * table managment */ - struct sdp_sock *lstn_next; /* next conn in the chain */ - struct sdp_sock **lstn_p_next; /* previous next conn in the chain */ + struct list_head lstn_next; struct sdp_sock *bind_next; /* next conn in the chain */ struct sdp_sock **bind_p_next; /* previous next conn in the chain */ Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (revision 2763) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (working copy) @@ -43,6 +43,7 @@ #include #include #include +#include #include /* * sdp types @@ -186,7 +187,7 @@ struct sdev_root { /* * connection managment */ - struct sdp_sock *listen_list; /* list of listening connections */ + struct list_head listen_list; struct sdp_sock *bind_list; /* connections bound to a port. */ /* * list locks From sean.hefty at intel.com Thu Jun 30 14:22:47 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 14:22:47 -0700 Subject: [openib-general] [PATCH] [CM] verify cm_id state before allocating messages Message-ID: The following patch moves verification of the cm_id state higher up in the CM function calls. This fixes an issue trying to allocate a message for a cm_id that may be in an invalid state, which could result in a crash. This problem was first reported by Hal a while ago, but was recently hit by Arlin running a usermode application. Arlin, can you verify that this fixes your issues? Signed-off-by: Sean Hefty Index: cm.c =================================================================== --- cm.c (revision 2759) +++ cm.c (working copy) @@ -1348,12 +1348,17 @@ int ib_send_cm_rep(struct ib_cm_id *cm_i int ret; if (param->private_data && - param->private_data_len > IB_CM_REP_PRIVATE_DATA_SIZE) { + param->private_data_len > IB_CM_REP_PRIVATE_DATA_SIZE) + return -EINVAL; + + cm_id_priv = container_of(cm_id, struct cm_id_private, id); + spin_lock_irqsave(&cm_id_priv->lock, flags); + if (cm_id->state != IB_CM_REQ_RCVD && + cm_id->state != IB_CM_MRA_REQ_SENT) { ret = -EINVAL; goto out; } - cm_id_priv = container_of(cm_id, struct cm_id_private, id); ret = cm_alloc_msg(cm_id_priv, &msg); if (ret) goto out; @@ -1363,18 +1368,12 @@ int ib_send_cm_rep(struct ib_cm_id *cm_i msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; msg->context[1] = (void *) (unsigned long) IB_CM_REP_SENT; - spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id->state == IB_CM_REQ_RCVD || - cm_id->state == IB_CM_MRA_REQ_SENT) - ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); - else - ret = -EINVAL; - + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); if (ret) { spin_unlock_irqrestore(&cm_id_priv->lock, flags); cm_free_msg(msg); - goto out; + return ret; } cm_id->state = IB_CM_REP_SENT; @@ -1383,8 +1382,8 @@ int ib_send_cm_rep(struct ib_cm_id *cm_i cm_id_priv->responder_resources = param->responder_resources; cm_id_priv->rq_psn = cm_rep_get_starting_psn(rep_msg); cm_id_priv->local_qpn = cm_rep_get_local_qpn(rep_msg); - spin_unlock_irqrestore(&cm_id_priv->lock, flags); -out: + +out: spin_unlock_irqrestore(&cm_id_priv->lock, flags); return ret; } EXPORT_SYMBOL(ib_send_cm_rep); @@ -1416,39 +1415,41 @@ int ib_send_cm_rtu(struct ib_cm_id *cm_i if (private_data && private_data_len > IB_CM_RTU_PRIVATE_DATA_SIZE) return -EINVAL; + data = cm_copy_private_data(private_data, private_data_len); + if (IS_ERR(data)) + return PTR_ERR(data); + cm_id_priv = container_of(cm_id, struct cm_id_private, id); + spin_lock_irqsave(&cm_id_priv->lock, flags); + if (cm_id->state != IB_CM_REP_RCVD && + cm_id->state != IB_CM_MRA_REP_SENT) { + ret = -EINVAL; + goto error; + } + ret = cm_alloc_msg(cm_id_priv, &msg); if (ret) - return ret; - - data = cm_copy_private_data(private_data, private_data_len); - if (IS_ERR(data)) { - ret = PTR_ERR(data); - goto error1; - } + goto error; cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv, private_data, private_data_len); - spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id->state == IB_CM_REP_RCVD || - cm_id->state == IB_CM_MRA_REP_SENT) - ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); - else - ret = -EINVAL; - - if (ret) - goto error2; + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); + if (ret) { + spin_unlock_irqrestore(&cm_id_priv->lock, flags); + cm_free_msg(msg); + kfree(data); + return ret; + } cm_id->state = IB_CM_ESTABLISHED; cm_set_private_data(cm_id_priv, data, private_data_len); spin_unlock_irqrestore(&cm_id_priv->lock, flags); return 0; -error2: spin_unlock_irqrestore(&cm_id_priv->lock, flags); +error: spin_unlock_irqrestore(&cm_id_priv->lock, flags); kfree(data); -error1: cm_free_msg(msg); return ret; } EXPORT_SYMBOL(ib_send_cm_rtu); @@ -1691,38 +1692,41 @@ int ib_send_cm_dreq(struct ib_cm_id *cm_ struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; - int msg_ret, ret; + int ret; if (private_data && private_data_len > IB_CM_DREQ_PRIVATE_DATA_SIZE) return -EINVAL; cm_id_priv = container_of(cm_id, struct cm_id_private, id); - msg_ret = cm_alloc_msg(cm_id_priv, &msg); - if (!msg_ret) { - cm_format_dreq((struct cm_dreq_msg *) msg->mad, cm_id_priv, - private_data, private_data_len); - msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; - msg->context[1] = (void *) (unsigned long) IB_CM_DREQ_SENT; - } - spin_lock_irqsave(&cm_id_priv->lock, flags); if (cm_id->state != IB_CM_ESTABLISHED) { - spin_unlock_irqrestore(&cm_id_priv->lock, flags); ret = -EINVAL; goto out; } - ret = msg_ret ? msg_ret : - ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); - if (!ret) { - cm_id->state = IB_CM_DREQ_SENT; - cm_id_priv->msg = msg; - } else + + ret = cm_alloc_msg(cm_id_priv, &msg); + if (ret) { cm_enter_timewait(cm_id_priv); - spin_unlock_irqrestore(&cm_id_priv->lock, flags); -out: - if (!msg_ret && ret) + goto out; + } + + cm_format_dreq((struct cm_dreq_msg *) msg->mad, cm_id_priv, + private_data, private_data_len); + msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; + msg->context[1] = (void *) (unsigned long) IB_CM_DREQ_SENT; + + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); + if (ret) { + cm_enter_timewait(cm_id_priv); + spin_unlock_irqrestore(&cm_id_priv->lock, flags); cm_free_msg(msg); + return ret; + } + + cm_id->state = IB_CM_DREQ_SENT; + cm_id_priv->msg = msg; +out: spin_unlock_irqrestore(&cm_id_priv->lock, flags); return ret; } EXPORT_SYMBOL(ib_send_cm_dreq); @@ -1754,45 +1758,37 @@ int ib_send_cm_drep(struct ib_cm_id *cm_ if (private_data && private_data_len > IB_CM_DREP_PRIVATE_DATA_SIZE) return -EINVAL; - cm_id_priv = container_of(cm_id, struct cm_id_private, id); - ret = cm_alloc_msg(cm_id_priv, &msg); - if (ret) - goto error1; - data = cm_copy_private_data(private_data, private_data_len); - if (IS_ERR(data)) { - ret = PTR_ERR(data); - goto error2; - } - - cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv, - private_data, private_data_len); + if (IS_ERR(data)) + return PTR_ERR(data); + cm_id_priv = container_of(cm_id, struct cm_id_private, id); spin_lock_irqsave(&cm_id_priv->lock, flags); if (cm_id->state != IB_CM_DREQ_RCVD) { spin_unlock_irqrestore(&cm_id_priv->lock, flags); - cm_free_msg(msg); kfree(data); return -EINVAL; } + cm_set_private_data(cm_id_priv, data, private_data_len); + cm_enter_timewait(cm_id_priv); + + ret = cm_alloc_msg(cm_id_priv, &msg); + if (ret) + goto out; + + cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv, + private_data, private_data_len); + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); - if (ret) + if (ret) { + spin_unlock_irqrestore(&cm_id_priv->lock, flags); cm_free_msg(msg); + return ret; + } - cm_set_private_data(cm_id_priv, data, private_data_len); - cm_enter_timewait(cm_id_priv); - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - return ret; - -error2: - cm_free_msg(msg); -error1: - spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id->state == IB_CM_DREQ_RCVD) - cm_enter_timewait(cm_id_priv); - spin_unlock_irqrestore(&cm_id_priv->lock, flags); +out: spin_unlock_irqrestore(&cm_id_priv->lock, flags); return ret; } EXPORT_SYMBOL(ib_send_cm_drep); @@ -1912,18 +1908,13 @@ int ib_send_cm_rej(struct ib_cm_id *cm_i struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; - int msg_ret, ret; + int ret; if ((private_data && private_data_len > IB_CM_REJ_PRIVATE_DATA_SIZE) || (ari && ari_length > IB_CM_REJ_ARI_LENGTH)) return -EINVAL; cm_id_priv = container_of(cm_id, struct cm_id_private, id); - msg_ret = cm_alloc_msg(cm_id_priv, &msg); - if (!msg_ret) - cm_format_rej((struct cm_rej_msg *) msg->mad, cm_id_priv, - reason, ari, ari_length, private_data, - private_data_len); spin_lock_irqsave(&cm_id_priv->lock, flags); switch (cm_id->state) { @@ -1933,25 +1924,38 @@ int ib_send_cm_rej(struct ib_cm_id *cm_i case IB_CM_MRA_REQ_SENT: case IB_CM_REP_RCVD: case IB_CM_MRA_REP_SENT: + ret = cm_alloc_msg(cm_id_priv, &msg); + if (!ret) + cm_format_rej((struct cm_rej_msg *) msg->mad, + cm_id_priv, reason, ari, ari_length, + private_data, private_data_len); + cm_reset_to_idle(cm_id_priv); break; case IB_CM_REP_SENT: case IB_CM_MRA_REP_RCVD: + ret = cm_alloc_msg(cm_id_priv, &msg); + if (!ret) + cm_format_rej((struct cm_rej_msg *) msg->mad, + cm_id_priv, reason, ari, ari_length, + private_data, private_data_len); + cm_enter_timewait(cm_id_priv); break; default: - spin_unlock_irqrestore(&cm_id_priv->lock, flags); ret = -EINVAL; goto out; } - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - ret = msg_ret ? msg_ret : - ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); -out: - if (!msg_ret && ret) + if (ret) + goto out; + + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); + if (ret) cm_free_msg(msg); + +out: spin_unlock_irqrestore(&cm_id_priv->lock, flags); return ret; } EXPORT_SYMBOL(ib_send_cm_rej); @@ -2078,20 +2082,19 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i if (private_data && private_data_len > IB_CM_MRA_PRIVATE_DATA_SIZE) return -EINVAL; - cm_id_priv = container_of(cm_id, struct cm_id_private, id); - ret = cm_alloc_msg(cm_id_priv, &msg); - if (ret) - return ret; - data = cm_copy_private_data(private_data, private_data_len); - if (IS_ERR(data)) { - ret = PTR_ERR(data); - goto error1; - } + if (IS_ERR(data)) + return PTR_ERR(data); + + cm_id_priv = container_of(cm_id, struct cm_id_private, id); spin_lock_irqsave(&cm_id_priv->lock, flags); switch(cm_id_priv->id.state) { case IB_CM_REQ_RCVD: + ret = cm_alloc_msg(cm_id_priv, &msg); + if (ret) + goto error1; + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, CM_MSG_RESPONSE_REQ, service_timeout, private_data, private_data_len); @@ -2102,6 +2105,10 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i cm_id->state = IB_CM_MRA_REQ_SENT; break; case IB_CM_REP_RCVD: + ret = cm_alloc_msg(cm_id_priv, &msg); + if (ret) + goto error1; + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, CM_MSG_RESPONSE_REP, service_timeout, private_data, private_data_len); @@ -2112,6 +2119,10 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i cm_id->state = IB_CM_MRA_REP_SENT; break; case IB_CM_ESTABLISHED: + ret = cm_alloc_msg(cm_id_priv, &msg); + if (ret) + goto error1; + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, CM_MSG_RESPONSE_OTHER, service_timeout, private_data, private_data_len); @@ -2123,16 +2134,20 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i break; default: ret = -EINVAL; - goto error2; + goto error1; } cm_id_priv->service_timeout = service_timeout; cm_set_private_data(cm_id_priv, data, private_data_len); spin_unlock_irqrestore(&cm_id_priv->lock, flags); return 0; +error1: spin_unlock_irqrestore(&cm_id_priv->lock, flags); + kfree(data); + return ret; + error2: spin_unlock_irqrestore(&cm_id_priv->lock, flags); kfree(data); -error1: cm_free_msg(msg); + cm_free_msg(msg); return ret; } EXPORT_SYMBOL(ib_send_cm_mra); @@ -2260,6 +2275,13 @@ int ib_send_cm_lap(struct ib_cm_id *cm_i return -EINVAL; cm_id_priv = container_of(cm_id, struct cm_id_private, id); + spin_lock_irqsave(&cm_id_priv->lock, flags); + if (cm_id->state != IB_CM_ESTABLISHED || + cm_id->lap_state != IB_CM_LAP_IDLE) { + ret = -EINVAL; + goto out; + } + ret = cm_alloc_msg(cm_id_priv, &msg); if (ret) goto out; @@ -2269,24 +2291,18 @@ int ib_send_cm_lap(struct ib_cm_id *cm_i msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; msg->context[1] = (void *) (unsigned long) IB_CM_ESTABLISHED; - spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id->state == IB_CM_ESTABLISHED && - cm_id->lap_state == IB_CM_LAP_IDLE) - ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); - else - ret = -EINVAL; - + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); if (ret) { spin_unlock_irqrestore(&cm_id_priv->lock, flags); cm_free_msg(msg); - goto out; + return ret; } cm_id->lap_state = IB_CM_LAP_SENT; cm_id_priv->msg = msg; - spin_unlock_irqrestore(&cm_id_priv->lock, flags); -out: + +out: spin_unlock_irqrestore(&cm_id_priv->lock, flags); return ret; } EXPORT_SYMBOL(ib_send_cm_lap); @@ -2420,30 +2436,30 @@ int ib_send_cm_apr(struct ib_cm_id *cm_i return -EINVAL; cm_id_priv = container_of(cm_id, struct cm_id_private, id); + spin_lock_irqsave(&cm_id_priv->lock, flags); + if (cm_id->state != IB_CM_ESTABLISHED || + (cm_id->lap_state != IB_CM_LAP_RCVD && + cm_id->lap_state != IB_CM_MRA_LAP_SENT)) { + ret = -EINVAL; + goto out; + } + ret = cm_alloc_msg(cm_id_priv, &msg); if (ret) goto out; cm_format_apr((struct cm_apr_msg *) msg->mad, cm_id_priv, status, info, info_length, private_data, private_data_len); - - spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id->state == IB_CM_ESTABLISHED && - (cm_id->lap_state == IB_CM_LAP_RCVD || - cm_id->lap_state == IB_CM_MRA_LAP_SENT)) - ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); - else - ret = -EINVAL; - + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); if (ret) { spin_unlock_irqrestore(&cm_id_priv->lock, flags); cm_free_msg(msg); - goto out; + return ret; } + cm_id->lap_state = IB_CM_LAP_IDLE; - spin_unlock_irqrestore(&cm_id_priv->lock, flags); -out: +out: spin_unlock_irqrestore(&cm_id_priv->lock, flags); return ret; } EXPORT_SYMBOL(ib_send_cm_apr); @@ -2703,24 +2719,24 @@ int ib_send_cm_sidr_rep(struct ib_cm_id return -EINVAL; cm_id_priv = container_of(cm_id, struct cm_id_private, id); + spin_lock_irqsave(&cm_id_priv->lock, flags); + if (cm_id->state != IB_CM_SIDR_REQ_RCVD) { + ret = -EINVAL; + goto error; + } + ret = cm_alloc_msg(cm_id_priv, &msg); if (ret) - goto out; + goto error; cm_format_sidr_rep((struct cm_sidr_rep_msg *) msg->mad, cm_id_priv, param); - - spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id->state == IB_CM_SIDR_REQ_RCVD) - ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); - else - ret = -EINVAL; - + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); if (ret) { spin_unlock_irqrestore(&cm_id_priv->lock, flags); cm_free_msg(msg); - goto out; + return ret; } cm_id->state = IB_CM_IDLE; spin_unlock_irqrestore(&cm_id_priv->lock, flags); @@ -2728,7 +2744,9 @@ int ib_send_cm_sidr_rep(struct ib_cm_id spin_lock_irqsave(&cm.lock, flags); rb_erase(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table); spin_unlock_irqrestore(&cm.lock, flags); -out: + return 0; + +error: spin_unlock_irqrestore(&cm_id_priv->lock, flags); return ret; } EXPORT_SYMBOL(ib_send_cm_sidr_rep); From mshefty at ichips.intel.com Thu Jun 30 14:32:22 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 14:32:22 -0700 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <52d5q365uo.fsf@topspin.com> References: <1120149968.4371.81.camel@hal.voltaire.com> <52d5q365uo.fsf@topspin.com> Message-ID: <42C464E6.8040406@ichips.intel.com> Roland Dreier wrote: > I understand and agree with the sentiment of not wanting to add > another ioctl() to get the length. Instead, how about returning a > ib_user_mad_hdr with a status of ENOSPC and putting the actual length > somewhere. I'm not sure if it's better to change the ABI and add a > length field to ib_user_mad_hdr, or if we want to return the first 36 > bytes of an RMPP MAD so the user can figure out the correct length. Unless the MAD layer modifies the received MAD, the length may not be set in the header. Setting this doesn't seem like a big deal. If it is set, then I'm guessing that we'd want to set the PayloadLength to the value indicated by the spec, but that's not the easiest value to use in order to determine the size of the read. Given that, I think that it makes more sense to add a length field to the ib_user_mad_hdr. - Sean From halr at voltaire.com Thu Jun 30 14:28:49 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 Jun 2005 17:28:49 -0400 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <52d5q365uo.fsf@topspin.com> References: <1120149968.4371.81.camel@hal.voltaire.com> <52d5q365uo.fsf@topspin.com> Message-ID: <1120166929.4371.606.camel@hal.voltaire.com> On Thu, 2005-06-30 at 14:45, Roland Dreier wrote: > Looking at the code, I have to say that I don't like the strategy of > returning the actual length of the MAD but not copying anything if the > MAD is too big. It seems error prone to return a length bigger than > the user passed in to read(), and it doesn't feel right either. > > I understand and agree with the sentiment of not wanting to add > another ioctl() to get the length. Instead, how about returning a > ib_user_mad_hdr with a status of ENOSPC and putting the actual length > somewhere. I'm not sure if it's better to change the ABI and add a > length field to ib_user_mad_hdr, or if we want to return the first 36 > bytes of an RMPP MAD so the user can figure out the correct length. > > Also, since we check if the MAD at the head of the receive queue won't > fit in the user's buffer when we're about to pop it off the queue, > instead of dequeuing and then requeuing it, we can just leave it at > the head of the queue. (I'm not sure if this buys us anything but it > seems there might be a race MADs get out of order) > > Sorry for not bringing this up when you asked about how to handle > too-big MADs a few days ago -- I didn't understand your question > really. You understand it now and answered it well :-) I'm not wedded to the read approach and you have offered 2 alternatives. Unfortunately, I don't think the length in the first segment is reliable (it works for OpenIB but may not for other RMPP implementations) so I would rather not rely on that. So I guess this means an ABI change :-( Do you concur ? If so, I will work on that approach. -- Hal From ftillier at silverstorm.com Thu Jun 30 14:41:56 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 30 Jun 2005 14:41:56 -0700 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <42C464E6.8040406@ichips.intel.com> Message-ID: <000301c57dbc$8aa5ea20$6501a8c0@infiniconsys.com> > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > Sent: Thursday, June 30, 2005 2:32 PM > > Roland Dreier wrote: > > I understand and agree with the sentiment of not wanting to add > > another ioctl() to get the length. Instead, how about returning a > > ib_user_mad_hdr with a status of ENOSPC and putting the actual length > > somewhere. I'm not sure if it's better to change the ABI and add a > > length field to ib_user_mad_hdr, or if we want to return the first 36 > > bytes of an RMPP MAD so the user can figure out the correct length. > > Unless the MAD layer modifies the received MAD, the length may not be set in > the header. Setting this doesn't seem like a big deal. If it is set, then > I'm guessing that we'd want to set the PayloadLength to the value indicated > by the spec, but that's not the easiest value to use in order to determine > the size of the read. > > Given that, I think that it makes more sense to add a length field to the > ib_user_mad_hdr. Why not just expect the user to read a MAD until the read returns zero? I'm thinking something like this: read( offset = 0, len = 256 ) read( offset 256, len = ) So a read at offset 0 would block waiting for the next MAD, but a read with a non-zero offset would return EOF if the full MAD was read. Thoughts? - Fab From halr at voltaire.com Thu Jun 30 14:40:38 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 Jun 2005 17:40:38 -0400 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <42C464E6.8040406@ichips.intel.com> References: <1120149968.4371.81.camel@hal.voltaire.com> <52d5q365uo.fsf@topspin.com> <42C464E6.8040406@ichips.intel.com> Message-ID: <1120167637.4371.630.camel@hal.voltaire.com> On Thu, 2005-06-30 at 17:32, Sean Hefty wrote: > Roland Dreier wrote: > > I understand and agree with the sentiment of not wanting to add > > another ioctl() to get the length. Instead, how about returning a > > ib_user_mad_hdr with a status of ENOSPC and putting the actual length > > somewhere. I'm not sure if it's better to change the ABI and add a > > length field to ib_user_mad_hdr, or if we want to return the first 36 > > bytes of an RMPP MAD so the user can figure out the correct length. > > Unless the MAD layer modifies the received MAD, the length may not be set in > the header. Setting this doesn't seem like a big deal. If it is set, then > I'm guessing that we'd want to set the PayloadLength to the value indicated > by the spec, but that's not the easiest value to use in order to determine > the size of the read. > > Given that, I think that it makes more sense to add a length field to the > ib_user_mad_hdr. That's my conclusion too. ABI change :-( -- Hal From mshefty at ichips.intel.com Thu Jun 30 14:46:22 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 14:46:22 -0700 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <000301c57dbc$8aa5ea20$6501a8c0@infiniconsys.com> References: <000301c57dbc$8aa5ea20$6501a8c0@infiniconsys.com> Message-ID: <42C4682E.8040305@ichips.intel.com> Fab Tillier wrote: > Why not just expect the user to read a MAD until the read returns zero? > > I'm thinking something like this: > > read( offset = 0, len = 256 ) > read( offset 256, len = ) > > So a read at offset 0 would block waiting for the next MAD, but a read with a > non-zero offset would return EOF if the full MAD was read. > > Thoughts? The user would need to know how large of a buffer to allocate for the read. If the user needs to allocate two buffers, then they either need to be able to process data the spans multiple buffers, or a second data copy is required. There's also an issue if the user is using multiple threads... - Sean From ftillier at silverstorm.com Thu Jun 30 14:53:48 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 30 Jun 2005 14:53:48 -0700 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <42C4682E.8040305@ichips.intel.com> Message-ID: <000401c57dbe$32998fb0$6501a8c0@infiniconsys.com> > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > Sent: Thursday, June 30, 2005 2:46 PM > > Fab Tillier wrote: > > Why not just expect the user to read a MAD until the read returns zero? > > > > I'm thinking something like this: > > > > read( offset = 0, len = 256 ) > > read( offset 256, len = ) > > > > So a read at offset 0 would block waiting for the next MAD, but a read with > > a non-zero offset would return EOF if the full MAD was read. > > > > Thoughts? > > The user would need to know how large of a buffer to allocate for the read. > If the user needs to allocate two buffers, then they either need to be > able to process data the spans multiple buffers, or a second data copy is > required. There's also an issue if the user is using multiple threads... Ok, so my idea sucked. What about not coalescing in the kernel, and just handing up MADs to be coalesced in user-mode? It would require a buffer allocation in UM, as well as copies, but those currently happen in the kernel and could be eliminated, no? - Fab From mshefty at ichips.intel.com Thu Jun 30 15:01:01 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 15:01:01 -0700 Subject: [openib-general] [PATCH] user_mad: Add receive side RMPP support In-Reply-To: <000401c57dbe$32998fb0$6501a8c0@infiniconsys.com> References: <000401c57dbe$32998fb0$6501a8c0@infiniconsys.com> Message-ID: <42C46B9D.2030405@ichips.intel.com> Fab Tillier wrote: >>The user would need to know how large of a buffer to allocate for the read. >> If the user needs to allocate two buffers, then they either need to be >>able to process data the spans multiple buffers, or a second data copy is >>required. There's also an issue if the user is using multiple threads... > > > Ok, so my idea sucked. What about not coalescing in the kernel, and just > handing up MADs to be coalesced in user-mode? It would require a buffer > allocation in UM, as well as copies, but those currently happen in the kernel > and could be eliminated, no? Nah... your idea was fine as something to throw out. :) Currently, only a single data copy is performed to transfer received RMPP MADs from the kernel to a usermode app. The coalescing occurs as the data is copied from a chain of MADs in the kernel to the app's buffer. - Sean From tduffy at sun.com Thu Jun 30 15:10:07 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 30 Jun 2005 15:10:07 -0700 Subject: [openib-general] [oops] recent opensm crash Message-ID: <1120169407.29522.17.camel@duffman> #0 stack_dump () at src/stack.c:72 72 if (!__builtin_frame_address(2)) (gdb) bt #0 stack_dump () at src/stack.c:72 #1 0x00002aaaaacbd1a6 in handler (x=11) at src/stack.c:151 #2 #3 __osm_sm_mad_ctrl_send_err_cb (bind_context=0x550dd8, p_madw=0x567820) at osm_sm_mad_ctrl.c:832 #4 0x00002aaaaaaaeeed in osm_vendor_send (h_bind=0x586920, p_madw=0x567820, resp_expected=1) at osm_vendor_ibumad.c:889 #5 0x000000000042ef72 in __osm_vl15_poller (p_ptr=0x552620) at osm_madw.h:933 #6 0x00002aaaaadc911e in __cl_thread_wrapper (arg=0x0) at cl_thread.c:61 #7 0x00000036d28060aa in start_thread () from /lib64/tls/libpthread.so.0 #8 0x00000036d19c53d3 in clone () from /lib64/tls/libc.so.6 #9 0x0000000000000000 in ?? () I was bringing up and down an node when this happened. Attached are the last 500 lines from osm.log. -tduffy -------------- next part -------------- base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x1 (SubnGet) status..................0x0 hop_ptr.................0x0 hop_count...............0x1 trans_id................0x16c6a attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x50000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][0] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 30 14:59:00 [43806960] -> osm_vendor_send: [ Jun 30 14:59:00 [43005960] -> PortInfo dump: port number.............0x8 node_guid...............0x000617000000000d port_guid...............0x000617000000000d m_key...................0x0000000000000000 subnet_prefix...........0x0000000000000000 base_lid................0x0 master_sm_base_lid......0x0 capability_mask.........0x0 diag_code...............0x0 m_key_lease_period......0x0 local_port_num..........0x7 link_width_enabled......0x3 link_width_supported....0x3 link_width_active.......0x0 link_speed_supported....0x1 port_state..............DOWN state_info2.............0x22 m_key_protect_bits......0x0 lmc.....................0x0 link_speed..............0x11 mtu_smsl................0x10 vl_cap..................0x40 vl_high_limit...........0x0 vl_arb_high_cap.........0x20 vl_arb_low_cap..........0x20 mtu_cap.................0x5 vl_stall_life...........0x8 vl_enforce..............0x10 m_key_violations........0x0 p_key_violations........0x0 q_key_violations........0x0 guid_cap................0x0 subnet_timeout..........0x0 resp_time_value.........0x0 error_threshold.........0xFF Jun 30 14:59:00 [43005960] -> Capabilities Mask: Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_process_switch_port: [ Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_process_switch_port: ] Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: [ Jun 30 14:59:00 [43005960] -> osm_physp_has_pkey: [ Jun 30 14:59:00 [43005960] -> osm_req_get: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [ Jun 30 14:59:00 [43806960] -> __osm_mtl_send_callback: Completed Sending Request MADW: 0x5b0ac0. Jun 30 14:59:00 [43806960] -> osm_vendor_send: ] Jun 30 14:59:00 [43005960] -> osm_vendor_get: [ Jun 30 14:59:00 [44808960] -> osm_vendor_put: [ Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x591a50. Jun 30 14:59:00 [44808960] -> osm_vendor_put: ] Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43806960] -> __osm_vl15_poller: 1 on wire, 11 outstanding, 10 unicasts sent, 88641 sent total. Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_NODE_INFO. Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ] Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ] Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x567428, size = 256. Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: [ Jun 30 14:59:00 [44808960] -> osm_vendor_get: [ Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x592620, size = 256. Jun 30 14:59:00 [43005960] -> osm_vendor_get: ] Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x560268, size = 256. Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquired UMAD 0x591a50, size = 256. Jun 30 14:59:00 [44808960] -> osm_vendor_get: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x567410, p_mad = 0x592654, size = 256. Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ] Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x560250, p_mad = 0x591a84, size = 256. Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: ] Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: [ Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: 88641 QP0 MADs received. Jun 30 14:59:00 [43005960] -> osm_req_get: Getting P_KeyTable (0x16), modifier = 0x80000, TID = 0x16c6d. Jun 30 14:59:00 [43005960] -> osm_vl15_post: [ Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x567410 (mad 0x592654 req 1) Jun 30 14:59:00 [44808960] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x8000 hop_ptr.................0x0 hop_count...............0x1 trans_id................0x16c6a attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x50000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][7] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: [ Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: [ Jun 30 14:59:00 [43005960] -> osm_vl15_post: 1 MADs on wire, 12 MADs outstanding. Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [ Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread. Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: 0 SMPs on the wire, 12 outstanding. Jun 30 14:59:00 [44808960] -> osm_vl15_poll: [ Jun 30 14:59:00 [44808960] -> osm_vl15_poll: Signalling poller thread. Jun 30 14:59:00 [44808960] -> osm_vl15_poll: ] Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: ] Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0ac0, p_mad = 0x5930e4. Jun 30 14:59:00 [44808960] -> osm_vendor_put: [ Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x5930b0. Jun 30 14:59:00 [44808960] -> osm_vendor_put: ] Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43806960] -> __osm_vl15_poller: Servicing p_madw = 0x567820 (mad 0x591bc4 req 1) Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_PKEY. Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ] Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ] Jun 30 14:59:00 [43806960] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x1 (SubnGet) status..................0x0 hop_ptr.................0x0 hop_count...............0x1 trans_id................0x16c6b attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x60000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][0] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ] Jun 30 14:59:00 [43806960] -> osm_vendor_send: [ Jun 30 14:59:00 [43005960] -> osm_vl15_post: ] Jun 30 14:59:00 [43005960] -> osm_req_get: ] Jun 30 14:59:00 [43005960] -> osm_physp_has_pkey: ] Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: ] Jun 30 14:59:00 [43005960] -> osm_pi_rcv_process: ] Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: [ Jun 30 14:59:00 [44808960] -> osm_vendor_get: [ Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5b0ad8, size = 256. Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquired UMAD 0x5930b0, size = 256. Jun 30 14:59:00 [44808960] -> osm_vendor_get: ] Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x5b0ac0, p_mad = 0x5930e4, size = 256. Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: ] Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: [ Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: 88642 QP0 MADs received. Jun 30 14:59:00 [43806960] -> osm_vendor_send: Send failed -5 (Success). Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43806960] -> __osm_sm_mad_ctrl_send_err_cb: [ Jun 30 14:59:00 [44808960] -> SMP dump: base_ver................0x1 mgmt_class..............0x81 class_ver...............0x1 method..................0x81 (SubnGetResp) status..................0x8000 hop_ptr.................0x0 hop_count...............0x1 trans_id................0x16c6b attr_id.................0x16 (P_KeyTable) resv....................0x0 attr_mod................0x60000 m_key...................0x0000000000000000 dr_slid.................0xFFFF dr_dlid.................0xFFFF Initial path: [0][1] Return path: [0][7] Reserved: [0][0][0][0][0][0][0] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: [ Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: [ Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: 0 SMPs on the wire, 12 outstanding. Jun 30 14:59:00 [44808960] -> osm_vl15_poll: [ Jun 30 14:59:00 [44808960] -> osm_vl15_poll: Signalling poller thread. Jun 30 14:59:00 [44808960] -> osm_vl15_poll: ] Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: ] Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: Releasing p_madw = 0x567820, p_mad = 0x591bc4. Jun 30 14:59:00 [44808960] -> osm_vendor_put: [ Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x591b90. Jun 30 14:59:00 [44808960] -> osm_vendor_put: ] Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_PKEY. Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ] Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c62. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0370, p_mad = 0x5b1244. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5b1210. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 11 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [ Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 0 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c63. Jun 30 14:59:00 [43005960] -> P_Key table dump: port_guid...........0x000617000000000d block_num...........0x0 port_num............0x0 P_Key Table: 0XFFFF | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c63. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x55df60, p_mad = 0x5b0c74. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5b0c40. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 10 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [ Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 1 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c64. Jun 30 14:59:00 [43005960] -> P_Key table dump: port_guid...........0x000617000000000d block_num...........0x0 port_num............0x1 P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c64. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0920, p_mad = 0x5916c4. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591690. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 9 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [ Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 2 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c65. Jun 30 14:59:00 [43005960] -> P_Key table dump: port_guid...........0x000617000000000d block_num...........0x0 port_num............0x2 P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c65. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x55add0, p_mad = 0x591d04. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591cd0. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 8 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: [ Jun 30 14:59:00 [43005960] -> NodeInfo dump: base_version............0x1 class_version...........0x1 node_type...............Channel Adapter num_ports...............0x2 sys_guid................0x0002c9000100d050 node_guid...............0x0002c901097624c0 port_guid...............0x0002c901097624c1 partition_cap...........0x40 device_id...............0x5A44 revision................0xA1 port_num................0x1 vendor_id...............0x2C9 Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: [ Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: Rediscovered Channel Adapter node 0x2c901097624c0 TID = 0x16c66, discovered 0 times already. Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: [ Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: [ Jun 30 14:59:00 [43005960] -> osm_req_get: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [ Jun 30 14:59:00 [43005960] -> osm_vendor_get: [ Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x55ade8, size = 256. Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x5b1210, size = 256. Jun 30 14:59:00 [43005960] -> osm_vendor_get: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x55add0, p_mad = 0x5b1244, size = 256. Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ] Jun 30 14:59:00 [43005960] -> osm_req_get: Getting PortInfo (0x15), modifier = 0x1, TID = 0x16c6e. Jun 30 14:59:00 [43005960] -> osm_vl15_post: [ Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x55add0 (mad 0x5b1244 req 1) Jun 30 14:59:00 [43005960] -> osm_vl15_post: 0 MADs on wire, 9 MADs outstanding. Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [ Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread. Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ] Jun 30 14:59:00 [43005960] -> osm_vl15_post: ] Jun 30 14:59:00 [43005960] -> osm_req_get: ] Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: ] Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: ] Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: [ Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: Link already exists. Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: ] Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: ] Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c66. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b09f0, p_mad = 0x592fa4. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592f70. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 8 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [ Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 3 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c67. Jun 30 14:59:00 [43005960] -> P_Key table dump: port_guid...........0x000617000000000d block_num...........0x0 port_num............0x3 P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c67. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x561f90, p_mad = 0x592e64. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592e30. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 7 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [ Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 4 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c68. Jun 30 14:59:00 [43005960] -> P_Key table dump: port_guid...........0x000617000000000d block_num...........0x0 port_num............0x4 P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c68. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0850, p_mad = 0x591804. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5917d0. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 6 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: [ Jun 30 14:59:00 [43005960] -> NodeInfo dump: base_version............0x1 class_version...........0x1 node_type...............Channel Adapter num_ports...............0x2 sys_guid................0x0002c90109765633 node_guid...............0x0002c90109765630 port_guid...............0x0002c90109765631 partition_cap...........0x20 device_id...............0x5A44 revision................0xA1 port_num................0x1 vendor_id...............0x2C9 Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: [ Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: Rediscovered Channel Adapter node 0x2c90109765630 TID = 0x16c69, discovered 0 times already. Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: [ Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: [ Jun 30 14:59:00 [43005960] -> osm_req_get: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [ Jun 30 14:59:00 [43005960] -> osm_vendor_get: [ Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5b0868, size = 256. Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x5b0c40, size = 256. Jun 30 14:59:00 [43005960] -> osm_vendor_get: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x5b0850, p_mad = 0x5b0c74, size = 256. Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ] Jun 30 14:59:00 [43005960] -> osm_req_get: Getting PortInfo (0x15), modifier = 0x1, TID = 0x16c6f. Jun 30 14:59:00 [43005960] -> osm_vl15_post: [ Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x5b0850 (mad 0x5b0c74 req 1) Jun 30 14:59:00 [43005960] -> osm_vl15_post: 0 MADs on wire, 7 MADs outstanding. Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [ Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread. Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ] Jun 30 14:59:00 [43005960] -> osm_vl15_post: ] Jun 30 14:59:00 [43005960] -> osm_req_get: ] Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: ] Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: ] Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: [ Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: Link already exists. Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: ] Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: ] Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c69. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b02a0, p_mad = 0x592144. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592110. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 6 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [ Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 5 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c6a. Jun 30 14:59:00 [43005960] -> P_Key table dump: port_guid...........0x000617000000000d block_num...........0x0 port_num............0x5 P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c6a. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x560250, p_mad = 0x592514. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5924e0. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 5 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [ Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 6 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c6b. Jun 30 14:59:00 [43005960] -> P_Key table dump: port_guid...........0x000617000000000d block_num...........0x0 port_num............0x6 P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [ Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c6b. Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [ Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0ac0, p_mad = 0x591a84. Jun 30 14:59:00 [43005960] -> osm_vendor_put: [ Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591a50. Jun 30 14:59:00 [43005960] -> osm_vendor_put: ] Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 4 QP0 MADs outstanding. Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ] Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ] Jun 30 14:59:00 [43806960] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_SUCCESS). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Thu Jun 30 15:14:25 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 30 Jun 2005 15:14:25 -0700 Subject: [openib-general] [PATCHv2] SDP: use linux/list.h for listen conns In-Reply-To: <1120165084.29522.7.camel@duffman> References: <1120165084.29522.7.camel@duffman> Message-ID: <1120169665.29522.21.camel@duffman> This patch changes sdp to use Linux native lists for the listen conn list. The first version had a bug where if the listen list was empty, the lookup would return a bogus conn. Signed-off-by: Tom Duffy Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2763) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -291,21 +291,11 @@ int sdp_inet_listen_start(struct sdp_soc conn->state = SDP_CONN_ST_LISTEN; conn->accept_next = conn; conn->accept_prev = conn; - /* - * table lock - */ - spin_lock_irqsave(&dev_root_s.listen_lock, flags); - /* - * insert into listening list. - */ - conn->lstn_next = dev_root_s.listen_list; - dev_root_s.listen_list = conn; - conn->lstn_p_next = &dev_root_s.listen_list; - - if (conn->lstn_next) - conn->lstn_next->lstn_p_next = &conn->lstn_next; + spin_lock_irqsave(&dev_root_s.listen_lock, flags); + list_add(&conn->lstn_next, &dev_root_s.listen_list); spin_unlock_irqrestore(&dev_root_s.listen_lock, flags); + return 0; } @@ -323,22 +313,11 @@ int sdp_inet_listen_stop(struct sdp_sock } listen_conn->state = SDP_CONN_ST_CLOSED; - /* - * table lock - */ - spin_lock_irqsave(&dev_root_s.listen_lock, flags); - /* - * remove from listening list. - */ - if (listen_conn->lstn_next) - listen_conn->lstn_next->lstn_p_next = listen_conn->lstn_p_next; - - *(listen_conn->lstn_p_next) = listen_conn->lstn_next; - - listen_conn->lstn_p_next = NULL; - listen_conn->lstn_next = NULL; + spin_lock_irqsave(&dev_root_s.listen_lock, flags); + list_del(&listen_conn->lstn_next); spin_unlock_irqrestore(&dev_root_s.listen_lock, flags); + /* * reject and delete all pending connections */ @@ -367,7 +346,7 @@ int sdp_inet_listen_stop(struct sdp_sock */ struct sdp_sock *sdp_inet_listen_lookup(u32 addr, u16 port) { - struct sdp_sock *conn; + struct sdp_sock *conn, *ret = NULL; unsigned long flags; /* * table lock @@ -376,15 +355,16 @@ struct sdp_sock *sdp_inet_listen_lookup( /* * first find a listening connection */ - for (conn = dev_root_s.listen_list; conn; conn = conn->lstn_next) + list_for_each_entry(conn, &dev_root_s.listen_list, lstn_next) if (port == conn->src_port && (INADDR_ANY == conn->src_addr || addr == conn->src_addr)) { sdp_conn_hold(conn); + ret = conn; break; } spin_unlock_irqrestore(&dev_root_s.listen_lock, flags); - return conn; + return ret; } /* @@ -1936,7 +1916,7 @@ int sdp_conn_table_init(int proto_family /* * list */ - dev_root_s.listen_list = NULL; + INIT_LIST_HEAD(&dev_root_s.listen_list); dev_root_s.bind_list = NULL; spin_lock_init(&dev_root_s.sock_lock); Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (revision 2763) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -36,6 +36,8 @@ #ifndef _SDP_CONN_H #define _SDP_CONN_H +#include + #include "sdp_advt.h" #include "sdp_iocb.h" #include "sdp_dev.h" @@ -336,8 +338,7 @@ struct sdp_sock { /* * table managment */ - struct sdp_sock *lstn_next; /* next conn in the chain */ - struct sdp_sock **lstn_p_next; /* previous next conn in the chain */ + struct list_head lstn_next; struct sdp_sock *bind_next; /* next conn in the chain */ struct sdp_sock **bind_p_next; /* previous next conn in the chain */ Index: linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h =================================================================== --- linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (revision 2763) +++ linux-2.6.12-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (working copy) @@ -43,6 +43,7 @@ #include #include #include +#include #include /* * sdp types @@ -186,7 +187,7 @@ struct sdev_root { /* * connection managment */ - struct sdp_sock *listen_list; /* list of listening connections */ + struct list_head listen_list; struct sdp_sock *bind_list; /* connections bound to a port. */ /* * list locks From jlentini at netapp.com Thu Jun 30 15:18:12 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 30 Jun 2005 18:18:12 -0400 (EDT) Subject: [openib-general] gen2/rnic-pi differences In-Reply-To: <8508251A6FC08A489844A94261D3693A057C74@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A057C74@fiona.siliquent.com> Message-ID: On Thu, 30 Jun 2005, Caitlin Bestler wrote: > os_data / Identification of Consumer Objects > > gen2 provides minimal support for identification of RDMA > resources using consumer supplied handles. A user-supplied > context is available in callbacks, but not in work completions. Gen2 work completions store a user-supplied context in the ib_wc structure's wr_id field. From tduffy at sun.com Thu Jun 30 15:41:55 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 30 Jun 2005 15:41:55 -0700 Subject: [openib-general] [PATCH] user_mad: In send_handler, only return header on timeout In-Reply-To: <1119993231.4848.2661.camel@hal.voltaire.com> References: <1119993231.4848.2661.camel@hal.voltaire.com> Message-ID: <1120171315.29522.24.camel@duffman> Hal, Would you mind versioning your patches? That way I don't think you are just resending the same email. Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Thu Jun 30 16:01:31 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 30 Jun 2005 16:01:31 -0700 Subject: [openib-general] [PATCH] kDAPL: remove redundant kfree checks Message-ID: <1120172491.29522.28.camel@duffman> kfree() already checks for NULL. No need to do it twice. Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel/dat-provider/dapl_cookie.c (revision 2761) +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) @@ -157,8 +157,7 @@ u32 dapl_cb_create(struct dapl_cookie_bu */ void dapl_cb_free(struct dapl_cookie_buffer *buffer) { - if (NULL != buffer->pool) - kfree(buffer->pool); + kfree(buffer->pool); } /* Index: linux-kernel/dat-provider/dapl_ring_buffer_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ring_buffer_util.c (revision 2761) +++ linux-kernel/dat-provider/dapl_ring_buffer_util.c (working copy) @@ -149,7 +149,7 @@ bail: */ void dapl_rbuf_destroy(struct dapl_ring_buffer *rbuf) { - if ((NULL == rbuf) || (NULL == rbuf->base)) + if (!rbuf) return; kfree(rbuf->base); Index: linux-kernel/dat-provider/dapl_ep.c =================================================================== --- linux-kernel/dat-provider/dapl_ep.c (revision 2761) +++ linux-kernel/dat-provider/dapl_ep.c (working copy) @@ -47,11 +47,9 @@ static void dapl_ep_dealloc(struct dapl_ dapl_cb_free(&ep->req_buffer); dapl_cb_free(&ep->recv_buffer); - if (ep->recv_iov) - kfree(ep->recv_iov); + kfree(ep->recv_iov); - if (ep->send_iov) - kfree(ep->send_iov); + kfree(ep->send_iov); kfree(ep); } Index: linux-kernel/dat-provider/dapl_evd.c =================================================================== --- linux-kernel/dat-provider/dapl_evd.c (revision 2761) +++ linux-kernel/dat-provider/dapl_evd.c (working copy) @@ -265,9 +265,7 @@ static u32 dapl_evd_dealloc(struct dapl_ dapl_rbuf_destroy(&evd->free_event_queue); dapl_rbuf_destroy(&evd->pending_event_queue); - if (evd->events) { - kfree(evd->events); - } + kfree(evd->events); kfree(evd); Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2761) +++ linux-kernel/dat/api.c (working copy) @@ -524,8 +524,7 @@ out_unlock: spin_unlock_irqrestore(&dat_provider_list_lock, flags); out: if (status != DAT_SUCCESS) - if (entry) - kfree(entry); + kfree(entry); return status; } EXPORT_SYMBOL(dat_registry_add_provider); From sean.hefty at intel.com Thu Jun 30 16:23:54 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 30 Jun 2005 16:23:54 -0700 Subject: [openib-general] [PATCH] [MAD] Convert RMPP code to use MAD APIs for send MADs Message-ID: The following patch converts the RMPP code to use the MAD layer APIs for allocating send MAD data buffers when sending ACKs. Signed-off-by: Sean Hefty Index: mad_rmpp.c =================================================================== --- mad_rmpp.c (revision 2760) +++ mad_rmpp.c (working copy) @@ -69,14 +69,6 @@ struct mad_rmpp_recv { u8 method; }; -struct rmpp_msg { - struct ib_rmpp_mad mad; - DECLARE_PCI_UNMAP_ADDR(mapping) - struct ib_mad_agent *mad_agent; - struct ib_send_wr send_wr; - struct ib_sge sge; -}; - static void destroy_rmpp_recv(struct mad_rmpp_recv *rmpp_recv) { atomic_dec(&rmpp_recv->refcount); @@ -238,47 +230,6 @@ insert_rmpp_recv(struct ib_mad_agent_pri return cur_rmpp_recv; } -static struct rmpp_msg * alloc_rmpp_msg(struct ib_mad_agent *mad_agent, - u32 remote_qpn, u16 pkey_index, - struct ib_ah *ah) -{ - struct rmpp_msg *msg; - - msg = kmalloc(sizeof *msg, GFP_KERNEL); - if (!msg) - return NULL; - memset(msg, 0, sizeof *msg); - - msg->sge.addr = dma_map_single(mad_agent->device->dma_device, - &msg->mad, sizeof msg->mad, - DMA_TO_DEVICE); - pci_unmap_addr_set(msg, mapping, msg->sge.addr); - msg->sge.length = sizeof msg->mad; - msg->sge.lkey = mad_agent->mr->lkey; - - msg->send_wr.wr_id = (unsigned long) msg; - msg->send_wr.sg_list = &msg->sge; - msg->send_wr.num_sge = 1; - msg->send_wr.opcode = IB_WR_SEND; - msg->send_wr.send_flags = IB_SEND_SIGNALED; - msg->send_wr.wr.ud.ah = ah; - msg->send_wr.wr.ud.mad_hdr = &msg->mad.mad_hdr; - msg->send_wr.wr.ud.remote_qpn = remote_qpn; - msg->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY; - msg->send_wr.wr.ud.pkey_index = pkey_index; - - msg->mad_agent = mad_agent; - return msg; -} - -static void free_rmpp_msg(struct rmpp_msg *msg) -{ - dma_unmap_single(msg->mad_agent->device->dma_device, - pci_unmap_addr(msg, mapping), - sizeof msg->mad, DMA_TO_DEVICE); - kfree(msg); -} - static int data_offset(u8 mgmt_class) { if (mgmt_class == IB_MGMT_CLASS_SUBN_ADM) @@ -313,21 +264,24 @@ static void format_ack(struct ib_rmpp_ma static void ack_recv(struct mad_rmpp_recv *rmpp_recv, struct ib_mad_recv_wc *recv_wc) { - struct rmpp_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; - int ret; + int hdr_len, ret; - msg = alloc_rmpp_msg(&rmpp_recv->agent->agent, recv_wc->wc->src_qp, - recv_wc->wc->pkey_index, rmpp_recv->ah); + hdr_len = sizeof(struct ib_mad_hdr) + sizeof(struct ib_rmpp_hdr); + msg = ib_create_send_mad(&rmpp_recv->agent->agent, recv_wc->wc->src_qp, + recv_wc->wc->pkey_index, rmpp_recv->ah, 1, + hdr_len, sizeof(struct ib_rmpp_mad) - hdr_len, + GFP_KERNEL); if (!msg) return; - format_ack(&msg->mad, (struct ib_rmpp_mad *) recv_wc->recv_buf.mad, - rmpp_recv); + format_ack((struct ib_rmpp_mad *) msg->mad, + (struct ib_rmpp_mad *) recv_wc->recv_buf.mad, rmpp_recv); ret = ib_post_send_mad(&rmpp_recv->agent->agent, &msg->send_wr, &bad_send_wr); if (ret) - free_rmpp_msg(msg); + ib_free_send_mad(msg); } static inline int get_last_flag(struct ib_mad_recv_buf *seg) @@ -749,7 +703,7 @@ int ib_process_rmpp_send_wc(struct ib_ma struct ib_mad_send_wc *mad_send_wc) { struct ib_rmpp_mad *rmpp_mad; - struct rmpp_msg *msg; + struct ib_mad_send_buf *msg; int ret; rmpp_mad = (struct ib_rmpp_mad *)mad_send_wr->send_wr.wr.ud.mad_hdr; @@ -758,8 +712,9 @@ int ib_process_rmpp_send_wc(struct ib_ma return IB_RMPP_RESULT_UNHANDLED; /* RMPP not active */ if (rmpp_mad->rmpp_hdr.rmpp_type != IB_MGMT_RMPP_TYPE_DATA) { - msg = (struct rmpp_msg *) (unsigned long) mad_send_wc->wr_id; - free_rmpp_msg(msg); + msg = (struct ib_mad_send_buf *) (unsigned long) + mad_send_wc->wr_id; + ib_free_send_mad(msg); return IB_RMPP_RESULT_INTERNAL; /* ACK, STOP, or ABORT */ } From libor at topspin.com Thu Jun 30 18:16:36 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 30 Jun 2005 18:16:36 -0700 Subject: [openib-general] Re: [PATCH (draft)] sdp: fix aio/sync completion race In-Reply-To: <20050628201857.GA5987@mellanox.co.il>; from mst@mellanox.co.il on Tue, Jun 28, 2005 at 11:18:57PM +0300 References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> <20050513135145.GA320@mellanox.co.il> <20050513100149.B24123@topspin.com> <20050525202128.GA7463@mellanox.co.il> <20050525172954.A2979@topspin.com> <20050628201857.GA5987@mellanox.co.il> Message-ID: <20050630181636.A29268@topspin.com> On Tue, Jun 28, 2005 at 11:18:57PM +0300, Michael S. Tsirkin wrote: > Libor, here's a stub at solving an old problem. > Most of the patch is passing sdp_opt to iocb complete and cancel functions. > > I didnt yet test it, and I wont have the time today, > but maybe you could tell me whether I'm going in the right > direction with this. I'm wondering if we should prevent a new call from proceeding until all IOCBs have been completed, like the patch you have, or if we should force the creation of an IOCB when there are any IOCBs outstanding, even if the call can be processed synchronously. Remember the original problem was IOCB completes in the work queue completing after a call to send/recv that was processed synchronously. Either forcing an IOCB or waiting on the lock should prevent the ordering issue. I think either case if means passing the sdp_opt to iocb_complete. In do_iocb_complete you cannot call conn_unlock unconditionally since the lock.users variable is only incremented in_atomic() or irqs_disabled() Also, I think the conn reference needs to be incremented before creating the work request, and decremeneted upon completion since nothing prevents it from going away before the thread is scheduled. -Libor > --- > > Use "users" in the lock to prevent more iocbs from completing before > a current aio is done. > Any synchronous transfer would then wait till socket is unlocked. > (Once we switch to get_user_pages, we'll be able to do it only for receive > iocbs). > > Signed-off-by: Michael S. Tsirkin > > Index: sdp_write.c > =================================================================== > --- sdp_write.c (revision 2726) > +++ sdp_write.c (working copy) > @@ -109,7 +109,7 @@ int sdp_event_write(struct sdp_opt *conn > SDP_CONN_STAT_WRITE_INC(conn, iocb->post); > SDP_CONN_STAT_WQ_DEC(conn, iocb->size); > > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > > break; > case SDP_DESC_TYPE_NONE: > Index: sdp_rcvd.c > =================================================================== > --- sdp_rcvd.c (revision 2726) > +++ sdp_rcvd.c (working copy) > @@ -152,7 +152,7 @@ static int sdp_rcvd_send_sm(struct sdp_o > > conn->src_sent--; > > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > } > /* > * Cancel complete, clear the state. > @@ -209,7 +209,7 @@ static int sdp_rcvd_rdma_wr(struct sdp_o > > conn->snk_sent--; > > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > > return 0; > } > @@ -270,7 +270,7 @@ static int sdp_rcvd_rdma_rd(struct sdp_o > SDP_CONN_STAT_WRITE_INC(conn, iocb->post); > SDP_CONN_STAT_WQ_DEC(conn, iocb->size); > > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > } > /* > * If Source Cancel was in process, and there are no more outstanding > @@ -562,7 +562,7 @@ static int sdp_rcvd_snk_cancel_ack(struc > while ((iocb = sdp_iocb_q_get_head(&conn->r_snk))) { > > conn->snk_sent--; > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > } > /* > * cancellation is complete. Cancel flag is cleared in RECV post. > @@ -684,7 +684,7 @@ static int sdp_rcvd_snk_avail(struct sdp > sdp_desc_q_put_head(&conn->send_queue, > (struct sdpc_desc *)iocb); > else > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > } > /* > * If Source Cancel was in process, it should now > Index: sdp_proto.h > =================================================================== > --- sdp_proto.h (revision 2726) > +++ sdp_proto.h (working copy) > @@ -162,7 +162,8 @@ void sdp_iocb_q_put_tail(struct sdpc_ioc > > struct sdpc_iocb *sdp_iocb_q_lookup(struct sdpc_iocb_q *table, u32 key); > > -void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp); > +void sdp_iocb_q_cancel(struct sdp_opt *conn, struct sdpc_iocb_q *table, > + u32 mask, ssize_t comp); > > void sdp_iocb_q_remove(struct sdpc_iocb *iocb); > > @@ -170,7 +171,8 @@ int sdp_iocb_register(struct sdpc_iocb * > > void sdp_iocb_release(struct sdpc_iocb *iocb); > > -void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status); > +void sdp_iocb_complete(struct sdp_opt *conn, struct sdpc_iocb *iocb, > + ssize_t status); > > int sdp_iocb_lock(struct sdpc_iocb *iocb); > > Index: sdp_read.c > =================================================================== > --- sdp_read.c (revision 2726) > +++ sdp_read.c (working copy) > @@ -205,7 +205,7 @@ int sdp_event_read(struct sdp_opt *conn, > SDP_CONN_STAT_READ_INC(conn, iocb->post); > SDP_CONN_STAT_RQ_DEC(conn, iocb->size); > > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > > break; > case SDP_DESC_TYPE_NONE: > @@ -226,7 +226,7 @@ int sdp_event_read(struct sdp_opt *conn, > SDP_CONN_STAT_READ_INC(conn, iocb->post); > SDP_CONN_STAT_RQ_DEC(conn, iocb->size); > > - sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0); > + sdp_iocb_complete(conn, sdp_iocb_q_get_head(&conn->r_pend), 0); > > break; > default: > Index: sdp_send.c > =================================================================== > --- sdp_send.c (revision 2726) > +++ sdp_send.c (working copy) > @@ -859,7 +859,7 @@ static int sdp_send_data_iocb(struct sdp > SDP_CONN_STAT_WRITE_INC(conn, iocb->post); > SDP_CONN_STAT_WQ_DEC(conn, iocb->size); > > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > } > > goto done; > @@ -1730,7 +1730,7 @@ static int sdp_inet_write_cancel(struct > * needs to be compelted. > */ > if (iocb->post > 0) { > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > result = -EAGAIN; > } else { > sdp_iocb_destroy(iocb); > Index: sdp_conn.c > =================================================================== > --- sdp_conn.c (revision 2726) > +++ sdp_conn.c (working copy) > @@ -697,7 +697,8 @@ static int sdp_desc_q_cancel_lookup_func > return ((element->type == SDP_DESC_TYPE_IOCB) ? 0 : -ERANGE); > } > > -static void sdp_desc_q_cancel_iocb(struct sdpc_desc_q *table, ssize_t error) > +static void sdp_desc_q_cancel_iocb(struct sdp_opt *conn, > + struct sdpc_desc_q *table, ssize_t error) > { > struct sdpc_iocb *iocb; > > @@ -707,24 +708,24 @@ static void sdp_desc_q_cancel_iocb(struc > NULL))) { > > sdp_iocb_q_remove(iocb); > - sdp_iocb_complete(iocb, error); > + sdp_iocb_complete(conn, iocb, error); > } > } > > void sdp_iocb_q_cancel_all_read(struct sdp_opt *conn, ssize_t error) > { > - sdp_iocb_q_cancel(&conn->r_pend, SDP_IOCB_F_ALL, error); > - sdp_iocb_q_cancel(&conn->r_snk, SDP_IOCB_F_ALL, error); > + sdp_iocb_q_cancel(conn, &conn->r_pend, SDP_IOCB_F_ALL, error); > + sdp_iocb_q_cancel(conn, &conn->r_snk, SDP_IOCB_F_ALL, error); > > - sdp_desc_q_cancel_iocb(&conn->r_src, error); > + sdp_desc_q_cancel_iocb(conn, &conn->r_src, error); > } > > void sdp_iocb_q_cancel_all_write(struct sdp_opt *conn, ssize_t error) > { > - sdp_iocb_q_cancel(&conn->w_src, SDP_IOCB_F_ALL, error); > + sdp_iocb_q_cancel(conn, &conn->w_src, SDP_IOCB_F_ALL, error); > > - sdp_desc_q_cancel_iocb(&conn->send_queue, error); > - sdp_desc_q_cancel_iocb(&conn->w_snk, error); > + sdp_desc_q_cancel_iocb(conn, &conn->send_queue, error); > + sdp_desc_q_cancel_iocb(conn, &conn->w_snk, error); > } > > void sdp_iocb_q_cancel_all(struct sdp_opt *conn, ssize_t error) > Index: sdp_recv.c > =================================================================== > --- sdp_recv.c (revision 2726) > +++ sdp_recv.c (working copy) > @@ -663,7 +663,7 @@ static int sdp_recv_buff_iocb_active(str > /* > * callback to complete IOCB > */ > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > > return (buff->tail - buff->data); > } > @@ -716,7 +716,7 @@ static int sdp_recv_buff_iocb_pending(st > /* > * callback to complete IOCB > */ > - sdp_iocb_complete(sdp_iocb_q_get_head(&conn->r_pend), 0); > + sdp_iocb_complete(conn, sdp_iocb_q_get_head(&conn->r_pend), 0); > } > > return (buff->tail - buff->data); > @@ -875,7 +875,7 @@ static int sdp_inet_read_cancel(struct k > /* > * callback to complete IOCB, or drop reference > */ > - sdp_iocb_complete(iocb, 0); > + sdp_iocb_complete(conn, iocb, 0); > result = -EAGAIN; > } > else { > Index: sdp_conn.h > =================================================================== > --- sdp_conn.h (revision 2726) > +++ sdp_conn.h (working copy) > @@ -471,7 +471,7 @@ static inline void sdp_conn_unlock(struc > sdp_conn_internal_unlock(conn); > } > > - conn->lock.users = 0; > + --conn->lock.users; > wake_up(&conn->lock.waitq); > > spin_unlock_irqrestore(&conn->lock.slock, flags); > Index: sdp_iocb.c > =================================================================== > --- sdp_iocb.c (revision 2726) > +++ sdp_iocb.c (working copy) > @@ -508,6 +508,8 @@ static void do_iocb_complete(void *arg) > * we ignore the value. > */ > (void)aio_complete(iocb->req, value, 0); > + > + sdp_conn_unlock(iocb->conn); > /* > * delete IOCB > */ > @@ -517,11 +519,14 @@ static void do_iocb_complete(void *arg) > /* > * sdp_iocb_complete - complete an IOCB > */ > -void sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status) > +void sdp_iocb_complete(struct sdp_opt *conn, struct sdpc_iocb *iocb, > + ssize_t status) > { > iocb->status = status; > > if (in_atomic() || irqs_disabled()) { > + conn->lock.users++; > + iocb->conn = conn; > INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb); > schedule_work(&iocb->completion); > } else > @@ -750,7 +755,8 @@ void sdp_iocb_q_put_head(struct sdpc_ioc > /* > * sdp_iocb_q_cancel - cancel all outstanding AIOs in a queue > */ > -void sdp_iocb_q_cancel(struct sdpc_iocb_q *table, u32 mask, ssize_t comp) > +void sdp_iocb_q_cancel(struct sdp_opt *conn, struct sdpc_iocb_q *table, > + u32 mask, ssize_t comp) > { > struct sdpc_iocb *iocb; > struct sdpc_iocb *next; > @@ -772,7 +778,7 @@ void sdp_iocb_q_cancel(struct sdpc_iocb_ > iocb->post, iocb->len); > > sdp_iocb_q_remove(iocb); > - sdp_iocb_complete(iocb, comp); > + sdp_iocb_complete(conn, iocb, comp); > } > > iocb = next; > Index: sdp_iocb.h > =================================================================== > --- sdp_iocb.h (revision 2726) > +++ sdp_iocb.h (working copy) > @@ -109,7 +109,8 @@ struct sdpc_iocb { > int page_count; /* number of physical pages. */ > int page_offset; /* offset into first page. */ > > - struct work_struct completion; /* task for defered completion. */ > + struct work_struct completion; /* task for deferred completion. */ > + struct sdp_opt *conn; /* conn to unlock for deferred completion. */ > /* > * kernel iocb structure > */ > > -- > MST From ardavis at ichips.intel.com Thu Jun 30 19:01:22 2005 From: ardavis at ichips.intel.com (Arlin Davis) Date: Thu, 30 Jun 2005 19:01:22 -0700 Subject: [openib-general] Re: uCM create connection ID In-Reply-To: <20050629181054.J26240@topspin.com> References: <42C1BA15.7060205@ichips.intel.com> <20050629111038.G26240@topspin.com> <42C2F991.60005@ichips.intel.com> <20050629181054.J26240@topspin.com> Message-ID: <42C4A3F2.2020000@ichips.intel.com> Libor Michalek wrote: > The listen id is in the req rcvd event. (event->param.req_rcvd.listen_id) >Do you mean that it is not being set correctly? > > Ok, I didn't look deep enough. It is set correctly and the polling seems to be working. The uDAPL code is now connecting properly but I am having difficulty setting the QP states properly without the ib_cm_init_qp_attr() call. Any chance of providing this call in uCM? Thanks, -arlin From ardavis at ichips.intel.com Thu Jun 30 19:12:13 2005 From: ardavis at ichips.intel.com (Arlin Davis) Date: Thu, 30 Jun 2005 19:12:13 -0700 Subject: [openib-general] [PATCH] [CM] verify cm_id state before allocating messages In-Reply-To: References: Message-ID: <42C4A67D.9050502@ichips.intel.com> Sean Hefty wrote: >The following patch moves verification of the cm_id state higher up >in the CM function calls. This fixes an issue trying to allocate a >message for a cm_id that may be in an invalid state, which could >result in a crash. > >This problem was first reported by Hal a while ago, but was recently >hit by Arlin running a usermode application. Arlin, can you verify >that this fixes your issues? > > Fix looks good, returns error instead of kernel oops. >Signed-off-by: Sean Hefty > > > > >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > >